Go to file

Andrey Petrov 10a5421bc3 Readme.		2014-05-19 15:30:18 -07:00
queue	Some race condition fixen.	2014-05-17 22:16:05 -07:00
.gitignore	Dont worry about jsons.	2014-05-14 15:47:36 -07:00
LICENSE	Initial commit	2014-04-18 21:07:31 -07:00
Makefile	Refactored the parallelized bits, still broken though.	2014-05-17 19:44:33 -07:00
README.md	Readme.	2014-05-19 15:30:18 -07:00
client.go	Flags for ALL the things.	2014-04-23 16:03:34 -07:00
cmd.go	Count failures as otherMessages	2014-05-19 15:19:45 -07:00
crawler.go	oops, actually give up when getting error messages.	2014-05-19 15:26:00 -07:00
ext_btcd.go	Importing code.	2014-04-18 21:26:47 -07:00
log.go	Superhappyfuntimelogging.	2014-04-24 19:13:33 -07:00
peer.go	Properly block on first item.	2014-05-15 17:04:09 -07:00
seed.go	README update.	2014-05-19 15:14:24 -07:00

README.md

btc-crawl

Bitcoin node network crawler (written in golang).

This is a for-fun project to explore the Bitcoin protocol and network.

Current status:

JSON streaming is in place, and graceful shutdown.
~~It crawls with all kinds of nice parameters but stores everything in memory until dumping a giant JSON blob at the end.~~
~~It crawls from hard-coded values and spits a bunch of stuff to stdout.~~

Usage

$ go get github.com/shazow/btc-crawl
$ btc-crawl --help
...
$ btc-crawl \
  --concurrency=100 \
  --output="btc-crawl.json" \
  --peer-age="24h" \
  --user-agent="/batman:1.0/" \
  --verbose
...

Estimated crawl time: Unknown.

There should be under 10,000 active network nodes at any given time, according to bitnodes.io. Each node returns around ~2,500 known nodes, but usually only several have timestamps within the last hour.

Todo

(In approximate order of priority)

Namespace useful sub-packages properly (outside of main)
Fix go build -race race condition warnings. (Not sure if this is feasible. golog and other fundamental pieces seem to trigger warnings possibly erroneously.)
Tests would be nice.

License

MIT (see LICENSE file).