Commit Graph

303 Commits

Author SHA1 Message Date
Péter Szilágyi 16a23ff740 Merge pull request #2637 from karalabe/downloader-always-cancel
eth/downloader: ensure cancel channel is closed post sync
2016-05-31 14:48:52 +03:00
Péter Szilágyi 4496a44f68 eth/downloader: ensure cancel channel is closed post sync 2016-05-30 12:01:50 +03:00
Péter Szilágyi 4f1d92b332 eth/downloader, trie: pull head state concurrently with chain 2016-05-27 14:26:00 +03:00
Péter Szilágyi 8906b2fe09 eth/downloader: fix reviewer comments 2016-05-20 10:56:01 +03:00
Péter Szilágyi e86619e75d eth/downloader: stream partial skeleton filling to processor 2016-05-17 10:03:34 +03:00
Péter Szilágyi b40dc8a1da eth/downloader: implement concurrent header downloads 2016-05-17 10:03:34 +03:00
Péter Szilágyi 39ce85cf5d eth/downloader: bound fork ancestry and allow heavy short forks 2016-05-13 12:12:13 +02:00
Felix Lange 6fdd0893c3 all: fix go vet warnings 2016-04-15 11:17:27 +02:00
Leif Jurvetson 434e4b31d8 core, eth: replace reorganiz with reorganis 2016-03-15 12:03:17 -07:00
Leif Jurvetson bbbe2360d0 eth: various typos 2016-03-15 11:27:49 -07:00
Péter Szilágyi 64ee5763ee eth/downloader: fix premature exit before notifying all part fetchers 2016-02-29 14:22:28 +02:00
Felix Lange 26e72b2ccd eth/downloader: fix header download limiting
Fixes #2201
2016-02-26 15:24:51 +01:00
Péter Szilágyi 216729009b eth/downloader: fix partial rollback and ancestor lookup 2016-02-19 11:17:02 +02:00
Péter Szilágyi 59cd60b266 eth, eth/downloader, jsre: surface state sync progress through the API 2016-02-10 14:30:41 +02:00
Péter Szilágyi f1ec226d80 eth/downloader: raise pending state limit that prevented concurrency 2016-02-08 14:57:02 +02:00
Péter Szilágyi 9e011ff1cd core, eth/downloader: ensure state presence in ancestor lookup 2016-01-04 16:27:23 +02:00
Péter Szilágyi b6f5523bdc eth/downloader: fetch data proportionally to peer capacity 2015-11-19 17:01:39 +02:00
Felix Lange b7b62d4b3c eth/downloader: also drain stateCh, receiptCh in eth/61 mode
State and receipt deliveries from a previous eth/62+ sync can hang if
the downloader has moved on to syncing with eth/61. Fix this by also
draining the eth/63 channels while waiting for eth/61 data.

A nicer solution would be to take care of the channels in a central
place, but that would involve a major rewrite.
2015-11-19 14:18:35 +01:00
Felix Lange 900da3d800 eth/downloader: don't hang for spurious deliveries
Unexpected deliveries could block indefinitely if they arrived at the
right time. The fix is to ensure that the cancellation channel is
always closed when the sync ends, unblocking any deliveries. Also remove
the atomic check for whether a sync is currently running because it
doesn't help and can be misleading.

Cancelling always seems to break the tests though. The downloader
spawned d.process whenever new data arrived, making it somewhat hard to
track when block processing was actually done. Fix this by running
d.process in a dedicated goroutine that is tied to the lifecycle of the
sync. d.process gets notified of new work by the queue instead of being
invoked all the time. This removes a ton of weird workaround code,
including a hairy use of atomic CAS.
2015-11-19 14:18:34 +01:00
Péter Szilágyi 2019ed71b4 eth: don't block sync goroutines that short circuit 2015-10-28 16:41:01 +02:00
Péter Szilágyi 5b0ee8ec30 core, eth, trie: fix data races and merge/review issues 2015-10-21 16:49:55 +03:00
Péter Szilágyi a9d8dfc8e7 core, eth: roll back uncertain headers in failed fast syncs 2015-10-19 10:03:10 +03:00
Péter Szilágyi b97e34a8e4 eth/downloader: concurrent receipt and state processing 2015-10-19 10:03:10 +03:00
Péter Szilágyi ab27bee25a core, eth, trie: direct state trie synchronization 2015-10-19 10:03:09 +03:00
Péter Szilágyi 832b37c822 core, eth: receipt chain reconstruction 2015-10-19 10:03:09 +03:00
Péter Szilágyi f186b39018 eth/downloader: add fast and light sync strategies 2015-10-19 10:03:09 +03:00
Péter Szilágyi 47f62a67aa eth/downloader: match capabilities when querying idle peers 2015-10-02 13:20:41 +03:00
Péter Szilágyi f459a3f0ae eth/downloader: always send termination wakes, clean leftover 2015-09-23 12:39:17 +03:00
Péter Szilágyi 99b62f36b6 eth/downloader: header-chain order and ancestry check 2015-09-15 14:45:53 +03:00
Péter Szilágyi 0a7d059b6a eth, rpc: standardize the chain sync progress counters 2015-09-15 14:45:53 +03:00
Péter Szilágyi cdc2662c40 core: split out TD from database and all internals 2015-09-11 17:42:25 +03:00
Péter Szilágyi 1f1d73ab74 eth/downloader: fix race causing occasional test failure 2015-09-01 16:11:14 +03:00
Péter Szilágyi 17f65cd1e5 eth: update metrics collection to handle eth/62 algos 2015-08-25 17:48:47 +03:00
Péter Szilágyi 47a7fe5d22 eth: port the synchronisation algo to eth/62 2015-08-25 17:48:47 +03:00
Péter Szilágyi ca88e18f59 eth: kill off protocol eth/60 in preparation for eth/62 2015-08-24 17:57:28 +03:00
Péter Szilágyi 42f44dda54 eth, eth/downloader: handle header requests, table driven proto tests 2015-08-24 17:57:28 +03:00
Péter Szilágyi c51e153b5c eth, metrics, p2p: prepare metrics and net packets to eth/62 2015-08-21 10:30:57 +03:00
Péter Szilágyi 6adbaabc65 eth, eth/downloader: don't report stall if fetcher filled the block 2015-07-29 15:39:08 +03:00
Felix Lange bfbcfbe4a9 all: fix license headers one more time
I forgot to update one instance of "go-ethereum" in commit 3f047be5a.
2015-07-23 18:35:11 +02:00
Felix Lange 3f047be5aa all: update license headers to distiguish GPL/LGPL
All code outside of cmd/ is licensed as LGPL. The headers
now reflect this by calling the whole work "the go-ethereum library".
2015-07-22 18:51:45 +02:00
Péter Szilágyi 492d5454b1 eth/downloader: drop peer if advertised TD but won't delvier 2015-07-09 14:40:18 +03:00
Felix Lange ea54283b30 all: update license information 2015-07-07 14:12:44 +02:00
Péter Szilágyi d6f2c0a76f eth, eth/downloader: fix #1231, DOS vulnerability in hash queueing 2015-07-01 15:21:35 +03:00
Péter Szilágyi 60454da650 eth/downloader: reduce hash fetches in prep for eth/61 2015-07-01 01:20:49 +03:00
Péter Szilágyi f43c07cb3c eth, eth/downloader: transition to eth 61 2015-06-30 19:05:06 +03:00
Péter Szilágyi af51dc4d63 eth, eth/downloader: pass the eth protocol version through 2015-06-30 19:00:01 +03:00
Péter Szilágyi 5ec6ecc511 eth, eth/fetcher: move propagated block import into fetcher 2015-06-18 15:56:08 +03:00
Péter Szilágyi 7c2af1c117 eth, eth/fetcher: separate notification sync mechanism 2015-06-18 15:56:07 +03:00
Péter Szilágyi 55dd8fd621 eth/downloader: always reenter processing if not exiting 2015-06-18 00:26:54 +03:00
Péter Szilágyi 2f4cbe22f5 eth, eth/downloader: fix processing interrupt caused by temp cancel 2015-06-18 00:04:57 +03:00
Péter Szilágyi 4a1e82cf3f eth/downloader: fix #1280, overlapping (good/bad) delivery hang 2015-06-17 12:03:16 +03:00
Péter Szilágyi cf7c44a7f6 eth/downloader: detailed comment for the race corner case 2015-06-15 15:18:04 +03:00
Péter Szilágyi 9c03c374e3 eth/downloader: fix import statistic reset, fetch hashes async 2015-06-15 13:05:01 +03:00
Péter Szilágyi fc7abd9886 eth, eth/downloader: move block processing into the downlaoder 2015-06-15 09:22:37 +03:00
Péter Szilágyi 80833f8137 eth/downloader: instreument and test the sync peer drop 2015-06-15 09:22:37 +03:00
Péter Szilágyi 66d3dc8690 eth, eth/downloader: move peer removal into downloader 2015-06-15 09:22:36 +03:00
Péter Szilágyi 3c1cccc801 eth/downloader: fetch the block hashes on the fly, when needed 2015-06-10 20:12:22 +03:00
Péter Szilágyi 271fb20ecb cmd/geth, eth/downloader: rough guess at the import eta 2015-06-10 18:01:05 +03:00
Péter Szilágyi b3d5ce7d48 cmd/geth, eth/downloader: collect and report import progress too 2015-06-10 01:20:35 +03:00
Felix Lange 8dc3048f65 eth/downloader: fix hash fetch timeout handling
Fixes #1206
2015-06-09 17:07:10 +02:00
Péter Szilágyi c4f224932f eth/downloader: reject peer registration if head is banned 2015-06-08 14:46:31 +03:00
Péter Szilágyi 63c6cedb14 eth/downloader: cap the hash ban set, add test for it 2015-06-08 14:12:00 +03:00
Péter Szilágyi b40c796ff7 eth/downloader: preallocate the block cache 2015-06-08 13:23:58 +03:00
Péter Szilágyi 1d7bf3d39f eth/downloader: fix merge compile error 2015-06-08 13:23:58 +03:00
Péter Szilágyi 6d497f61c6 eth/downloader: don't block hash deliveries while pulling blocks 2015-06-08 13:23:58 +03:00
Péter Szilágyi 0275fcb3d3 eth/downloader: clean up and simplify the code a bit 2015-06-08 13:23:58 +03:00
Péter Szilágyi abdfcda4dd eth/downloader: short circuit sync if head hash is banned 2015-06-08 13:23:58 +03:00
Péter Szilágyi 84bc93d8cb eth/downloader: accumulating hash bans for reconnecting attackers 2015-06-08 13:23:58 +03:00
Péter Szilágyi eedb25b22a eth/downloader: clean up tests and unused variables 2015-06-08 13:23:57 +03:00
Jeffrey Wilcke b94a76d17e Merge pull request #1189 from karalabe/downloader-polishes
eth/downloader: handle timeouts more gracefully
2015-06-05 08:31:57 -07:00
Péter Szilágyi 94e525ae12 eth, eth/downloader: fix #1098, elevate empty hash errors to peer drops 2015-06-05 12:52:48 +03:00
Péter Szilágyi 328ef60b85 eth/downloader: differentiate stale and nonexistent deliveries 2015-06-05 12:37:48 +03:00
Péter Szilágyi 94e4aa6ea9 eth/downloader: log hard timeouts and reset capacity 2015-06-05 11:53:46 +03:00
Péter Szilágyi 24cca2f18d eth/downloader: log after state updates, easier to debug 2015-06-04 15:10:43 +03:00
Péter Szilágyi 28c32d1b1b eth/downloader: fix #1178, don't request blocks beyond the cache bounds 2015-06-04 14:51:14 +03:00
Péter Szilágyi c9a546c310 eth/downloader: add a basic block download congestion control 2015-06-03 14:40:11 +03:00
Péter Szilágyi 29b0480cfb core, eth/downloader: expose the bad hashes, check in downloader 2015-05-28 14:03:10 +03:00
Péter Szilágyi 3083ec5e32 eth/downloader: silence "Added N blocks from..." if N == 0 2015-05-26 16:10:28 +03:00
Péter Szilágyi eafdc1f8e3 eth, eth/downloader: surface downloaded block origin, drop on error 2015-05-26 14:00:21 +03:00
Péter Szilágyi 06a041589f eth, eth/downloader: remove duplicate consts, bump hash fetch to 2K 2015-05-21 18:16:04 +03:00
Péter Szilágyi 52db6d8be5 eth/downloader: circumvent a forged block chain with known parent attack 2015-05-21 08:37:27 +03:00
Péter Szilágyi e8b22b9253 eth/downloader: prevent a peer from dripping bad hashes 2015-05-21 08:07:58 +03:00
obscuren c2ef8682fe eth/downloader: moved start event
Start event has moved because it could possibly could stall the miner
2015-05-16 13:02:30 +02:00
Péter Szilágyi f3ae8f50a5 eth/downloader: circumvent download race between crosscheck and hashes 2015-05-15 20:54:10 +03:00
Péter Szilágyi 4f0d88cb02 eth/downloader: fix cancel channel double close 2015-05-15 19:43:42 +03:00
Péter Szilágyi 5c1a7b965c eth/downloader: circumvent a fake blockchain attack 2015-05-15 15:04:15 +03:00
Péter Szilágyi b517967f86 eth/downloader: don't penalize for stale cross checks 2015-05-15 15:03:08 +03:00
Péter Szilágyi 83226762c2 eth, eth/downloader: detect and handle madeup hash attacks 2015-05-15 15:01:58 +03:00
Péter Szilágyi 72411eb24c eth/downloader: circumvent hash reordering attacks 2015-05-15 15:01:58 +03:00
Péter Szilágyi bcc2980179 eth/downloader: check sync after failed attacks 2015-05-15 15:01:58 +03:00
Péter Szilágyi 366e9627e8 eth/downloader: add a test for detecting missing blocks 2015-05-15 15:01:58 +03:00
Péter Szilágyi cd2fb09051 eth, eth/downloader: prevent hash repeater attack 2015-05-15 15:01:58 +03:00
obscuren b71091e337 eth, eth/downloader, miner: use download events to check miner start 2015-05-15 00:43:10 +02:00
Jeffrey Wilcke 060a07cf69 Merge pull request #974 from karalabe/downloader-fix-unknown-parent-attack
eth, eth/downloader: handle a potential unknown parent attack
2015-05-14 08:16:30 -07:00
Péter Szilágyi ebf1eb9359 eth/downloader: remove a previous leftover 2015-05-14 15:40:28 +03:00
Péter Szilágyi 3eda70c64c eth, eth/downloader: remove parent verification from the downlaoder 2015-05-14 15:38:49 +03:00
Péter Szilágyi a4246c2da6 eth, eth/downloader: handle a potential unknown parent attack 2015-05-14 15:24:18 +03:00
Jeffrey Wilcke 2b06fe3eff Merge pull request #950 from karalabe/downloader-cancel-sync
eth/downloader: fix deliveries to check for sync cancels
2015-05-14 04:05:13 -07:00
Péter Szilágyi 910a6d4e46 eth/downloader: fix nil panic caused by wrong variable use 2015-05-13 17:25:01 +03:00
Péter Szilágyi de3a71cafd eth/downloader: remove a redundant sync progress check 2015-05-13 16:56:01 +03:00
Péter Szilágyi 48ee0777a5 eth/downloader: add a user sync notificaton 2015-05-13 16:03:05 +03:00
Péter Szilágyi ec57aa64cd eth/downloader: sync the cancel channel during cancel too 2015-05-13 14:01:08 +03:00
Péter Szilágyi ee0c892303 eth/downloader: fix deliveries to check for sync cancels 2015-05-13 13:47:21 +03:00
Péter Szilágyi d2d5dbc6fb eth/downloader: fix active peer shadowing, polish func names 2015-05-13 13:13:43 +03:00
obscuren 48bd48876c eth, eth/downloader: moved pending queue error message to debug 2015-05-11 17:27:34 +02:00
Péter Szilágyi 064cf16099 eth/downloader: use count instead of peers, clearer 2015-05-11 17:21:22 +02:00
Péter Szilágyi d37a2559b9 eth/downloader: revert to demotion, use harsher penalty 2015-05-11 17:21:22 +02:00
Péter Szilágyi 70c65835f4 eth/downloader: fix #910, thread safe peers & polishes 2015-05-11 17:21:22 +02:00
obscuren 05715f27cf eth: added a cancel method for the downloader
Added a cancel method to the downloader which gracefully shuts down any
active syncing process (hash fetching or block downloading) and resets
the queue and remove any pending blocks.

Issue with the downloader which would stall because of an active ongoing
process when an invalid block was found.
2015-05-10 00:34:07 +02:00
Péter Szilágyi edad47bf0e eth/downloader: fix leftover state between syncs 2015-05-08 17:39:03 +03:00
Péter Szilágyi bd5720f480 eth, eth/downloader: handle sync errors a bit more gracefully 2015-05-08 15:22:48 +03:00
Péter Szilágyi 9d188f73b5 eth, eth/downloader: make synchronize thread safe 2015-05-07 21:07:20 +03:00
Péter Szilágyi 45f8304f3c eth/downloader: fix expiration not running while fetching 2015-05-07 12:59:19 +03:00
Péter Szilágyi 4800c94392 eth/downloader: prioritize block fetch based on chain position, cap memory use 2015-05-07 10:57:34 +03:00
obscuren 1470b22e90 downloader: hash downloading recovery
If a peer fails to respond (disconnect, etc) during hash downloading
switch to a different peer which has it's current_hash in the queue's
peer set.
2015-05-03 16:27:03 +02:00
obscuren 493181ea09 eth/downloader: changed has to blockHashes for chain linking 2015-05-03 14:10:31 +02:00
obscuren 016f152b36 eth, eth/downloader: Moved block processing & graceful shutdown
The downloader is no longer responsible for processing blocks. The
eth-protocol handler now takes care of this instead.

Added graceful shutdown during block processing. Closes #846
2015-05-01 15:58:44 +02:00
obscuren 8595198c1b eth/downloader: delete blocks from queue 2015-04-30 17:51:47 +02:00
obscuren 9b6e8f6195 eth, eth/downloader: remove bad peers from peer set
Peers in the eth protocol handler are now being ignored for catch up.
2015-04-30 12:38:16 +02:00
obscuren f8c27d7159 eth/downloader: drop block 2015-04-29 19:55:04 +02:00
obscuren dfbf580354 eth/downloader: ignore orphan blocks in the downloader.
When blocks have been sorted and are being processed, orphan blocks
should be ignored and thrown out. The protocol handler is responsible
for downloading blocks which have missing parents.
2015-04-29 14:49:37 +02:00
obscuren 735b029db9 core: return the index of the block that failed when inserting a chain 2015-04-29 14:00:24 +02:00
obscuren 13364a2dcf eth/downloader: reset the chain when parent is missing during processing 2015-04-29 13:40:39 +02:00
obscuren 4b7bdc3766 eth: check if downloader is busy before showing log message 2015-04-29 12:51:05 +02:00
obscuren 7ce3d06402 eth/downloader: removed update loop and synch channel 2015-04-24 18:28:46 +02:00
obscuren d84c2202e7 eth, eth/downloader: simplified synchronisation process 2015-04-24 15:38:10 +02:00
obscuren 31f82eb334 eth, eth/downloader: don't require td on downloader. Fixed tests 2015-04-24 15:04:58 +02:00
obscuren b86e7526e1 eth, eth/downloader: moved peer selection to protocol handler 2015-04-24 14:40:32 +02:00
obscuren 45da3e17e2 core: added chain head reset to known block 2015-04-20 12:29:02 +02:00
obscuren 5dd56bb474 geth: admin download status 2015-04-19 21:46:19 +02:00
obscuren ed07ffcde5 downloader: fixed tests with low ttl 2015-04-19 17:37:01 +02:00
obscuren dcf1a1988a moved 2015-04-19 17:14:15 +02:00
obscuren cce4158cfc Removed debug println 2015-04-19 13:33:14 +02:00
obscuren c58918c84a downloader: moved chunk ignoring. Fixes issue with catching up 2015-04-19 13:30:34 +02:00
obscuren 5c59d95532 downloader: defer peer reset after download 2015-04-19 01:39:37 +02:00
obscuren 86ecdcd5ff downloader: reset hashTtl on receive 2015-04-19 00:15:05 +02:00
obscuren 50e096e627 downloader: don't remove peers. keep them around 2015-04-18 23:56:08 +02:00
obscuren 6830ddb659 downloader: free up peers from work when the downloader resets 2015-04-18 20:25:55 +02:00
obscuren 7c5d50f627 downloader: throw an error if there are no peers available for download
If all peers have been tried during the block download process and some
hashes are unfetchable (available peers > 0 and fetching == 0) throw an
error so the process can be aborted.
2015-04-18 19:30:29 +02:00
obscuren 78e37e98e7 downloader: fixed a race condition for download status 2015-04-18 19:14:25 +02:00
obscuren 0d536734fe eth: adapted to new synchronous api of downloader's AddBlock 2015-04-18 18:57:59 +02:00
obscuren c2c24b3bb4 downloader: improved downloading and synchronisation
* Downloader's peers keeps track of peer's previously requested hashes
  so that we don't have to re-request
* Changed `AddBlock` to be fully synchronous
2015-04-18 18:55:13 +02:00
obscuren 60613b57d1 downloader: make sure that hashes are only accepted from the active peer 2015-04-18 17:35:03 +02:00
obscuren 8244825bbf downloader: reset the queue if a peer response with an empty hash set 2015-04-18 15:14:12 +02:00
obscuren 8f873b762b downloader: all handlers check for isBusy 2015-04-18 03:15:26 +02:00
obscuren 2339ee9910 Merge branch 'develop' into downloader-proto
Conflicts:
	eth/downloader/downloader.go
2015-04-18 01:10:32 +02:00
obscuren eac2df02d1 downloader: fixed a typo 2015-04-17 00:11:45 +02:00
obscuren 205378016f downloader: added demotion / promotion in prep. for rep. system 2015-04-16 02:16:33 +02:00
obscuren eda10c7317 downloader: updated downloader and fixed issues with catch up
Properly ignore blocks coming from peers not in our peer list (blocked)
and do never request anything from bad peers. Added some checks to
account for blocks known when requesting hashes (missing parents).
2015-04-16 01:29:32 +02:00
obscuren c4678ffd77 downloader: updated downloader and fixed issues with catch up
Properly ignore blocks coming from peers not in our peer list (blocked)
and do never request anything from bad peers. Added some checks to
account for blocks known when requesting hashes (missing parents).
2015-04-16 00:14:31 +02:00
obscuren a8a2b2a488 downloader: added missing blocks catchup functionality
When a parent is missing in the block list an attempt should be made to
fetch the missing parent and grandparents.
2015-04-13 16:38:32 +02:00
obscuren 7dcb9825c3 downloader: return an error for peer.fetch and return chunk to queue
If a peer was somehow already fetching and somehow managed to end up in
the `available` pool it should return it's work.
2015-04-12 13:36:01 +02:00
obscuren acf8452c33 downloader: implemented new downloader 2015-04-12 12:38:25 +02:00