Commit Graph

64 Commits

Author SHA1 Message Date
Péter Szilágyi 55599ee95d core, trie: intermediate mempool between trie and database (#15857)
This commit reduces database I/O by not writing every state trie to disk.
2018-02-05 17:40:32 +01:00
Nick Johnson ae11545bc5 eth, les: Refactor downloader peer to use structs 2017-06-29 12:49:18 +01:00
Felix Lange 0042f13d47 eth/downloader: separate state sync from queue (#14460)
* eth/downloader: separate state sync from queue

Scheduling of state node downloads hogged the downloader queue lock when
new requests were scheduled. This caused timeouts for other requests.
With this change, state sync is fully independent of all other downloads
and doesn't involve the queue at all.

State sync is started and checked on in processContent. This is slightly
awkward because processContent doesn't have a select loop. Instead, the
queue is closed by an auxiliary goroutine when state sync fails. We
tried several alternatives to this but settled on the current approach
because it's the least amount of change overall.

Handling of the pivot block has changed slightly: the queue previously
prevented import of pivot block receipts before the state of the pivot
block was available. In this commit, the receipt will be imported before
the state. This causes an annoyance where the pivot block is committed
as fast block head even when state downloads fail. Stay tuned for more
updates in this area ;)

* eth/downloader: remove cancelTimeout channel

* eth/downloader: retry state requests on timeout

* eth/downloader: improve comment

* eth/downloader: mark peers idle when state sync is done

* eth/downloader: move pivot block splitting to processContent

This change also ensures that pivot block receipts aren't imported
before the pivot block itself.

* eth/downloader: limit state node retries

* eth/downloader: improve state node error handling and retry check

* eth/downloader: remove maxStateNodeRetries

It fails the sync too much.

* eth/downloader: remove last use of cancelCh in statesync.go

Fixes TestDeliverHeadersHang*Fast and (hopefully)
the weird cancellation behaviour at the end of fast sync.

* eth/downloader: fix leak in runStateSync

* eth/downloader: don't run processFullSyncContent in LightSync mode

* eth/downloader: improve comments

* eth/downloader: fix vet, megacheck

* eth/downloader: remove unrequested tasks anyway

* eth/downloader, trie: various polishes around duplicate items

This commit explicitly tracks duplicate and unexpected state
delieveries done against a trie Sync structure, also adding
there to import info logs.

The commit moves the db batch used to commit trie changes one
level deeper so its flushed after every node insertion. This
is needed to avoid a lot of duplicate retrievals caused by
inconsistencies between Sync internals and database. A better
approach is to track not-yet-written states in trie.Sync and
flush on commit, but I'm focuing on correctness first now.

The commit fixes a regression around pivot block fail count.
The counter previously was reset to 1 if and only if a sync
cycle progressed (inserted at least 1 entry to the database).
The current code reset it already if a node was delivered,
which is not stong enough, because unless it ends up written
to disk, an attacker can just loop and attack ad infinitum.

The commit also fixes a regression around state deliveries
and timeouts. The old downloader tracked if a delivery is
stale (none of the deliveries were requestedt), in which
case it didn't mark the node idle and did not send further
requests, since it signals a past timeout. The current code
did mark it idle even on stale deliveries, which eventually
caused two requests to be in flight at the same time, making
the deliveries always stale and mass duplicating retrievals
between multiple peers.

* eth/downloader: fix state request leak

This commit fixes the hang seen sometimes while doing the state
sync. The cause of the hang was a rare combination of events:
request state data from peer, peer drops and reconnects almost
immediately. This caused a new download task to be assigned to
the peer, overwriting the old one still waiting for a timeout,
which in turned leaked the requests out, never to be retried.
The fix is to ensure that a task assignment moves any pending
one back into the retry queue.

The commit also fixes a regression with peer dropping due to
stalls. The current code considered a peer stalling if they
timed out delivering 1 item. However, the downloader never
requests only one, the minimum is 2 (attempt to fine tune
estimated latency/bandwidth). The fix is simply to drop if
a timeout is detected at 2 items.

Apart from the above bugfixes, the commit contains some code
polishes I made while debugging the hang.

* core, eth, trie: support batched trie sync db writes

* trie: rename SyncMemCache to syncMemBatch
2017-06-22 15:26:03 +03:00
Péter Szilágyi 2f28a12cdb
common, eth/downloader, log: support terminal log formatting 2017-02-27 19:15:18 +02:00
Péter Szilágyi 0a63c3e362
eth/downloader: port over old logs from glog to log15 2017-02-27 13:16:40 +02:00
Péter Szilágyi d4fd06c3dc
all: blidly swap out glog to our log15, logs need rework 2017-02-23 12:16:44 +02:00
Felix Lange f1069a30b9 eth/downloader: improve deliverNodeData (#3588)
Commit d3b751e accidentally deleted a crucial 'return' statement,
leading to a crash in case of an issue with node data. This change
improves the fix in PR #3591 by removing the lock entirely.
2017-01-24 13:20:37 +01:00
Péter Szilágyi ebc3d232f4 eth/downloader: fix mutex regression causing panics on fail (#3591) 2017-01-20 01:12:14 +01:00
Felix Lange 35a7dcb162 all: gofmt -w -s 2017-01-06 15:52:03 +01:00
Felix Lange d3b751e4d9 trie: remove dependency on ethdb
This removes the core/types -> leveldb dependency.
2017-01-06 14:15:22 +01:00
Péter Szilágyi e1b4acfb6e
eth/downloader: clear pending requests when switching trie root 2016-11-01 12:53:33 +02:00
Péter Szilágyi 90b16a3e85
core/state, eth/downloader, trie: reset fast-failure on progress 2016-10-31 14:19:14 +02:00
Felix Lange 016007bd25 eth, eth/downloader, eth/fetcher: delete eth/61 code
The eth/61 protocol was disabled in #2776, this commit removes its
message handlers and hash-chain sync logic.
2016-07-22 13:17:19 +02:00
Péter Szilágyi 4f1d92b332 eth/downloader, trie: pull head state concurrently with chain 2016-05-27 14:26:00 +03:00
Péter Szilágyi 8906b2fe09 eth/downloader: fix reviewer comments 2016-05-20 10:56:01 +03:00
Péter Szilágyi e86619e75d eth/downloader: stream partial skeleton filling to processor 2016-05-17 10:03:34 +03:00
Péter Szilágyi b40dc8a1da eth/downloader: implement concurrent header downloads 2016-05-17 10:03:34 +03:00
Felix Lange 6fdd0893c3 all: fix go vet warnings 2016-04-15 11:17:27 +02:00
Leif Jurvetson bbbe2360d0 eth: various typos 2016-03-15 11:27:49 -07:00
Péter Szilágyi 66b148dd8f Merge pull request #2251 from karalabe/increase-downlaoder-queue
eth/downloader: bump the download queue size to prevent starvation
2016-02-24 15:55:21 +02:00
Péter Szilágyi 93c0012000 eth/downloader: bump the download queue size to prevent starvation 2016-02-23 13:12:11 +02:00
Ricardo Catalinas Jiménez 436fc8d76a all: Rename crypto.Sha3{,Hash}() to crypto.Keccak256{,Hash}()
As we aren't really using the standarized SHA-3
2016-02-21 22:34:34 +00:00
Péter Szilágyi f1ec226d80 eth/downloader: raise pending state limit that prevented concurrency 2016-02-08 14:57:02 +02:00
Péter Szilágyi b6f5523bdc eth/downloader: fetch data proportionally to peer capacity 2015-11-19 17:01:39 +02:00
Felix Lange 900da3d800 eth/downloader: don't hang for spurious deliveries
Unexpected deliveries could block indefinitely if they arrived at the
right time. The fix is to ensure that the cancellation channel is
always closed when the sync ends, unblocking any deliveries. Also remove
the atomic check for whether a sync is currently running because it
doesn't help and can be misleading.

Cancelling always seems to break the tests though. The downloader
spawned d.process whenever new data arrived, making it somewhat hard to
track when block processing was actually done. Fix this by running
d.process in a dedicated goroutine that is tied to the lifecycle of the
sync. d.process gets notified of new work by the queue instead of being
invoked all the time. This removes a ton of weird workaround code,
including a hairy use of atomic CAS.
2015-11-19 14:18:34 +01:00
Péter Szilágyi b658a73ed5 eth/downloader: fix dysfunctional ignore list hidden by generic set 2015-11-04 13:11:52 +02:00
Péter Szilágyi 5b0ee8ec30 core, eth, trie: fix data races and merge/review issues 2015-10-21 16:49:55 +03:00
Péter Szilágyi aa0538db0b eth: clean out light node notions from eth 2015-10-19 10:03:10 +03:00
Péter Szilágyi b97e34a8e4 eth/downloader: concurrent receipt and state processing 2015-10-19 10:03:10 +03:00
Péter Szilágyi ab27bee25a core, eth, trie: direct state trie synchronization 2015-10-19 10:03:09 +03:00
Péter Szilágyi f186b39018 eth/downloader: add fast and light sync strategies 2015-10-19 10:03:09 +03:00
Péter Szilágyi 99b62f36b6 eth/downloader: header-chain order and ancestry check 2015-09-15 14:45:53 +03:00
Péter Szilágyi 17f65cd1e5 eth: update metrics collection to handle eth/62 algos 2015-08-25 17:48:47 +03:00
Péter Szilágyi 47a7fe5d22 eth: port the synchronisation algo to eth/62 2015-08-25 17:48:47 +03:00
Felix Lange bfbcfbe4a9 all: fix license headers one more time
I forgot to update one instance of "go-ethereum" in commit 3f047be5a.
2015-07-23 18:35:11 +02:00
Felix Lange 3f047be5aa all: update license headers to distiguish GPL/LGPL
All code outside of cmd/ is licensed as LGPL. The headers
now reflect this by calling the whole work "the go-ethereum library".
2015-07-22 18:51:45 +02:00
Felix Lange ea54283b30 all: update license information 2015-07-07 14:12:44 +02:00
Péter Szilágyi f43c07cb3c eth, eth/downloader: transition to eth 61 2015-06-30 19:05:06 +03:00
Péter Szilágyi 66d3dc8690 eth, eth/downloader: move peer removal into downloader 2015-06-15 09:22:36 +03:00
Péter Szilágyi 63c6cedb14 eth/downloader: cap the hash ban set, add test for it 2015-06-08 14:12:00 +03:00
Péter Szilágyi 2d627995cf eth/downloader: fix another rebase error 2015-06-08 13:23:58 +03:00
Péter Szilágyi b40c796ff7 eth/downloader: preallocate the block cache 2015-06-08 13:23:58 +03:00
Péter Szilágyi 328ef60b85 eth/downloader: differentiate stale and nonexistent deliveries 2015-06-05 12:37:48 +03:00
Péter Szilágyi 28c32d1b1b eth/downloader: fix #1178, don't request blocks beyond the cache bounds 2015-06-04 14:51:14 +03:00
Péter Szilágyi c9a546c310 eth/downloader: add a basic block download congestion control 2015-06-03 14:40:11 +03:00
Péter Szilágyi eafdc1f8e3 eth, eth/downloader: surface downloaded block origin, drop on error 2015-05-26 14:00:21 +03:00
Péter Szilágyi 06a041589f eth, eth/downloader: remove duplicate consts, bump hash fetch to 2K 2015-05-21 18:16:04 +03:00
Péter Szilágyi 83226762c2 eth, eth/downloader: detect and handle madeup hash attacks 2015-05-15 15:01:58 +03:00
Péter Szilágyi 72411eb24c eth/downloader: circumvent hash reordering attacks 2015-05-15 15:01:58 +03:00
Péter Szilágyi cd2fb09051 eth, eth/downloader: prevent hash repeater attack 2015-05-15 15:01:58 +03:00