456b187892
trie, eth/protocols/snap: less tiny db-writes during trie-healing ( #25681 )
...
This PR prevent making tiny writes during state healing, by only performing the batch-write if the accumulated data is large enough to be meaningful.
2022-09-28 08:08:18 +02:00
fb500d12d5
eth/protocols/snap: make log messages more clear that sync is ongoing ( #25837 )
...
* eth/protocols/snap: make log messages more clear that sync is ongoing
* Update sync.go
Co-authored-by: Felix Lange <fjl@twurst.com>
2022-09-23 11:11:41 +03:00
de8d5fa042
eth/protocols/snap: throttle trie heal requests when peers DoS us ( #25666 )
...
* eth/protocols/snap: throttle trie heal requests when peers DoS us
* eth/protocols/snap: lower heal throttle log to debug
Co-authored-by: Martin Holst Swende <martin@swende.se>
* eth/protocols/snap: fix comment
Co-authored-by: Martin Holst Swende <martin@swende.se>
2022-09-09 11:42:57 +03:00
dafa40e7a7
eth/protocols/snap: fix problems due to idle-but-busy peers
2022-08-31 17:58:18 +02:00
2c5648d891
all: fix some typos ( #25551 )
...
* Fix some typos
* Fix some mistakes
* Revert 4byte.json
* Fix an incorrect fix
* Change files to fails
2022-08-19 09:00:21 +03:00
1657e43931
core, les, eth: port snap sync changes ( #24898 )
...
core, eth, les, trie: rework snap sync
2022-07-15 14:55:51 +03:00
a907d7e81a
all: more linters ( #24783 )
...
This enables the following linters
- typecheck
- unused
- staticcheck
- bidichk
- durationcheck
- exportloopref
- gosec
WIth a few exceptions.
- We use a deprecated protobuf in trezor. I didn't want to mess with that, since I cannot meaningfully test any changes there.
- The deprecated TypeMux is used in a few places still, so the warning for it is silenced for now.
- Using string type in context.WithValue is apparently wrong, one should use a custom type, to prevent collisions between different places in the hierarchy of callers. That should be fixed at some point, but may require some attention.
- The warnings for using weak random generator are squashed, since we use a lot of random without need for cryptographic guarantees.
2022-06-13 16:24:45 +02:00
22defa5af7
all: introduce trie owner notion ( #24750 )
...
* cmd, core/state, light, trie, eth: add trie owner notion
* all: refactor
* tests: fix goimports
* core/state/snapshot: fix ineffasigns
Co-authored-by: Martin Holst Swende <martin@swende.se>
2022-06-06 17:14:55 +02:00
646503208e
eth/protocols/snap: sort trienode heal requests by path ( #24779 )
...
* sort snap trienode heal requests
* eth/protocols/snap: remove debug code
* eth/protocols/snap: simplify sort, generate pathsets later
* eth/protocols/snap: review concern
* eth/protocols/snap: renamings
* eth/protocols/snap: add comments in Merge
* eth/protocols/snap: remove variable 'last' in Merge
* eth/protocols/snap: fix lint flaws in test
Co-authored-by: Felix Lange <fjl@twurst.com>
2022-05-10 17:37:24 +03:00
86d5477079
core/state/snapshot: fix race condition ( #24685 )
...
Fixes three race conditions found through fuzzing by David Theodore
2022-05-06 17:20:41 +02:00
1b58e42802
Merge pull request #24529 from holiman/fix_inf
...
eth/protocols/snap: avoid estimating infinite percentage
2022-03-11 10:40:55 +02:00
7d3ecca451
eth/protocols/snap: a little typo Merkel -> Merkle ( #24530 )
2022-03-11 10:32:08 +02:00
658415960e
eth/protocols/snap: avoid estimating infinite percentage
2022-03-10 12:46:48 +01:00
893502e561
trie, core, eth: use db.has over db.get where possible
2021-12-15 16:16:45 +01:00
58d1988349
core, eth, les, trie: remove the sync bloom, used by fast sync
2021-12-03 12:32:41 +02:00
c10a0a62c3
eth: request id dispatcher and direct req/reply APIs ( #23576 )
...
* eth: request ID based message dispatcher
* eth: fix dispatcher cancellation, rework fetchers idleness tracker
* eth/downloader: drop peers who refuse to serve advertised chains
2021-11-26 13:26:03 +02:00
443afc975c
core/state: move state account to core/types + abstracted "write account to trie" ( #23567 )
...
* core/state: abstracted "write account to trie" method
* fix appveyor build
* Apply suggestions from code review
Co-authored-by: Martin Holst Swende <martin@swende.se>
* review feedback
* core/state/accounts: move Account to core/types
* core/types: rename Account -> StateAccount
* core/state: restore EncodeRLP for stateObject
* core/types: add the missing file
* more review feedback
* more review feedback
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
2021-09-28 10:48:07 +02:00
427175153c
p2p/msgrate: return capacity as integer, clamp to max uint32 ( #22943 )
...
* p2p/msgrate: return capacity as integer
* eth/protocols/snap: remove conversions
* p2p/msgrate: add overflow test
* p2p/msgrate: make the capacity overflow test actually overflow
* p2p/msgrate: clamp capacity to max int32
* p2p/msgrate: fix min/max confusion
2021-05-27 19:43:55 +03:00
3e795881ea
eth, p2p/msgrate: move peer QoS tracking to its own package and use it for snap ( #22876 )
...
This change extracts the peer QoS tracking logic from eth/downloader, moving
it into the new package p2p/msgrate. The job of msgrate.Tracker is determining
suitable timeout values and request sizes per peer.
The snap sync scheduler now uses msgrate.Tracker instead of the hard-coded 15s
timeout. This should make the sync work better on network links with high latency.
2021-05-19 14:09:03 +02:00
52b5d2d869
eth/protocols/snap: use storage batch, not account batch in st task
2021-04-30 18:24:34 +03:00
745757ac6b
core, eth: abort snapshot generation on snap sync and resume later
2021-04-30 17:03:10 +03:00
fae165a5de
core, eth, ethdb, trie: simplify range proofs
2021-04-29 10:59:08 +03:00
a81cf0d2b3
trie: remove redundant returns + use stacktrie where applicable ( #22760 )
...
* trie: add benchmark for proofless range
* trie: remove unused returns + use stacktrie
2021-04-28 22:47:48 +03:00
558bff4008
eth/protocols/snap: lower the packet size to avoid overloading link
2021-04-28 21:40:06 +03:00
caea6c4661
eth/protocols/snap: generate storage trie from full dirty snap data ( #22668 )
...
* eth/protocols/snap: generate storage trie from full dirty snap data
* eth/protocols/snap: get rid of some more dead code
* eth/protocols/snap: less frequent logs, also log during trie generation
* eth/protocols/snap: implement dirty account range stack-hashing
* eth/protocols/snap: don't loop on account trie generation
* eth/protocols/snap: fix account format in trie
* core, eth, ethdb: glue snap packets together, but not chunks
* eth/protocols/snap: print completion log for snap phase
* eth/protocols/snap: extended tests
* eth/protocols/snap: make testcase pass
* eth/protocols/snap: fix account stacktrie commit without defer
* ethdb: fix key counts on reset
* eth/protocols: fix typos
* eth/protocols/snap: make better use of delivered data (#44 )
* eth/protocols/snap: make better use of delivered data
* squashme
* eth/protocols/snap: reduce chunking
* squashme
* eth/protocols/snap: reduce chunking further
* eth/protocols/snap: break out hash range calculations
* eth/protocols/snap: use sort.Search instead of looping
* eth/protocols/snap: prevent crash on storage response with no keys
* eth/protocols/snap: nitpicks all around
* eth/protocols/snap: clear heal need on 1-chunk storage completion
* eth/protocols/snap: fix range chunker, add tests
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* trie: fix test API error
* eth/protocols/snap: fix some further liter issues
* eth/protocols/snap: fix accidental batch reuse
Co-authored-by: Martin Holst Swende <martin@swende.se>
2021-04-27 17:19:59 +03:00
9553c98de8
eth/protocols/snap: use ephemeral channels to avoid cross-sync delveries
2021-04-15 21:16:54 +03:00
1e207342b5
all: make logs a bit easier on the eye to digest ( #22665 )
...
* all: add thousandths separators for big numbers on log messages
* p2p/sentry: drop accidental file
* common, log: add fast number formatter
* common, eth/protocols/snap: simplifty fancy num types
* log: handle nil big ints
2021-04-15 20:35:00 +03:00
7088f1e814
core, eth: faster snapshot generation ( #22504 )
...
* eth/protocols: persist received state segments
* core: initial implementation
* core/state/snapshot: add tests
* core, eth: updates
* eth/protocols/snapshot: count flat state size
* core/state: add metrics
* core/state/snapshot: skip unnecessary deletion
* core/state/snapshot: rename
* core/state/snapshot: use the global batch
* core/state/snapshot: add logs and fix wiping
* core/state/snapshot: fix
* core/state/snapshot: save generation progress even if the batch is empty
* core/state/snapshot: fixes
* core/state/snapshot: fix initial account range length
* core/state/snapshot: fix initial account range
* eth/protocols/snap: store flat states during the healing
* eth/protocols/snap: print logs
* core/state/snapshot: refactor (#4 )
* core/state/snapshot: refactor
* core/state/snapshot: tiny fix and polish
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
* core, eth: fixes
* core, eth: fix healing writer
* core, trie, eth: fix paths
* eth/protocols/snap: fix encoding
* eth, core: add debug log
* core/state/generate: release iterator asap (#5 )
core/state/snapshot: less copy
core/state/snapshot: revert split loop
core/state/snapshot: handle storage becoming empty, improve test robustness
core/state: test modified codehash
core/state/snapshot: polish
* core/state/snapshot: optimize stats counter
* core, eth: add metric
* core/state/snapshot: update comments
* core/state/snapshot: improve tests
* core/state/snapshot: replace secure trie with standard trie
* core/state/snapshot: wrap return as the struct
* core/state/snapshot: skip wiping correct states
* core/state/snapshot: updates
* core/state/snapshot: fixes
* core/state/snapshot: fix panic due to reference flaw in closure
* core/state/snapshot: fix errors in state generation logic + fix log output
* core/state/snapshot: remove an error case
* core/state/snapshot: fix condition-check for exhausted snap state
* core/state/snapshot: use stackTrie for small tries
* core/state/snapshot: don't resolve small storage tries in vain
* core/state/snapshot: properly clean up storage of deleted accounts
* core/state/snapshot: avoid RLP-encoding in some cases + minor nitpicks
* core/state/snapshot: fix error (+testcase)
* core/state/snapshot: clean up tests a bit
* core/state/snapshot: work in progress on better tests
* core/state/snapshot: polish code
* core/state/snapshot: fix trie iteration abortion trigger
* core/state/snapshot: fixes flaws
* core/state/snapshot: remove panic
* core/state/snapshot: fix abort
* core/state/snapshot: more tests (plus failing testcase)
* core/state/snapshot: more testcases + fix for failing test
* core/state/snapshot: testcase for malformed data
* core/state/snapshot: some test nitpicks
* core/state/snapshot: improvements to logging
* core/state/snapshot: testcase to demo error in abortion
* core/state/snapshot: fix abortion
* cmd/geth: make verify-state report the root
* trie: fix failing test
* core/state/snapshot: add timer metrics
* core/state/snapshot: fix metrics
* core/state/snapshot: udpate tests
* eth/protocols/snap: write snapshot account even if code or state is needed
* core/state/snapshot: fix diskmore check
* core/state/snapshot: review fixes
* core/state/snapshot: improve error message
* cmd/geth: rename 'error' to 'err' in logs
* core/state/snapshot: fix some review concerns
* core/state/snapshot, eth/protocols/snap: clear snapshot marker when starting/resuming snap sync
* core: add error log
* core/state/snapshot: use proper timers for metrics collection
* core/state/snapshot: address some review concerns
* eth/protocols/snap: improved log message
* eth/protocols/snap: fix heal logs to condense infos
* core/state/snapshot: wait for generator termination before restarting
* core/state/snapshot: revert timers to counters to track total time
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-04-14 23:23:11 +03:00
7644795950
eth/protocols/snap: try to prevent requests timing out
2021-03-29 14:17:35 +02:00
099be04100
eth/protocols/snap: add peer id and req id to the timeout logs
2021-03-26 22:29:22 +02:00
c5df05b9a9
eth/protocols/snap: fix the flaws in the snap sync ( #22553 )
...
* eth/protocols/snap: fix snap sync
* eth/protocols/snap: fix tests
* eth: fix tiny
* eth: update tests
* eth: update tests
* core/state/snapshot: testcase for #22534
* eth/protocols/snap: fix boundary loss on full-but-proven range
* core/state/snapshot: lintfix
* eth: address comment
* eth: fix handler
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-03-24 16:33:34 +02:00
410089afea
eth/protocols/snap, eth/downloader: don't use bloom filter in snap sync
2021-03-17 09:36:34 +01:00
b2b5c82aca
eth/protocols/snap: lower abortion and resumption logs to debug
2021-02-25 12:56:18 +02:00
bfdff4c5b8
eth: fix snap sync cancellation
2021-02-16 16:11:33 +02:00
ef84da8481
all: remove unneeded parentheses ( #21921 )
...
* remove uneeded convertion type
* remove redundant type in composite literal
* omit explicit type where implicit
* remove unused redundant parenthesis
* remove redundant import alias duktape
2021-02-02 11:32:44 +02:00
797b0812ab
eth/protocols/snap: snap sync testing ( #22179 )
...
* eth/protocols/snap: make timeout configurable
* eth/protocols/snap: snap sync testing
* eth/protocols/snap: test to trigger panic
* eth/protocols/snap: fix race condition on timeouts
* eth/protocols/snap: return error on cancelled sync
* squashme: updates + test causing panic + properly serve accounts in order
* eth/protocols/snap: revert failing storage response
* eth/protocols/snap: revert on bad responses (storage, code)
* eth/protocols/snap: fix account handling stall
* eth/protocols/snap: fix remaining revertal-issues
* eth/protocols/snap: timeouthandler for bytecode requests
* eth/protocols/snap: debugging + fix log message
* eth/protocols/snap: fix misspelliings in docs
* eth/protocols/snap: fix race in bytecode handling
* eth/protocols/snap: undo deduplication of storage roots
* synctests: refactor + minify panic testcase
* eth/protocols/snap: minor polishes
* eth: minor polishes to make logs more useful
* eth/protocols/snap: remove excessive logs from the test runs
* eth/protocols/snap: stress tests with concurrency
* eth/protocols/snap: further fixes to test cancel channel handling
* eth/protocols/snap: extend test timeouts on CI
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-01-25 08:17:05 +02:00
4bb5c6ca7a
eth/protocols/snap: speed up hash checks ( #22023 )
...
* eth/protocols/snap: speed up hash checks
* eth/protocols/snap: nit fix
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-01-07 18:12:41 +02:00
58b9db5f7c
eth/protocols/snap: track reverts when peer rejects request ( #22016 )
...
* eth/protocols/snap: reschedule missed deliveries
* eth/protocols/snap: clarify log message
* eth/protocols/snap: revert failures async and update runloop
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-01-07 12:58:07 +02:00
a425a47ddc
core/rawdb, eth/protocols : Method name typo fix ( #22026 )
2020-12-27 22:38:16 +01:00
017831dd5b
core, eth: split eth package, implement snap protocol ( #21482 )
...
This commit splits the eth package, separating the handling of eth and snap protocols. It also includes the capability to run snap sync (https://github.com/ethereum/devp2p/blob/master/caps/snap.md ) , but does not enable it by default.
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Martin Holst Swende <martin@swende.se>
2020-12-14 10:27:15 +01:00
Martin Holst Swende
Péter Szilágyi
Justin Traglia
rjl493456442
Marius van der Wijden
沉风
Guillaume Ballet
Felix Lange
Alex Prut
ucwong