52b5d2d869
eth/protocols/snap: use storage batch, not account batch in st task
2021-04-30 18:24:34 +03:00
745757ac6b
core, eth: abort snapshot generation on snap sync and resume later
2021-04-30 17:03:10 +03:00
fae165a5de
core, eth, ethdb, trie: simplify range proofs
2021-04-29 10:59:08 +03:00
a81cf0d2b3
trie: remove redundant returns + use stacktrie where applicable ( #22760 )
...
* trie: add benchmark for proofless range
* trie: remove unused returns + use stacktrie
2021-04-28 22:47:48 +03:00
558bff4008
eth/protocols/snap: lower the packet size to avoid overloading link
2021-04-28 21:40:06 +03:00
0c99868416
cmd/devp2p, eth/protocols/eth: fix tests + make sanity checks earlier ( #22749 )
2021-04-28 08:48:07 +02:00
caea6c4661
eth/protocols/snap: generate storage trie from full dirty snap data ( #22668 )
...
* eth/protocols/snap: generate storage trie from full dirty snap data
* eth/protocols/snap: get rid of some more dead code
* eth/protocols/snap: less frequent logs, also log during trie generation
* eth/protocols/snap: implement dirty account range stack-hashing
* eth/protocols/snap: don't loop on account trie generation
* eth/protocols/snap: fix account format in trie
* core, eth, ethdb: glue snap packets together, but not chunks
* eth/protocols/snap: print completion log for snap phase
* eth/protocols/snap: extended tests
* eth/protocols/snap: make testcase pass
* eth/protocols/snap: fix account stacktrie commit without defer
* ethdb: fix key counts on reset
* eth/protocols: fix typos
* eth/protocols/snap: make better use of delivered data (#44 )
* eth/protocols/snap: make better use of delivered data
* squashme
* eth/protocols/snap: reduce chunking
* squashme
* eth/protocols/snap: reduce chunking further
* eth/protocols/snap: break out hash range calculations
* eth/protocols/snap: use sort.Search instead of looping
* eth/protocols/snap: prevent crash on storage response with no keys
* eth/protocols/snap: nitpicks all around
* eth/protocols/snap: clear heal need on 1-chunk storage completion
* eth/protocols/snap: fix range chunker, add tests
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
* trie: fix test API error
* eth/protocols/snap: fix some further liter issues
* eth/protocols/snap: fix accidental batch reuse
Co-authored-by: Martin Holst Swende <martin@swende.se>
2021-04-27 17:19:59 +03:00
1fb9a6dd32
eth/protocols, prp/tracker: add support for req/rep rtt tracking ( #22608 )
...
* eth/protocols, prp/tracker: add support for req/rep rtt tracking
* p2p/tracker: sanity cap the number of pending requests
* pap/tracker: linter <3
* p2p/tracker: disable entire tracker if no metrics are enabled
2021-04-22 11:42:46 +03:00
9553c98de8
eth/protocols/snap: use ephemeral channels to avoid cross-sync delveries
2021-04-15 21:16:54 +03:00
1e207342b5
all: make logs a bit easier on the eye to digest ( #22665 )
...
* all: add thousandths separators for big numbers on log messages
* p2p/sentry: drop accidental file
* common, log: add fast number formatter
* common, eth/protocols/snap: simplifty fancy num types
* log: handle nil big ints
2021-04-15 20:35:00 +03:00
7088f1e814
core, eth: faster snapshot generation ( #22504 )
...
* eth/protocols: persist received state segments
* core: initial implementation
* core/state/snapshot: add tests
* core, eth: updates
* eth/protocols/snapshot: count flat state size
* core/state: add metrics
* core/state/snapshot: skip unnecessary deletion
* core/state/snapshot: rename
* core/state/snapshot: use the global batch
* core/state/snapshot: add logs and fix wiping
* core/state/snapshot: fix
* core/state/snapshot: save generation progress even if the batch is empty
* core/state/snapshot: fixes
* core/state/snapshot: fix initial account range length
* core/state/snapshot: fix initial account range
* eth/protocols/snap: store flat states during the healing
* eth/protocols/snap: print logs
* core/state/snapshot: refactor (#4 )
* core/state/snapshot: refactor
* core/state/snapshot: tiny fix and polish
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
* core, eth: fixes
* core, eth: fix healing writer
* core, trie, eth: fix paths
* eth/protocols/snap: fix encoding
* eth, core: add debug log
* core/state/generate: release iterator asap (#5 )
core/state/snapshot: less copy
core/state/snapshot: revert split loop
core/state/snapshot: handle storage becoming empty, improve test robustness
core/state: test modified codehash
core/state/snapshot: polish
* core/state/snapshot: optimize stats counter
* core, eth: add metric
* core/state/snapshot: update comments
* core/state/snapshot: improve tests
* core/state/snapshot: replace secure trie with standard trie
* core/state/snapshot: wrap return as the struct
* core/state/snapshot: skip wiping correct states
* core/state/snapshot: updates
* core/state/snapshot: fixes
* core/state/snapshot: fix panic due to reference flaw in closure
* core/state/snapshot: fix errors in state generation logic + fix log output
* core/state/snapshot: remove an error case
* core/state/snapshot: fix condition-check for exhausted snap state
* core/state/snapshot: use stackTrie for small tries
* core/state/snapshot: don't resolve small storage tries in vain
* core/state/snapshot: properly clean up storage of deleted accounts
* core/state/snapshot: avoid RLP-encoding in some cases + minor nitpicks
* core/state/snapshot: fix error (+testcase)
* core/state/snapshot: clean up tests a bit
* core/state/snapshot: work in progress on better tests
* core/state/snapshot: polish code
* core/state/snapshot: fix trie iteration abortion trigger
* core/state/snapshot: fixes flaws
* core/state/snapshot: remove panic
* core/state/snapshot: fix abort
* core/state/snapshot: more tests (plus failing testcase)
* core/state/snapshot: more testcases + fix for failing test
* core/state/snapshot: testcase for malformed data
* core/state/snapshot: some test nitpicks
* core/state/snapshot: improvements to logging
* core/state/snapshot: testcase to demo error in abortion
* core/state/snapshot: fix abortion
* cmd/geth: make verify-state report the root
* trie: fix failing test
* core/state/snapshot: add timer metrics
* core/state/snapshot: fix metrics
* core/state/snapshot: udpate tests
* eth/protocols/snap: write snapshot account even if code or state is needed
* core/state/snapshot: fix diskmore check
* core/state/snapshot: review fixes
* core/state/snapshot: improve error message
* cmd/geth: rename 'error' to 'err' in logs
* core/state/snapshot: fix some review concerns
* core/state/snapshot, eth/protocols/snap: clear snapshot marker when starting/resuming snap sync
* core: add error log
* core/state/snapshot: use proper timers for metrics collection
* core/state/snapshot: address some review concerns
* eth/protocols/snap: improved log message
* eth/protocols/snap: fix heal logs to condense infos
* core/state/snapshot: wait for generator termination before restarting
* core/state/snapshot: revert timers to counters to track total time
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-04-14 23:23:11 +03:00
fe1586b094
eth, les: drop support for eth/64, fix eth/66 tests
2021-04-09 10:39:45 +03:00
7644795950
eth/protocols/snap: try to prevent requests timing out
2021-03-29 14:17:35 +02:00
099be04100
eth/protocols/snap: add peer id and req id to the timeout logs
2021-03-26 22:29:22 +02:00
2550e46269
eth/protocols, metrics: use resetting histograms for rare packets
2021-03-26 16:14:12 +02:00
6d7ff6acea
eth/protocols, metrics, p2p: add handler performance metrics
2021-03-26 14:00:06 +02:00
c5df05b9a9
eth/protocols/snap: fix the flaws in the snap sync ( #22553 )
...
* eth/protocols/snap: fix snap sync
* eth/protocols/snap: fix tests
* eth: fix tiny
* eth: update tests
* eth: update tests
* core/state/snapshot: testcase for #22534
* eth/protocols/snap: fix boundary loss on full-but-proven range
* core/state/snapshot: lintfix
* eth: address comment
* eth: fix handler
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-03-24 16:33:34 +02:00
477ec75323
Merge pull request #22513 from holiman/no_snap_bloom
...
eth/protocols/snap, eth/downloader: don't use bloom filter in snap sync
2021-03-23 11:37:32 +02:00
0c70b83e00
all: add read-only option to database ( #22407 )
...
* all: add read-only option to database
* all: fixes tests
* cmd/geth: migrate flags
* cmd/geth: fix the compact
* cmd/geth: fix the format
* cmd/geth: fix log
* cmd: add chain-readonly
* core: add readonly notion to freezer
* core/rawdb: add log
* core/rawdb: fix freezer close
* cmd: fix
* cmd, core: construct db
* core: update tests
2021-03-22 19:06:30 +01:00
117fa7d4a1
eth/protocols/snap: fix typo ( #22530 )
2021-03-19 08:49:24 +01:00
410089afea
eth/protocols/snap, eth/downloader: don't use bloom filter in snap sync
2021-03-17 09:36:34 +01:00
b2b5c82aca
eth/protocols/snap: lower abortion and resumption logs to debug
2021-02-25 12:56:18 +02:00
6ec1561044
eth: implement eth66 ( #22241 )
...
* eth/protocols/eth: split up the eth protocol handlers
* eth/protocols/eth: define eth-66 protocol messages
* eth/protocols/eth: poc implement getblockheaders on eth/66
* eth/protocols/eth: implement remaining eth-66 handlers
* eth/protocols: define handler map for eth 66
* eth/downloader: use protocol constants from eth package
* eth/protocols/eth: add ETH66 capability
* eth/downloader: tests for eth66
* eth/downloader: fix error in tests
* eth/protocols/eth: use eth66 for outgoing requests
* eth/protocols/eth: remove unused error type
* eth/protocols/eth: define protocol length
* eth/protocols/eth: fix pooled tx over eth66
* protocols/eth/handlers: revert behavioural change which caused tests to fail
* eth/downloader: fix failing test
* eth/protocols/eth: add testcases + fix flaw with header requests
* eth/protocols: change comments
* eth/protocols/eth: review fixes + fixed flaw in RequestOneHeader
* eth/protocols: documentation
* eth/protocols/eth: review concerns about types
2021-02-18 18:54:29 +02:00
e01096f531
eth/handler, broadcast: optimize tx broadcast mechanism ( #22176 )
...
This PR optimizes the broadcast loop. Instead of iterating twice through a given set of transactions to weed out which peers have and which do not have a tx, to send/announce transactions, we do it only once.
2021-02-17 14:59:00 +01:00
bfdff4c5b8
eth: fix snap sync cancellation
2021-02-16 16:11:33 +02:00
ef84da8481
all: remove unneeded parentheses ( #21921 )
...
* remove uneeded convertion type
* remove redundant type in composite literal
* omit explicit type where implicit
* remove unused redundant parenthesis
* remove redundant import alias duktape
2021-02-02 11:32:44 +02:00
e3430ac7df
eth: check snap satelliteness, delegate drop to eth ( #22235 )
...
* eth: check snap satelliteness, delegate drop to eth
* eth: better handle eth/snap satellite relation, merge reg/unreg paths
2021-02-02 10:44:36 +02:00
797b0812ab
eth/protocols/snap: snap sync testing ( #22179 )
...
* eth/protocols/snap: make timeout configurable
* eth/protocols/snap: snap sync testing
* eth/protocols/snap: test to trigger panic
* eth/protocols/snap: fix race condition on timeouts
* eth/protocols/snap: return error on cancelled sync
* squashme: updates + test causing panic + properly serve accounts in order
* eth/protocols/snap: revert failing storage response
* eth/protocols/snap: revert on bad responses (storage, code)
* eth/protocols/snap: fix account handling stall
* eth/protocols/snap: fix remaining revertal-issues
* eth/protocols/snap: timeouthandler for bytecode requests
* eth/protocols/snap: debugging + fix log message
* eth/protocols/snap: fix misspelliings in docs
* eth/protocols/snap: fix race in bytecode handling
* eth/protocols/snap: undo deduplication of storage roots
* synctests: refactor + minify panic testcase
* eth/protocols/snap: minor polishes
* eth: minor polishes to make logs more useful
* eth/protocols/snap: remove excessive logs from the test runs
* eth/protocols/snap: stress tests with concurrency
* eth/protocols/snap: further fixes to test cancel channel handling
* eth/protocols/snap: extend test timeouts on CI
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-01-25 08:17:05 +02:00
c76573a97b
eth/protocols/eth: fix slice resize flaw ( #22181 )
2021-01-16 18:15:18 +01:00
4bb5c6ca7a
eth/protocols/snap: speed up hash checks ( #22023 )
...
* eth/protocols/snap: speed up hash checks
* eth/protocols/snap: nit fix
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-01-07 18:12:41 +02:00
58b9db5f7c
eth/protocols/snap: track reverts when peer rejects request ( #22016 )
...
* eth/protocols/snap: reschedule missed deliveries
* eth/protocols/snap: clarify log message
* eth/protocols/snap: revert failures async and update runloop
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-01-07 12:58:07 +02:00
a425a47ddc
core/rawdb, eth/protocols : Method name typo fix ( #22026 )
2020-12-27 22:38:16 +01:00
0fe66f8ae4
eth/protocols/eth: remove magic numbers in test ( #21999 )
2020-12-14 14:31:23 +01:00
017831dd5b
core, eth: split eth package, implement snap protocol ( #21482 )
...
This commit splits the eth package, separating the handling of eth and snap protocols. It also includes the capability to run snap sync (https://github.com/ethereum/devp2p/blob/master/caps/snap.md ) , but does not enable it by default.
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Martin Holst Swende <martin@swende.se>
2020-12-14 10:27:15 +01:00
Péter Szilágyi
Martin Holst Swende
gary rong
wuff1996
Alex Prut
Dan DeGreef
ucwong
Mr-Leshiy