8321fe2fda
tests: fix goroutine leak related to state snapshot generation ( #28974 )
...
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
2024-02-14 17:02:56 +01:00
fe91d476ba
all: remove the dependency from trie to triedb ( #28824 )
...
This change removes the dependency from trie package to triedb package.
2024-02-13 14:49:53 +01:00
c1d5a012ea
core/state, tests: fix memory leak via fastcache ( #28387 )
...
This change fixes a memory leak, when running either state-tests or blockchain-tests, we allocate a `1MB` fastcache during snapshot generation. `fastcache` is a bit special, and requires a `Reset()` (it has it's own memory allocator).
The `1MB` was hidden [here](https://github.com/ethereum/go-ethereum/blob/master/tests/state_test_util.go#L333 ) and [here](https://github.com/ethereum/go-ethereum/blob/master/tests/block_test_util.go#L146 ) respectively.
2023-10-20 13:35:49 +02:00
56d2366699
core/state/snapshot: replace diffToDisk ideal batch size with 64MB ( #27977 )
2023-08-25 15:48:10 +03:00
0c6bbeb423
core, eth, trie: expose more detailed dirty ram tracking for diff layers ( #27971 )
2023-08-23 14:08:39 +03:00
0e5d2c7c53
core/state/snapshot, core/types, eth: move account definition to type ( #27323 )
...
* core/state/snapshot, core/types, eth: move account definition to type
* core, eth: revert snapshot Account API change
2023-06-06 11:17:39 +03:00
fb8a3aaf1e
core/state: use atomic.Bool ( #26992 )
2023-03-28 03:06:50 -04:00
8860b39754
all: prepare for path-based trie storage ( #26603 )
...
This PR moves some trie-related db accessor methods to a different file, and also removes the schema type. Instead of the schema type, a string is used to distinguish between hashbased/pathbased db accessors.
This also moves some code from trie package to rawdb package.
This PR is intended to be a no-functionality-change prep PR for #25963 .
---------
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
2023-02-06 10:28:40 -05:00
743e404906
core, eth, les, tests, trie: abstract node scheme ( #25532 )
...
This PR introduces a node scheme abstraction. The interface is only implemented by `hashScheme` at the moment, but will be extended by `pathScheme` very soon.
Apart from that, a few changes are also included which is worth mentioning:
- port the changes in the stacktrie, tracking the path prefix of nodes during commit
- use ethdb.Database for constructing trie.Database. This is not necessary right now, but it is required for path-based used to open reverse diff freezer
2022-11-28 14:31:28 +01:00
60e30a940b
core/rawdb: refactor db inspector for extending multiple ancient store ( #25896 )
...
This PR ports a few changes from PBSS:
- Fix the snapshot generator waiter in case the generation is not even initialized
- Refactor db inspector for ancient store
2022-10-28 10:23:49 +02:00
ee301c750b
all: fix docstrings
2022-10-04 09:18:02 +02:00
3da42f85d9
all: clean up the configs for pruner and snapshotter ( #22396 )
...
This PR cleans up the configurations for pruner and snapshotter by passing a config struct.
And also, this PR disables the snapshot background generation if the chain is opened in "read-only" mode. The read-only mode is necessary in some cases. For example, we have a list of commands to open the etheruem node in "read-only" mode, like export-chain. In these cases, the snapshot background generation is non expected and should be banned explicitly.
2022-09-23 20:20:36 +02:00
b628d72766
build: upgrade to go 1.19 ( #25726 )
...
This changes the CI / release builds to use the latest Go version. It also
upgrades golangci-lint to a newer version compatible with Go 1.19.
In Go 1.19, godoc has gained official support for links and lists. The
syntax for code blocks in doc comments has changed and now requires a
leading tab character. gofmt adapts comments to the new syntax
automatically, so there are a lot of comment re-formatting changes in this
PR. We need to apply the new format in order to pass the CI lint stage with
Go 1.19.
With the linter upgrade, I have decided to disable 'gosec' - it produces
too many false-positive warnings. The 'deadcode' and 'varcheck' linters
have also been removed because golangci-lint warns about them being
unmaintained. 'unused' provides similar coverage and we already have it
enabled, so we don't lose much with this change.
2022-09-10 13:25:40 +02:00
c3701b265e
core/rawdb: add specified key length iterator ( #24535 )
2022-03-15 10:28:26 +02:00
f915f6873f
core/state/snapshot: fix data race in layer flattening ( #23628 )
...
* core/state/snapshot: fix data race in layer flattening
* core/state/snapshot: fix typo
2021-10-15 10:52:40 +03:00
9d6480c3cd
core/state/snapshot: gofmt
2021-08-10 16:58:38 +03:00
66948316f7
core/state/snapshot: clarify comment about snapshot repair ( #23305 )
...
Co-authored-by: Tyler Chambers <me@tylerchambers.net>
Co-authored-by: Felix Lange <fjl@twurst.com>
2021-08-10 11:16:53 +02:00
57d9e0ac75
core/state/snapshot: fix typo in comment ( #23219 )
2021-08-10 11:04:29 +02:00
745757ac6b
core, eth: abort snapshot generation on snap sync and resume later
2021-04-30 17:03:10 +03:00
d6ffa14035
core: nuke legacy snapshot supporting ( #22663 )
2021-04-20 08:27:46 +03:00
7088f1e814
core, eth: faster snapshot generation ( #22504 )
...
* eth/protocols: persist received state segments
* core: initial implementation
* core/state/snapshot: add tests
* core, eth: updates
* eth/protocols/snapshot: count flat state size
* core/state: add metrics
* core/state/snapshot: skip unnecessary deletion
* core/state/snapshot: rename
* core/state/snapshot: use the global batch
* core/state/snapshot: add logs and fix wiping
* core/state/snapshot: fix
* core/state/snapshot: save generation progress even if the batch is empty
* core/state/snapshot: fixes
* core/state/snapshot: fix initial account range length
* core/state/snapshot: fix initial account range
* eth/protocols/snap: store flat states during the healing
* eth/protocols/snap: print logs
* core/state/snapshot: refactor (#4 )
* core/state/snapshot: refactor
* core/state/snapshot: tiny fix and polish
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
* core, eth: fixes
* core, eth: fix healing writer
* core, trie, eth: fix paths
* eth/protocols/snap: fix encoding
* eth, core: add debug log
* core/state/generate: release iterator asap (#5 )
core/state/snapshot: less copy
core/state/snapshot: revert split loop
core/state/snapshot: handle storage becoming empty, improve test robustness
core/state: test modified codehash
core/state/snapshot: polish
* core/state/snapshot: optimize stats counter
* core, eth: add metric
* core/state/snapshot: update comments
* core/state/snapshot: improve tests
* core/state/snapshot: replace secure trie with standard trie
* core/state/snapshot: wrap return as the struct
* core/state/snapshot: skip wiping correct states
* core/state/snapshot: updates
* core/state/snapshot: fixes
* core/state/snapshot: fix panic due to reference flaw in closure
* core/state/snapshot: fix errors in state generation logic + fix log output
* core/state/snapshot: remove an error case
* core/state/snapshot: fix condition-check for exhausted snap state
* core/state/snapshot: use stackTrie for small tries
* core/state/snapshot: don't resolve small storage tries in vain
* core/state/snapshot: properly clean up storage of deleted accounts
* core/state/snapshot: avoid RLP-encoding in some cases + minor nitpicks
* core/state/snapshot: fix error (+testcase)
* core/state/snapshot: clean up tests a bit
* core/state/snapshot: work in progress on better tests
* core/state/snapshot: polish code
* core/state/snapshot: fix trie iteration abortion trigger
* core/state/snapshot: fixes flaws
* core/state/snapshot: remove panic
* core/state/snapshot: fix abort
* core/state/snapshot: more tests (plus failing testcase)
* core/state/snapshot: more testcases + fix for failing test
* core/state/snapshot: testcase for malformed data
* core/state/snapshot: some test nitpicks
* core/state/snapshot: improvements to logging
* core/state/snapshot: testcase to demo error in abortion
* core/state/snapshot: fix abortion
* cmd/geth: make verify-state report the root
* trie: fix failing test
* core/state/snapshot: add timer metrics
* core/state/snapshot: fix metrics
* core/state/snapshot: udpate tests
* eth/protocols/snap: write snapshot account even if code or state is needed
* core/state/snapshot: fix diskmore check
* core/state/snapshot: review fixes
* core/state/snapshot: improve error message
* cmd/geth: rename 'error' to 'err' in logs
* core/state/snapshot: fix some review concerns
* core/state/snapshot, eth/protocols/snap: clear snapshot marker when starting/resuming snap sync
* core: add error log
* core/state/snapshot: use proper timers for metrics collection
* core/state/snapshot: address some review concerns
* eth/protocols/snap: improved log message
* eth/protocols/snap: fix heal logs to condense infos
* core/state/snapshot: wait for generator termination before restarting
* core/state/snapshot: revert timers to counters to track total time
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-04-14 23:23:11 +03:00
61ff3e86b2
core/state/snapshot, ethdb: track deletions more accurately ( #22582 )
...
* core/state/snapshot, ethdb: track deletions more accurately
* core/state/snapshot: don't reset the iterator, leveldb's screwy
* ethdb: don't mess with the insert batches for now
2021-03-30 19:04:22 +03:00
a31f6d54df
core/state/snapshot: fix panic on missing parent
2021-03-22 22:41:28 +02:00
9ec3329899
core/state/snapshot: ensure Cap retains a min number of layers
2021-02-16 15:25:20 +02:00
f566dd305e
all: bloom-filter based pruning mechanism ( #21724 )
...
* cmd, core, tests: initial state pruner
core: fix db inspector
cmd/geth: add verify-state
cmd/geth: add verification tool
core/rawdb: implement flatdb
cmd, core: fix rebase
core/state: use new contract code layout
core/state/pruner: avoid deleting genesis state
cmd/geth: add helper function
core, cmd: fix extract genesis
core: minor fixes
contracts: remove useless
core/state/snapshot: plugin stacktrie
core: polish
core/state/snapshot: iterate storage concurrently
core/state/snapshot: fix iteration
core: add comments
core/state/snapshot: polish code
core/state: polish
core/state/snapshot: rebase
core/rawdb: add comments
core/rawdb: fix tests
core/rawdb: improve tests
core/state/snapshot: fix concurrent iteration
core/state: run pruning during the recovery
core, trie: implement martin's idea
core, eth: delete flatdb and polish pruner
trie: fix import
core/state/pruner: add log
core/state/pruner: fix issues
core/state/pruner: don't read back
core/state/pruner: fix contract code write
core/state/pruner: check root node presence
cmd, core: polish log
core/state: use HEAD-127 as the target
core/state/snapshot: improve tests
cmd/geth: fix verification tool
cmd/geth: use HEAD as the verification default target
all: replace the bloomfilter with martin's fork
cmd, core: polish code
core, cmd: forcibly delete state root
core/state/pruner: add hash64
core/state/pruner: fix blacklist
core/state: remove blacklist
cmd, core: delete trie clean cache before pruning
cmd, core: fix lint
cmd, core: fix rebase
core/state: fix the special case for clique networks
core/state/snapshot: remove useless code
core/state/pruner: capping the snapshot after pruning
cmd, core, eth: fixes
core/rawdb: update db inspector
cmd/geth: polish code
core/state/pruner: fsync bloom filter
cmd, core: print warning log
core/state/pruner: adjust the parameters for bloom filter
cmd, core: create the bloom filter by size
core: polish
core/state/pruner: sanitize invalid bloomfilter size
cmd: address comments
cmd/geth: address comments
cmd/geth: address comment
core/state/pruner: address comments
core/state/pruner: rename homedir to datadir
cmd, core: address comments
core/state/pruner: address comment
core/state: address comments
core, cmd, tests: address comments
core: address comments
core/state/pruner: release the iterator after each commit
core/state/pruner: improve pruner
cmd, core: adjust bloom paramters
core/state/pruner: fix lint
core/state/pruner: fix tests
core: fix rebase
core/state/pruner: remove atomic rename
core/state/pruner: address comments
all: run go mod tidy
core/state/pruner: avoid false-positive for the middle state roots
core/state/pruner: add checks for middle roots
cmd/geth: replace crit with error
* core/state/pruner: fix lint
* core: drop legacy bloom filter
* core/state/snapshot: improve pruner
* core/state/snapshot: polish concurrent logs to report ETA vs. hashes
* core/state/pruner: add progress report for pruning and compaction too
* core: fix snapshot test API
* core/state: fix some pruning logs
* core/state/pruner: support recovering from bloom flush fail
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-02-08 13:16:30 +02:00
d2e1b17f18
snapshot, trie: fixed typos, mostly in snapshot pkg ( #22133 )
2021-01-07 08:36:21 +02:00
7b7b327ff2
core/state/snapshot: update generator marker in sync with flushes
2020-11-09 16:03:58 +02:00
b63e3c37a6
core: improve snapshot journal recovery ( #21594 )
...
* core/state/snapshot: introduce snapshot journal version
* core: update the disk layer in an atomic way
* core: persist the disk layer generator periodically
* core/state/snapshot: improve logging
* core/state/snapshot: forcibly ensure the legacy snapshot is matched
* core/state/snapshot: add debug logs
* core, tests: fix tests and special recovery case
* core: polish
* core: add more blockchain tests for snapshot recovery
* core/state: fix comment
* core: add recovery flag for snapshot
* core: add restart after start-after-crash tests
* core/rawdb: fix imports
* core: fix tests
* core: remove log
* core/state/snapshot: fix snapshot
* core: avoid callbacks in SetHead
* core: fix setHead cornercase where the threshold root has state
* core: small docs for the test cases
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-10-29 21:01:58 +02:00
43c278cdf9
core/state: disable snapshot iteration if it's not fully constructed ( #21682 )
...
* core/state/snapshot: add diskRoot function
* core/state/snapshot: disable iteration if the snapshot is generating
* core/state/snapshot: simplify the function
* core/state: panic for undefined layer
2020-10-28 14:27:37 +02:00
3ee91b9f2e
core/state/snapshot: reduce disk layer depth during generation
2020-08-24 13:22:36 +03:00
4f2784b38f
all: fix typos in comments ( #21118 )
2020-05-25 10:21:28 +02:00
26d271dfbb
core/state/snapshot: implement storage iterator ( #20971 )
...
* core/state/snapshot: implement storage iterator
* core/state/snapshot, tests: implement helper function
* core/state/snapshot: fix storage issue
If an account is deleted in the tx_1 but recreated in the tx_2,
the it can happen that in this diff layer, both destructedSet
and storageData records this account. In this case, the storage
iterator should be able to iterate the slots belong to new account
but disable further iteration in deeper layers(belong to old account)
* core/state/snapshot: address peter and martin's comment
* core/state: address comments
* core/state/snapshot: fix test
2020-04-29 12:53:08 +03:00
a4cf279494
core/state: extend snapshotter to handle account resurrections
2020-03-03 15:52:00 +02:00
6e05ccd845
core/state/snapshot, tests: sync snap gen + snaps in consensus tests
2020-03-03 09:17:13 +02:00
19099421dc
core/state/snapshot: faster account iteration, CLI integration
2020-02-25 12:51:15 +02:00
6ddb92a089
core/state/snapshot: full featured account iteration
2020-02-25 12:51:14 +02:00
22c494d399
core/state/snapshot: bloom, metrics and prefetcher fixes
2020-02-25 12:51:11 +02:00
fd39f722a3
core: journal the snapshot inside leveldb, not a flat file
2020-02-25 12:51:10 +02:00
351a5903b0
core/rawdb, core/state/snapshot: runtime snapshot generation
2020-02-25 12:51:08 +02:00
f300c0df01
core/state/snapshot: replace bigcache with fastcache
2020-02-25 12:51:08 +02:00
d754091a87
core/state/snapshot: unlink snapshots from blocks, quad->linear cleanup
2020-02-25 12:51:07 +02:00
cdf3f016df
snapshot: iteration and buffering optimizations
2020-02-25 12:51:06 +02:00
d7d81d7c12
core/state/snapshot: extract and split cap method, cover corners
2020-02-25 12:51:05 +02:00
e146fbe4e7
core/state: lazy sorting, snapshot invalidation
2020-02-25 12:51:05 +02:00
542df8898e
core: initial version of state snapshots
2020-02-25 12:51:04 +02:00
Martin HS
rjl493456442
aaronbuchwald
Péter Szilágyi
s7v7nislands
Felix Lange
Tyler Chambers
Ziyuan Zhong
Melvin Junhee Woo
Martin Michlmayr