Commit Graph

237 Commits

Author SHA1 Message Date
Darioush Jalali 2dfa4bcf6c
trie: test for edgecase in VerifyRangeProof (#24257)
* trie/proof: edge case for VerifyRangeProof

* more consistency with other tests in the file

* trie: fix test todo

Co-authored-by: Martin Holst Swende <martin@swende.se>
2022-01-21 14:35:30 +01:00
rjl493456442 ae45c97d3d
trie: fix range prover (#24266) 2022-01-21 10:41:51 +01:00
Darioush Jalali 2c58e6b62d
trie: use keyvalue reader for non-mutating methods (#24221) 2022-01-11 22:36:48 +02:00
Péter Szilágyi af2ca5a654
Merge pull request #24117 from holiman/db_has
trie, core, eth: use db.has over db.get where possible
2022-01-06 11:30:11 +02:00
Water 98be5f9a72
trie: fix spelling mistake (#24185)
mispelled words in comments: th enext
2022-01-04 17:23:52 +02:00
Martin Holst Swende 893502e561
trie, core, eth: use db.has over db.get where possible 2021-12-15 16:16:45 +01:00
Péter Szilágyi 58d1988349
core, eth, les, trie: remove the sync bloom, used by fast sync 2021-12-03 12:32:41 +02:00
Taeik Lim 85064ed09b
all: fix 'the the' in comments (#24036) 2021-12-02 15:42:09 +01:00
Paweł Bylica 86fe359a56
trie: simplify StackTrie implementation (#23950)
Trim the search key from head as it's being pushed deeper into the trie. Previously the search key was never modified but each node kept information how to slice and compare it in keyOffset. Now the keyOffset is not needed as this information is included in the slice of the search key. This way the keyOffset can be removed and key manipulation
simplified.
2021-11-29 11:02:40 +01:00
Péter Szilágyi 743769f48e
trie: reject deletions when verifying range proofs 2021-11-23 19:28:17 +02:00
Paweł Bylica 5d4bcbc14f
trie: more tests for stacktrie (#23936) 2021-11-22 08:49:18 +01:00
Ha ĐANG c36f8fefc3
trie: fix typos (#23768) 2021-10-18 20:53:56 +02:00
Martin Holst Swende 3a6fe69f23
eth/protocols/snap, trie: better error-handling (#23657) 2021-09-29 15:19:40 +02:00
Guillaume Ballet 443afc975c
core/state: move state account to core/types + abstracted "write account to trie" (#23567)
* core/state: abstracted "write account to trie" method

* fix appveyor build

* Apply suggestions from code review

Co-authored-by: Martin Holst Swende <martin@swende.se>

* review feedback

* core/state/accounts: move Account to core/types

* core/types: rename Account -> StateAccount

* core/state: restore EncodeRLP for stateObject

* core/types: add the missing file

* more review feedback

* more review feedback

Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
2021-09-28 10:48:07 +02:00
aaronbuchwald de2c44ab5c
trie: add missing copyright header (#23530)
* Add missing copyright header

* Update stacktrie_test.go

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-09-07 07:25:54 +03:00
gary rong a5a5237178
core, light, tests, trie: add state metrics (#23433) 2021-08-24 22:00:42 +03:00
chuwt 79bb9300c1
trie, les: add missing calls to Ticker.Stop (#23415) 2021-08-24 10:51:28 +02:00
Martin Holst Swende c131e812ae
eth/fetcher, trie: unit test reliability fixes (#23020)
Some tests take quite some time during exit, which I think causes
some appveyor fails like this:

    https://ci.appveyor.com/project/ethereum/go-ethereum/builds/39511210/job/xhom84eg2e4uulq3

One of the things that seem to take time during exit is waiting
(up to 100ms) for the syncbloom to close. This PR changes it to use
a channel, instead of looping with a 100ms wait.

This also includes some unrelated changes improving the reliability of
eth/fetcher tests, which fail a lot because they are time-dependent.
2021-06-30 22:24:17 +02:00
Evolution404 732a6a3666
trie: small optimization of delete in fullNode case (#22979)
When deleting in fullNode, and the new child node nn is not nil, there is no need
to check the number of non-nil entries in the node. This is because the fullNode 
must've contained at least two children before deletion, so there must be another
child node other than nn.

Co-authored-by: Felix Lange <fjl@twurst.com>
2021-06-20 15:59:00 +02:00
Evolution404 08379b5533
trie: remove the duplicate batch-write for 'preimage' (#23001) 2021-06-07 09:11:07 +02:00
Péter Szilágyi fae165a5de
core, eth, ethdb, trie: simplify range proofs 2021-04-29 10:59:08 +03:00
Martin Holst Swende a81cf0d2b3
trie: remove redundant returns + use stacktrie where applicable (#22760)
* trie: add benchmark for proofless range

* trie: remove unused returns + use stacktrie
2021-04-28 22:47:48 +03:00
Péter Szilágyi caea6c4661
eth/protocols/snap: generate storage trie from full dirty snap data (#22668)
* eth/protocols/snap: generate storage trie from full dirty snap data

* eth/protocols/snap: get rid of some more dead code

* eth/protocols/snap: less frequent logs, also log during trie generation

* eth/protocols/snap: implement dirty account range stack-hashing

* eth/protocols/snap: don't loop on account trie generation

* eth/protocols/snap: fix account format in trie

* core, eth, ethdb: glue snap packets together, but not chunks

* eth/protocols/snap: print completion log for snap phase

* eth/protocols/snap: extended tests

* eth/protocols/snap: make testcase pass

* eth/protocols/snap: fix account stacktrie commit without defer

* ethdb: fix key counts on reset

* eth/protocols: fix typos

* eth/protocols/snap: make better use of delivered data (#44)

* eth/protocols/snap: make better use of delivered data

* squashme

* eth/protocols/snap: reduce chunking

* squashme

* eth/protocols/snap: reduce chunking further

* eth/protocols/snap: break out hash range calculations

* eth/protocols/snap: use sort.Search instead of looping

* eth/protocols/snap: prevent crash on storage response with no keys

* eth/protocols/snap: nitpicks all around

* eth/protocols/snap: clear heal need on 1-chunk storage completion

* eth/protocols/snap: fix range chunker, add tests

Co-authored-by: Péter Szilágyi <peterke@gmail.com>

* trie: fix test API error

* eth/protocols/snap: fix some further liter issues

* eth/protocols/snap: fix accidental batch reuse

Co-authored-by: Martin Holst Swende <martin@swende.se>
2021-04-27 17:19:59 +03:00
Martin Holst Swende 49281ab84f
core/state/snapshot, true: reuse dirty data instead of hitting disk when generating (#22667)
* core/state/snapshot: reuse memory data instead of hitting disk when generating

* trie: minor nitpicks wrt the resolver optimization

* core/state/snapshot, trie: use key/value store for resolver

* trie: fix linter

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-04-23 14:39:18 +03:00
Martin Holst Swende 4b783c0064
trie: improve the node iterator seek operation (#22470)
This change improves the efficiency of the nodeIterator seek
operation. Previously, seek essentially ran the iterator forward
until it found the matching node. With this change, it skips
over fullnode children and avoids resolving them from the database.
2021-04-21 12:25:26 +02:00
Martin Holst Swende 581539c6ee
trie: make stacktrie support binary marshal/unmarshal (#22685) 2021-04-20 10:42:02 +02:00
Martin Holst Swende 09d44e9925
core/state/snapshot: avoid copybytes for stacktrie 2021-04-16 14:58:23 +02:00
Martin Holst Swende 4f3ba6742f
trie: make stacktrie not mutate input values (#22673)
The stacktrie is a bit un-untuitive, API-wise: since it mutates input values.
Such behaviour is dangerous, and easy to get wrong if the calling code 'forgets' this quirk. The behaviour is fixed by this PR, so that the input values are not modified by the stacktrie. 

Note: just as with the Trie, the stacktrie still references the live input objects, so it's still _not_ safe to mutate the values form the callsite.
2021-04-16 14:21:01 +02:00
gary rong 7088f1e814
core, eth: faster snapshot generation (#22504)
* eth/protocols: persist received state segments

* core: initial implementation

* core/state/snapshot: add tests

* core, eth: updates

* eth/protocols/snapshot: count flat state size

* core/state: add metrics

* core/state/snapshot: skip unnecessary deletion

* core/state/snapshot: rename

* core/state/snapshot: use the global batch

* core/state/snapshot: add logs and fix wiping

* core/state/snapshot: fix

* core/state/snapshot: save generation progress even if the batch is empty

* core/state/snapshot: fixes

* core/state/snapshot: fix initial account range length

* core/state/snapshot: fix initial account range

* eth/protocols/snap: store flat states during the healing

* eth/protocols/snap: print logs

* core/state/snapshot: refactor (#4)

* core/state/snapshot: refactor

* core/state/snapshot: tiny fix and polish

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>

* core, eth: fixes

* core, eth: fix healing writer

* core, trie, eth: fix paths

* eth/protocols/snap: fix encoding

* eth, core: add debug log

* core/state/generate: release iterator asap (#5)

core/state/snapshot: less copy

core/state/snapshot: revert split loop

core/state/snapshot: handle storage becoming empty, improve test robustness

core/state: test modified codehash

core/state/snapshot: polish

* core/state/snapshot: optimize stats counter

* core, eth: add metric

* core/state/snapshot: update comments

* core/state/snapshot: improve tests

* core/state/snapshot: replace secure trie with standard trie

* core/state/snapshot: wrap return as the struct

* core/state/snapshot: skip wiping correct states

* core/state/snapshot: updates

* core/state/snapshot: fixes

* core/state/snapshot: fix panic due to reference flaw in closure

* core/state/snapshot: fix errors in state generation logic + fix log output

* core/state/snapshot: remove an error case

* core/state/snapshot: fix condition-check for exhausted snap state

* core/state/snapshot: use stackTrie for small tries

* core/state/snapshot: don't resolve small storage tries in vain

* core/state/snapshot: properly clean up storage of deleted accounts

* core/state/snapshot: avoid RLP-encoding in some cases + minor nitpicks

* core/state/snapshot: fix error (+testcase)

* core/state/snapshot: clean up tests a bit

* core/state/snapshot: work in progress on better tests

* core/state/snapshot: polish code

* core/state/snapshot: fix trie iteration abortion trigger

* core/state/snapshot: fixes flaws

* core/state/snapshot: remove panic

* core/state/snapshot: fix abort

* core/state/snapshot: more tests (plus failing testcase)

* core/state/snapshot: more testcases + fix for failing test

* core/state/snapshot: testcase for malformed data

* core/state/snapshot: some test nitpicks

* core/state/snapshot: improvements to logging

* core/state/snapshot: testcase to demo error in abortion

* core/state/snapshot: fix abortion

* cmd/geth: make verify-state report the root

* trie: fix failing test

* core/state/snapshot: add timer metrics

* core/state/snapshot: fix metrics

* core/state/snapshot: udpate tests

* eth/protocols/snap: write snapshot account even if code or state is needed

* core/state/snapshot: fix diskmore check

* core/state/snapshot: review fixes

* core/state/snapshot: improve error message

* cmd/geth: rename 'error' to 'err' in logs

* core/state/snapshot: fix some review concerns

* core/state/snapshot, eth/protocols/snap: clear snapshot marker when starting/resuming snap sync

* core: add error log

* core/state/snapshot: use proper timers for metrics collection

* core/state/snapshot: address some review concerns

* eth/protocols/snap: improved log message

* eth/protocols/snap: fix heal logs to condense infos

* core/state/snapshot: wait for generator termination before restarting

* core/state/snapshot: revert timers to counters to track total time

Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-04-14 23:23:11 +03:00
gary rong 0c70b83e00
all: add read-only option to database (#22407)
* all: add read-only option to database

* all: fixes tests

* cmd/geth: migrate flags

* cmd/geth: fix the compact

* cmd/geth: fix the format

* cmd/geth: fix log

* cmd: add chain-readonly

* core: add readonly notion to freezer

* core/rawdb: add log

* core/rawdb: fix freezer close

* cmd: fix

* cmd, core: construct db

* core: update tests
2021-03-22 19:06:30 +01:00
lightclient bbfb1e4008
all: add support for EIP-2718, EIP-2930 transactions (#21502)
This adds support for EIP-2718 typed transactions as well as EIP-2930
access list transactions (tx type 1). These EIPs are scheduled for the
Berlin fork.

There very few changes to existing APIs in core/types, and several new APIs
to deal with access list transactions. In particular, there are two new
constructor functions for transactions: types.NewTx and types.SignNewTx.
Since the canonical encoding of typed transactions is not RLP-compatible,
Transaction now has new methods for encoding and decoding: MarshalBinary
and UnmarshalBinary.

The existing EIP-155 signer does not support the new transaction types.
All code dealing with transaction signatures should be updated to use the
newer EIP-2930 signer. To make this easier for future updates, we have
added new constructor functions for types.Signer: types.LatestSigner and
types.LatestSignerForChainID. 

This change also adds support for the YoloV3 testnet.

Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Ryan Schneider <ryanleeschneider@gmail.com>
2021-02-25 15:26:57 +01:00
Péter Szilágyi e991bdae24
trie: fix bloom crash on fast sync restart 2021-02-16 10:44:38 +02:00
gary rong f566dd305e
all: bloom-filter based pruning mechanism (#21724)
* cmd, core, tests: initial state pruner

core: fix db inspector

cmd/geth: add verify-state

cmd/geth: add verification tool

core/rawdb: implement flatdb

cmd, core: fix rebase

core/state: use new contract code layout

core/state/pruner: avoid deleting genesis state

cmd/geth: add helper function

core, cmd: fix extract genesis

core: minor fixes

contracts: remove useless

core/state/snapshot: plugin stacktrie

core: polish

core/state/snapshot: iterate storage concurrently

core/state/snapshot: fix iteration

core: add comments

core/state/snapshot: polish code

core/state: polish

core/state/snapshot: rebase

core/rawdb: add comments

core/rawdb: fix tests

core/rawdb: improve tests

core/state/snapshot: fix concurrent iteration

core/state: run pruning during the recovery

core, trie: implement martin's idea

core, eth: delete flatdb and polish pruner

trie: fix import

core/state/pruner: add log

core/state/pruner: fix issues

core/state/pruner: don't read back

core/state/pruner: fix contract code write

core/state/pruner: check root node presence

cmd, core: polish log

core/state: use HEAD-127 as the target

core/state/snapshot: improve tests

cmd/geth: fix verification tool

cmd/geth: use HEAD as the verification default target

all: replace the bloomfilter with martin's fork

cmd, core: polish code

core, cmd: forcibly delete state root

core/state/pruner: add hash64

core/state/pruner: fix blacklist

core/state: remove blacklist

cmd, core: delete trie clean cache before pruning

cmd, core: fix lint

cmd, core: fix rebase

core/state: fix the special case for clique networks

core/state/snapshot: remove useless code

core/state/pruner: capping the snapshot after pruning

cmd, core, eth: fixes

core/rawdb: update db inspector

cmd/geth: polish code

core/state/pruner: fsync bloom filter

cmd, core: print warning log

core/state/pruner: adjust the parameters for bloom filter

cmd, core: create the bloom filter by size

core: polish

core/state/pruner: sanitize invalid bloomfilter size

cmd: address comments

cmd/geth: address comments

cmd/geth: address comment

core/state/pruner: address comments

core/state/pruner: rename homedir to datadir

cmd, core: address comments

core/state/pruner: address comment

core/state: address comments

core, cmd, tests: address comments

core: address comments

core/state/pruner: release the iterator after each commit

core/state/pruner: improve pruner

cmd, core: adjust bloom paramters

core/state/pruner: fix lint

core/state/pruner: fix tests

core: fix rebase

core/state/pruner: remove atomic rename

core/state/pruner: address comments

all: run go mod tidy

core/state/pruner: avoid false-positive for the middle state roots

core/state/pruner: add checks for middle roots

cmd/geth: replace crit with error

* core/state/pruner: fix lint

* core: drop legacy bloom filter

* core/state/snapshot: improve pruner

* core/state/snapshot: polish concurrent logs to report ETA vs. hashes

* core/state/pruner: add progress report for pruning and compaction too

* core: fix snapshot test API

* core/state: fix some pruning logs

* core/state/pruner: support recovering from bloom flush fail

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2021-02-08 13:16:30 +02:00
Alex Prut ef84da8481
all: remove unneeded parentheses (#21921)
* remove uneeded convertion type

* remove redundant type in composite literal

* omit explicit type where implicit

* remove unused redundant parenthesis

* remove redundant import alias duktape
2021-02-02 11:32:44 +02:00
gary rong 9e1bd0f367
trie: fix range prover (#22210)
Fixes a special case when the trie only has a single trie node and the range proof only contains a single element.
2021-01-22 10:11:24 +01:00
Martin Holst Swende 93a89b2681
go.mod: use github.com/holiman/bloomfilter/v2 (#22044)
* deps: use improved bloom filter implementation

* eth/handler, trie: use 4 keys for syncbloom + minor fixes

* eth/protocols, trie: revert change on syncbloom method signature
2021-01-12 17:39:31 +01:00
Melvin Junhee Woo d2e1b17f18
snapshot, trie: fixed typos, mostly in snapshot pkg (#22133) 2021-01-07 08:36:21 +02:00
Péter Szilágyi 017831dd5b
core, eth: split eth package, implement snap protocol (#21482)
This commit splits the eth package, separating the handling of eth and snap protocols. It also includes the capability to run snap sync (https://github.com/ethereum/devp2p/blob/master/caps/snap.md) , but does not enable it by default. 

Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Martin Holst Swende <martin@swende.se>
2020-12-14 10:27:15 +01:00
Martin Holst Swende b44f24e3e6
core, trie: speed up some tests with quadratic processing flaw (#21987)
This commit fixes a flaw in two testcases, and brings down the exec-time from ~40s to ~8s for trie/TestIncompleteSync.

The checkConsistency was performed over and over again on the complete set of nodes, not just the recently added, turning it into a quadratic runtime.
2020-12-10 14:48:32 +01:00
Alex Prut 810f9e057d
all: remove redundant conversions and import names (#21903) 2020-11-25 21:00:23 +01:00
Alex Prut c92faee66e
all: simplify nested complexity and if blocks ending with a return statement (#21854)
Changes:

    Simplify nested complexity
    If an if blocks ends with a return statement then remove the else nesting.

Most of the changes has also been reported in golint https://goreportcard.com/report/github.com/ethereum/go-ethereum#golint
2020-11-25 09:24:50 +01:00
Martin Holst Swende 6f88d6530a
trie, rpc, cmd/geth: fix tests on 32-bit and windows + minor rpc fixes (#21871)
* trie: fix tests to work on 32-bit systems

* les: make test work on 32-bit platform

* cmd/geth: fix windows-issues on tests

* trie: improve balance

* cmd/geth: make account tests less verbose + less mem intense

* rpc: make debug-level log output less verbose

* cmd/geth: lint
2020-11-19 22:50:47 +01:00
gary rong 23524f8900
all: disable recording preimage of trie keys (#21402)
* cmd, core, eth, light, trie: disable recording preimage by default

* core, eth: fix unit tests

* core: fix import

* all: change to nopreimage

* cmd, core, eth, trie: use cache.preimages flag

* cmd: enable preimages for archive node

* cmd/utils, trie: simplify preimage tracking a bit

* core: fix linter

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-11-18 11:51:33 +02:00
Martin Holst Swende 81678971db
trie, tests/fuzzers: implement a stacktrie fuzzer + stacktrie fixes (#21799)
* trie: fix error in stacktrie not committing small roots

* fuzzers: make trie-fuzzer use correct returnvalues

* trie: improved tests

* tests/fuzzers: fuzzer for stacktrie vs regular trie

* test/fuzzers: make stacktrie fuzzer use 32-byte keys

* trie: fix error in stacktrie with small nodes

* trie: add (skipped) testcase for stacktrie

* tests/fuzzers: address review comments for stacktrie fuzzer

* trie: fix docs in stacktrie
2020-11-09 15:08:12 +01:00
Martin Holst Swende 348c3bc47d
trie: fix flaw in stacktrie pool reuse (#21699) 2020-10-13 13:21:25 +02:00
gary rong 86dd005544
trie: polish commit function (#21692)
* trie: polish commit function

* trie: fix typo
2020-10-12 12:08:04 +02:00
Martin Holst Swende 9d1e2027a0
trie: add Commit-sequence tests for stacktrie commit (#21643) 2020-09-30 19:49:20 +02:00
gary rong 053ed9cc84
trie: polishes to trie committer (#21351)
* trie: update tests to check commit integrity

* trie: polish committer

* trie: fix typo

* trie: remove hasvalue notion

According to the benchmarks, type assertion between the pointer and
interface is extremely fast.

BenchmarkIntmethod-12           1000000000               1.91 ns/op
BenchmarkInterface-12           1000000000               2.13 ns/op
BenchmarkTypeSwitch-12          1000000000               1.81 ns/op
BenchmarkTypeAssertion-12       2000000000               1.78 ns/op

So the overhead for asserting whether the shortnode has "valuenode"
child is super tiny. No necessary to have another field.

* trie: linter nitpicks

Co-authored-by: Martin Holst Swende <martin@swende.se>
2020-09-30 13:45:56 +02:00
Guillaume Ballet 6c8310ebb4
trie: use stacktrie for Derivesha operation (#21407)
core/types: use stacktrie for derivesha

trie: add stacktrie file

trie: fix linter

core/types: use stacktrie for derivesha

rebased: adapt stacktrie to the newer version of DeriveSha

Co-authored-by: Martin Holst Swende <martin@swende.se>

More linter fixes

review feedback: no key offset for nodes converted to hashes

trie: use EncodeRLP for full nodes

core/types: insert txs in order in derivesha

trie: tests for derivesha with stacktrie

trie: make stacktrie use pooled hashers

trie: make stacktrie reuse tmp slice space

trie: minor polishes on stacktrie

trie/stacktrie: less rlp dancing

core/types: explain the contorsions in DeriveSha

ci: fix goimport errors

trie: clear mem on subtrie hashing

squashme: linter fix

stracktrie: use pooling, less allocs (#3)

trie: in-place hex prefix, reduce allocs and add rawNode.EncodeRLP

Reintroduce the `[]node` method, add the missing `EncodeRLP` implementation for `rawNode` and calculate the hex prefix in place.

Co-authored-by: Martin Holst Swende <martin@swende.se>

Co-authored-by: Martin Holst Swende <martin@swende.se>
2020-09-29 17:38:13 +02:00
gary rong e5defccd58
trie: extend range proof (#21250)
* trie: support non-existent right proof

* trie: improve test

* trie: minor linter fix

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-09-23 12:44:09 +03:00