Commit Graph

103 Commits

Author SHA1 Message Date
rjl493456442 9dcf8aae47
eth/protocols/snap: skip retrieval for completed storages ()
* eth/protocols/snap: skip retrieval for completed storages

* eth/protocols/snap: address comments from peter

* eth/protocols/snap: add comments
2024-04-10 12:02:45 +03:00
rjl493456442 304879da20
eth/protocols/snap: check storage root existence for hash scheme () 2024-03-27 09:35:33 +08:00
Aaron Chen 723b1e36ad
all: fix mismatched names in comments ()
* all: fix mismatched names in comments

* metrics: fix mismatched name in UpdateIfGt
2024-03-26 21:01:28 +01:00
Martin HS 14cc967d19
all: remove dependency on golang.org/exp ()
This change includes a leftovers from https://github.com/ethereum/go-ethereum/pull/29307
- using the [new `slices` package](https://go.dev/doc/go1.21#slices) and
- using the [new `cmp.Ordered`](https://go.dev/doc/go1.21#cmp) instead of exp `constraints.Ordered`
2024-03-25 07:50:18 +01:00
Martin HS 04bf1c802f
eth/protocols/snap, internal/testlog: fix dataraces () 2024-03-20 15:22:52 +01:00
rjl493456442 15eb9773f9
triedb/pathdb: improve tests () 2024-03-19 10:50:08 +08:00
hyhnet cd490608e3
all: fix typos in comments () 2024-03-07 22:56:19 +01:00
Undefinedor 00905f7dc4
all: remove redundant import aliases () 2024-03-02 22:42:50 +02:00
buddho bba3fa9af9
core,eth,internal: fix typo () 2024-02-20 19:42:48 +08:00
Sina Mahmoodi 95741b1844
core: move genesis alloc types to core/types ()
We want to use these types in public user-facing APIs, so they shouldn't be in core.

Co-authored-by: Felix Lange <fjl@twurst.com>
2024-02-16 19:05:33 +01:00
rjl493456442 fe91d476ba
all: remove the dependency from trie to triedb ()
This change removes the dependency from trie package to triedb package.
2024-02-13 14:49:53 +01:00
Martin HS a5a4fa7032
all: use uint256 in state ()
This change makes use of uin256 to represent balance in state. It touches primarily upon statedb, stateobject and state processing, trying to avoid changes in transaction pools, core types, rpc and tracers.
2024-01-23 14:51:58 +01:00
Martin Holst Swende 2391fbc676
tests/fuzzers: move fuzzers into native packages ()
This PR moves our fuzzers from tests/fuzzers into whatever their respective 'native' package is.

The historical reason why they were placed in an external location, is that when they were based on go-fuzz, they could not be "hidden" via the _test.go prefix. So in order to shove them away from the go-ethereum "production code", they were put aside.

But now we've rewritten them to be based on golang testing, and thus can be brought back. I've left (in tests/) the ones that are not production (bls128381), require non-standard imports (secp requires btcec, bn256 requires gnark/google/cloudflare deps).

This PR also adds a fuzzer for precompiled contracts, because why not.

This PR utilizes a newly rewritten replacement for go-118-fuzz-build, namely gofuzz-shim, which utilises the inputs from the fuzzing engine better.
2023-11-14 14:34:29 +01:00
rjl493456442 ab04aeb855
core, eth, trie: filter out boundary nodes and remove dangling nodes in stacktrie ()
* core, eth, trie: filter out boundary nodes in stacktrie

* eth/protocol/snap: add comments

* Update trie/stacktrie.go

Co-authored-by: Martin Holst Swende <martin@swende.se>

* eth, trie: remove onBoundary callback

* eth/protocols/snap: keep complete boundary nodes

* eth/protocols/snap: skip healing if the storage trie is already complete

* eth, trie: add more metrics

* eth, trie: address comment

---------

Co-authored-by: Martin Holst Swende <martin@swende.se>
2023-10-23 18:31:56 +03:00
rjl493456442 1b1611b8d0
core, trie, eth: refactor stacktrie constructor ()
This change enhances the stacktrie constructor by introducing an option struct. It also simplifies the `Hash` and `Commit` operations, getting rid of the special handling round root node.
2023-10-17 14:09:25 +02:00
Martin Holst Swende f62c58f8de
trie: make rhs-proof align with last key in range proofs ()
During snap-sync, we request ranges of values: either a range of accounts or a range of storage values. For any large trie, e.g. the main account trie or a large storage trie, we cannot fetch everything at once.

Short version; we split it up and request in multiple stages. To do so, we use an origin field, to say "Give me all storage key/values where key > 0x20000000000000000". When the server fulfils this, the server provides the first key after origin, let's say 0x2e030000000000000 -- never providing the exact origin. However, the client-side needs to be able to verify that the 0x2e03.. indeed is the first one after 0x2000.., and therefore the attached proof concerns the origin, not the first key.

So, short-short version: the left-hand side of the proof relates to the origin, and is free-standing from the first leaf.

On the other hand, (pun intended), the right-hand side, there's no such 'gap' between "along what path does the proof walk" and the last provided leaf. The proof must prove the last element (unless there are no elements).

Therefore, we can simplify the semantics for trie.VerifyRangeProof by removing an argument. This doesn't make much difference in practice, but makes it so that we can remove some tests. The reason I am raising this is that the upcoming stacktrie-based verifier does not support such fancy features as standalone right-hand borders.
2023-10-13 16:05:29 +02:00
rjl493456442 1cb3b6aee4
eth/protocols/snap: fix snap sync failure on empty storage range ()
This change addresses an issue in snap sync, specifically when the entire sync process can be halted due to an encountered empty storage range.

Currently, on the snap sync client side, the response to an empty (partial) storage range is discarded as a non-delivery. However, this response can be a valid response, when the particular range requested does not contain any slots.

For instance, consider a large contract where the entire key space is divided into 16 chunks, and there are no available slots in the last chunk [0xf] -> [end]. When the node receives a request for this particular range, the response includes:

    The proof with origin [0xf]
    A nil storage slot set

If we simply discard this response, the finalization of the last range will be skipped, halting the entire sync process indefinitely. The test case TestSyncWithUnevenStorage can reproduce the scenario described above.

In addition, this change also defines the common variables MaxAddress and MaxHash.
2023-10-13 09:08:26 +02:00
Martin Holst Swende 8976a0c97a
trie: remove owner and binary marshaling from stacktrie ()
This change
  - Removes the owner-notion from a stacktrie; the owner is only ever needed for comitting to the database, but the commit-function, the `writeFn` is provided by the caller, so the caller can just set the owner into the `writeFn` instead of having it passed through the stacktrie.
  - Removes the `encoding.BinaryMarshaler`/`encoding.BinaryUnmarshaler` interface from stacktrie. We're not using it, and it is doubtful whether anyone downstream is either.
2023-10-11 06:12:45 +02:00
Martin Holst Swende 6b1e4f4211
all: move light.NodeSet to trienode.ProofSet ()
This is a minor refactor in preparation of changes to range verifier. This PR contains no intentional functional changes but moves (and renames) the light.NodeSet
2023-10-10 10:30:47 +02:00
Péter Szilágyi be65b47645
all: update golang/x/ext and fix slice sorting fallout ()
The Go authors updated golang/x/ext to change the function signature of the slices sort method. 
It's an entire shitshow now because x/ext is not tagged, so everyone's codebase just 
picked a new version that some other dep depends on, causing our code to fail building.

This PR updates the dep on our code too and does all the refactorings to follow upstream...
2023-08-12 00:04:12 +02:00
rjl493456442 503f1f7ada
all: activate pbss as experimental feature ()
* all: activate pbss

* core/rawdb: fix compilation error

* cma, core, eth, les, trie: address comments

* cmd, core, eth, trie: polish code

* core, cmd, eth: address comments

* cmd, core, eth, les, light, tests: address comment

* cmd/utils: shorten log message

* trie/triedb/pathdb: limit node buffer size to 1gb

* cmd/utils: fix opening non-existing db

* cmd/utils: rename flag name

* cmd, core: group chain history flags and fix tests

* core, eth, trie: fix memory leak in snapshot generation

* cmd, eth, internal: deprecate flags

* all: enable state tests for pathdb, fixes

* cmd, core: polish code

* trie/triedb/pathdb: limit the node buffer size to 256mb

---------

Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2023-08-10 22:21:36 +03:00
Péter Szilágyi 6e934f40f9
eth/protocols/snap: fix batch writer when resuming an aborted sync () 2023-08-03 14:51:02 +03:00
rjl493456442 88f3d61468
all: expose block number information to statedb ()
* core/state: clean up

* all: add block number infomration to statedb

* core, trie: rename blockNumber to block
2023-07-24 13:22:09 +03:00
rjl493456442 4b06e4f25e
core/state: value diff tracking in StateDB ()
This change makes the StateDB track the state key value diff of a block transition.
We already tracked current account and storage values for the purpose of updating
the state snapshot. With this PR, we now also track the original (pre-transition) values
of accounts and storage slots.
2023-07-11 15:43:23 +02:00
lightclient cbf2579691
p2p, p2p/discover: add dial metrics ()
This PR adds metrics for p2p dialing, which gives us visibility into the quality of the dial 
candidates  returned by our discovery methods.
2023-07-06 16:20:31 +02:00
Guillaume Ballet c7b099b2ea
trie, core/state: revert error removal in (*state.Trie).Commit ()
* trie, core/state: revert error removal in (*state.Trie).Commit

* Gary's nitpick :)

Co-Authored-By:  rjl493456442 <garyrong0905@gmail.com>

---------

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
2023-06-27 15:36:38 +03:00
rjl493456442 6d2aeb43d5
cmd, core/state, eth, tests, trie: improve state reader ()
The state availability is checked during the creation of a state reader.

-    In hash-based database, if the specified root node does not exist on disk disk, then
    the state reader won't be created and an error will be returned.

-    In path-based database, if the specified state layer is not available, then the
    state reader won't be created and an error will be returned.

This change also contains a stricter semantics regarding the `Commit` operation: once it has been performed, the trie is no longer usable, and certain operations will return an error.
2023-06-20 15:31:45 -04:00
Dan Laine b1ef0bfe03
eth: use slices package for sorting ()
Also adds Hash.Less method for sorting purposes.

---------

Co-authored-by: Felix Lange <fjl@twurst.com>
2023-06-19 23:38:57 +02:00
rjl493456442 ceca4578ca
trie: remove parameter 'fromLevel' in Prove ()
This removes the feature where top nodes of the proof can be elided.
It was intended to be used by the LES server, to save bandwidth 
when the client had already fetched parts of the state and only needed
some extra nodes to complete the proof. Alas, it never got implemented
in the client.
2023-06-19 16:28:40 +02:00
rjl493456442 0e5d2c7c53
core/state/snapshot, core/types, eth: move account definition to type ()
* core/state/snapshot, core/types, eth: move account definition to type

* core, eth: revert snapshot Account API change
2023-06-06 11:17:39 +03:00
rjl493456442 a14301823e
all: new empty trie with types.EmptyRootHash instead of null () 2023-05-11 10:19:42 +03:00
rjl493456442 5021d36d35
all: port boring changes from pbss ()
* all: port boring changes from pbss

* core, trie: address comments from martin

* trie: minor fixes

* core/rawdb: update comment

* core, eth, tests, trie: address comments

* tests, trie: add extra check when update trie database

* trie/triedb/hashdb: degrade the error to warning
2023-05-09 10:11:04 +03:00
s7v7nislands f541cad272
eth: use new atomic types () 2023-04-25 12:06:50 +02:00
rjl493456442 99f81d2724
all: refactor trie API ()
In this PR, all TryXXX(e.g. TryGet) APIs of trie are renamed to XXX(e.g. Get) with an error returned.

The original XXX(e.g. Get) APIs are renamed to MustXXX(e.g. MustGet) and does not return any error -- they print a log output. A future PR will change the behaviour to panic on errorrs.
2023-04-20 06:57:24 -04:00
Guillaume Ballet 41f89ca944
core/state, trie: remove Try prefix in Trie accessors ()
This change renames StateTrie methods to remove the Try* prefix. 

We added the Trie methods with prefix 'Try' a long time ago, working
around the problem that most existing methods of Trie did not return the
database error. This weird naming convention has persisted until now.

Co-authored-by: Gary Rong <garyrong0905@gmail.com>
2023-03-27 10:48:46 +02:00
rjl493456442 fe01a2f63b
all: use unified emptyRootHash and emptyCodeHash ()
The EmptyRootHash and EmptyCodeHash are defined everywhere in the codebase, this PR replaces all of them with unified one defined in core/types package, and also defines constants for TxRoot, WithdrawalsRoot and UncleRoot
2023-02-21 06:12:27 -05:00
Martin Holst Swende 22c3ad1d12
core/state, trie: remove unused error-return from trie Commit operation () 2023-02-09 08:56:59 -05:00
Martin Holst Swende 8860b39754
all: prepare for path-based trie storage ()
This PR moves some trie-related db accessor methods to a different file, and also removes the schema type. Instead of the schema type, a string is used to distinguish between hashbased/pathbased db accessors.
This also moves some code from trie package to rawdb package.

This PR is intended to be a no-functionality-change prep PR for  .

---------

Co-authored-by: Gary Rong <garyrong0905@gmail.com>
2023-02-06 10:28:40 -05:00
rjl493456442 6c149fd4ad
core, eth, trie, light: clean up trie interface ()
* all: cleanup trie interface

* eth, trie: address comments
2023-01-03 15:41:40 +02:00
rjl493456442 743e404906
core, eth, les, tests, trie: abstract node scheme ()
This PR introduces a node scheme abstraction. The interface is only implemented by `hashScheme` at the moment, but will be extended by `pathScheme` very soon.

Apart from that, a few changes are also included which is worth mentioning:

-  port the changes in the stacktrie, tracking the path prefix of nodes during commit
-  use ethdb.Database for constructing trie.Database. This is not necessary right now, but it is required for path-based used to open reverse diff freezer
2022-11-28 14:31:28 +01:00
Martin Holst Swende 5a02b2d6d0
all: fix spelling mistakes () 2022-10-11 09:37:00 +02:00
Péter Szilágyi 4f7a425aa8
Merge pull request from holiman/comments_fix
all: fix docstrings
2022-10-04 13:30:00 +03:00
Martin Holst Swende ee301c750b
all: fix docstrings 2022-10-04 09:18:02 +02:00
Martin Holst Swende f61b50b1e8
eth/protocols/snap: serve snap requests when possible ()
This PR makes it so that the snap server responds to trie heal requests when possible, even if the snapshot does not exist. The idea being that it might prolong the lifetime of a state root, so we don't have to pivot quite as often.
2022-10-03 13:37:17 +02:00
Martin Holst Swende 456b187892
trie, eth/protocols/snap: less tiny db-writes during trie-healing ()
This PR prevent making tiny writes during state healing, by only performing the batch-write if the accumulated data is large enough to be meaningful.
2022-09-28 08:08:18 +02:00
rjl493456442 bff84a99fe
cmd, core, eth, les, light: track deleted nodes ()
* cmd, core, eth, les, light: track deleted nodes

* trie: add docs

* trie: address comments

* cmd, core, eth, les, light, trie: trie id

* trie: add tests

* trie, core: updates

* trie: fix imports

* trie: add utility print-method for nodeset

* trie: import err

* trie: fix go vet warnings

Co-authored-by: Martin Holst Swende <martin@swende.se>
2022-09-27 10:01:02 +02:00
Martin Holst Swende fb500d12d5
eth/protocols/snap: make log messages more clear that sync is ongoing ()
* eth/protocols/snap: make log messages more clear that sync is ongoing

* Update sync.go

Co-authored-by: Felix Lange <fjl@twurst.com>
2022-09-23 11:11:41 +03:00
Felix Lange b628d72766
build: upgrade to go 1.19 ()
This changes the CI / release builds to use the latest Go version. It also
upgrades golangci-lint to a newer version compatible with Go 1.19.

In Go 1.19, godoc has gained official support for links and lists. The
syntax for code blocks in doc comments has changed and now requires a
leading tab character. gofmt adapts comments to the new syntax
automatically, so there are a lot of comment re-formatting changes in this
PR. We need to apply the new format in order to pass the CI lint stage with
Go 1.19.

With the linter upgrade, I have decided to disable 'gosec' - it produces
too many false-positive warnings. The 'deadcode' and 'varcheck' linters
have also been removed because golangci-lint warns about them being
unmaintained. 'unused' provides similar coverage and we already have it
enabled, so we don't lose much with this change.
2022-09-10 13:25:40 +02:00
Péter Szilágyi de8d5fa042
eth/protocols/snap: throttle trie heal requests when peers DoS us ()
* eth/protocols/snap: throttle trie heal requests when peers DoS us

* eth/protocols/snap: lower heal throttle log to debug

Co-authored-by: Martin Holst Swende <martin@swende.se>

* eth/protocols/snap: fix comment

Co-authored-by: Martin Holst Swende <martin@swende.se>
2022-09-09 11:42:57 +03:00
Martin Holst Swende dafa40e7a7
eth/protocols/snap: fix problems due to idle-but-busy peers 2022-08-31 17:58:18 +02:00