Commit Graph

208 Commits

Author SHA1 Message Date
Guillaume Ballet 43cbcd78ea
core, core/state: move TriesInMemory to state package (#29701) 2024-05-06 13:28:53 +02:00
Péter Szilágyi 682ee820fa
core/state: parallelise parts of state commit (#29681)
* core/state, internal/workerpool: parallelize parts of state commit

* core, internal: move workerpool into syncx

* core/state: use errgroups, commit accounts concurrently

* core: resurrect detailed commit timers to almost-accuracy
2024-05-02 11:18:27 +03:00
Péter Szilágyi 4253030ef6
core/state: move metrics out of state objects (#29665) 2024-04-26 18:35:52 +03:00
Martin HS 0d4c38865e
core/state: remove account reset operation v2 (#29520)
* core/state, tests: remove account reset operation

* core/state, core/vm: implement createcontract journal event

* core/state: make createcontract not emit dirtied account, unskip tests

* core/state: add createcontract to journal fuzzing

* core/state: fix journal

* core/state: address comments

* core/state: remove useless code

---------

Co-authored-by: Gary Rong <garyrong0905@gmail.com>
2024-04-24 12:59:06 +03:00
Aaron Chen 1e9bf2a09e
core/state: fix bug in statedb.Copy and remove unnecessary preallocation (#29563)
This change removes an unnecessary preallocation and fixes a flaw with no-op copies of some parts of the statedb
2024-04-17 13:55:31 +02:00
rjl493456442 d3c4466edd
core, eth/protocols/snap, trie: fix cause for snap-sync corruption, implement gentrie (#29313)
This pull request defines a gentrie for snap sync purpose.

The stackTrie is used to generate the merkle tree nodes upon receiving a state batch. Several additional options have been added into stackTrie to handle incomplete states (either missing states before or after).

In this pull request, these options have been relocated from stackTrie to genTrie, which serves as a wrapper for stackTrie specifically for snap sync purposes.

Further, the logic for managing incomplete state has been enhanced in this change. Originally, there are two cases handled:

-    boundary node filtering
-    internal (covered by extension node) node clearing

This changes adds one more:
 
- Clearing leftover nodes on the boundaries.

This feature is necessary if there are leftover trie nodes in database, otherwise node inconsistency may break the state healing.
2024-04-16 09:05:36 +02:00
cui ab6419ccd8
core/state: use maps.Clone (#29365)
core: using maps.Clone
2024-04-02 15:56:12 +03:00
cui 7481398a24
core/state: using slices.Clone (#29366) 2024-03-28 12:13:41 +01:00
Guillaume Ballet da7469e5c4
core: add an end-to-end verkle test (#29262)
core: add a simple verkle test

triedb, core: skip hash comparison in verkle

core: remove legacy daoFork logic in verkle chain maker

fix: nil pointer in tests

triedb/pathdb: add blob hex

core: less defensive

Co-authored-by: Ignacio Hagopian <jsign.uy@gmail.com>
Co-authored-by: Martin HS <martin@swende.se>
Co-authored-by: Gary Rong <garyrong0905@gmail.com>
2024-03-26 21:25:41 +01:00
jwasinger 58a3e2f180
core/state: perform updates before deletions when mutating tries (#29201)
This addresses an edge-case (detailed in the code comment) where the computation of the intermediate trie root would force the unnecessary resolution of a hash node. The change makes it so that when we process changes from a block, we first process trie-updates and afterwards process trie-deletions.
2024-03-26 15:21:39 +01:00
Sina M 064f37d6f6
eth/tracers: live chain tracing with hooks (#29189)
Here we add a Go API for running tracing plugins within the main block import process. 

As an advanced user of geth, you can now create a Go file in eth/tracers/live/, and within
that file register your custom tracer implementation. Then recompile geth and select your tracer
on the command line. Hooks defined in the tracer will run whenever a block is processed.

The hook system is defined in package core/tracing. It uses a struct with callbacks, instead of 
requiring an interface, for several reasons:

- We plan to keep this API stable long-term. The core/tracing hook API does not depend on
  on deep geth internals.
- There are a lot of hooks, and tracers will only need some of them. Using a struct allows you
   to implement only the hooks you want to actually use.

All existing tracers in eth/tracers/native have been rewritten to use the new hook system.

This change breaks compatibility with the vm.EVMLogger interface that we used to have.
If you are a user of vm.EVMLogger, please migrate to core/tracing, and sorry for breaking
your stuff. But we just couldn't have both the old and new tracing APIs coexist in the EVM.

---------

Co-authored-by: Matthieu Vachon <matthieu.o.vachon@gmail.com>
Co-authored-by: Delweng <delweng@gmail.com>
Co-authored-by: Martin HS <martin@swende.se>
2024-03-22 18:53:53 +01:00
Péter Szilágyi b393ad8d29
cmd, core, metrics: always report expensive metrics (#29191)
* cmd, core, metrics: always report expensive metrics

* core, metrics: report block processing metrics as resetting timer

* metrics: update reporter tests
2024-03-11 10:06:57 +02:00
rjl493456442 7b81cf6362
core/state, trie/triedb/pathdb: remove storage incomplete flag (#28940)
As SELF-DESTRUCT opcode is disabled in the cancun fork(unless the
account is created within the same transaction, nothing to delete
in this case). The account will only be deleted in the following
cases:

- The account is created within the same transaction. In this case
the original storage was empty.

- The account is empty(zero nonce, zero balance, zero code) and
is touched within the transaction. Fortunately this kind of accounts
are not-existent on ethereum-mainnet.

All in all, after cancun, we are pretty sure there is no large contract
deletion and we don't need this mechanism for oom protection.
2024-03-05 14:31:55 +01:00
Martin HS a5a4fa7032
all: use uint256 in state (#28598)
This change makes use of uin256 to represent balance in state. It touches primarily upon statedb, stateobject and state processing, trying to avoid changes in transaction pools, core types, rpc and tracers.
2024-01-23 14:51:58 +01:00
Darioush Jalali 29b73555ae
core/state: unexport GetOrNewStateObject (#28804) 2024-01-14 12:32:23 +01:00
cygaar d2e3cb894b
core/state: logic equivalence for GetCodeHash (#28733) 2023-12-26 16:38:11 +08:00
rjl493456442 ab0eb46a84
core/state: make stateobject.create selfcontain (#28459) 2023-11-29 16:07:51 +08:00
Guillaume Ballet fa8d39807d
cmd, core, trie: verkle-capable `geth init` (#28270)
This change allows the creation of a genesis block for verkle testnets. This makes for a chunk of code that is easier to review and still touches many discussion points.
2023-11-14 13:09:40 +01:00
rjl493456442 1b1611b8d0
core, trie, eth: refactor stacktrie constructor (#28350)
This change enhances the stacktrie constructor by introducing an option struct. It also simplifies the `Hash` and `Commit` operations, getting rid of the special handling round root node.
2023-10-17 14:09:25 +02:00
Martin Holst Swende 8976a0c97a
trie: remove owner and binary marshaling from stacktrie (#28291)
This change
  - Removes the owner-notion from a stacktrie; the owner is only ever needed for comitting to the database, but the commit-function, the `writeFn` is provided by the caller, so the caller can just set the owner into the `writeFn` instead of having it passed through the stacktrie.
  - Removes the `encoding.BinaryMarshaler`/`encoding.BinaryUnmarshaler` interface from stacktrie. We're not using it, and it is doubtful whether anyone downstream is either.
2023-10-11 06:12:45 +02:00
Darioush Jalali 48fdb79de5
core/state: check err for iter.Error in fastDeleteStorage (#28122)
core/state: check err for iter.Error
2023-09-15 14:09:07 +08:00
Martin Holst Swende 8b6cf128af
metrics: refactor metrics (#28035)
This change includes a lot of things, listed below. 

### Split up interfaces, write vs read

The interfaces have been split up into one write-interface and one read-interface, with `Snapshot` being the gateway from write to read. This simplifies the semantics _a lot_. 

Example of splitting up an interface into one readonly 'snapshot' part, and one updatable writeonly part: 

```golang
type MeterSnapshot interface {
	Count() int64
	Rate1() float64
	Rate5() float64
	Rate15() float64
	RateMean() float64
}

// Meters count events to produce exponentially-weighted moving average rates
// at one-, five-, and fifteen-minutes and a mean rate.
type Meter interface {
	Mark(int64)
	Snapshot() MeterSnapshot
	Stop()
}
```

### A note about concurrency

This PR makes the concurrency model clearer. We have actual meters and snapshot of meters. The `meter` is the thing which can be accessed from the registry, and updates can be made to it. 

- For all `meters`, (`Gauge`, `Timer` etc), it is assumed that they are accessed by different threads, making updates. Therefore, all `meters` update-methods (`Inc`, `Add`, `Update`, `Clear` etc) need to be concurrency-safe. 
- All `meters` have a `Snapshot()` method. This method is _usually_ called from one thread, a backend-exporter. But it's fully possible to have several exporters simultaneously: therefore this method should also be concurrency-safe. 

TLDR: `meter`s are accessible via registry, all their methods must be concurrency-safe. 

For all `Snapshot`s, it is assumed that an individual exporter-thread has obtained a `meter` from the registry, and called the `Snapshot` method to obtain a readonly snapshot. This snapshot is _not_ guaranteed to be concurrency-safe. There's no need for a snapshot to be concurrency-safe, since exporters should not share snapshots. 

Note, though: that by happenstance a lot of the snapshots _are_ concurrency-safe, being unmutable minimal representations of a value. Only the more complex ones are _not_ threadsafe, those that lazily calculate things like `Variance()`, `Mean()`.

Example of how a background exporter typically works, obtaining the snapshot and sequentially accessing the non-threadsafe methods in it: 
```golang
		ms := metric.Snapshot()
                ...
		fields := map[string]interface{}{
			"count":    ms.Count(),
			"max":      ms.Max(),
			"mean":     ms.Mean(),
			"min":      ms.Min(),
			"stddev":   ms.StdDev(),
			"variance": ms.Variance(),
```

TLDR: `snapshots` are not guaranteed to be concurrency-safe (but often are).

### Sample changes

I also changed the `Sample` type: previously, it iterated the samples fully every time `Mean()`,`Sum()`, `Min()` or `Max()` was invoked. Since we now have readonly base data, we can just iterate it once, in the constructor, and set all four values at once. 

The same thing has been done for runtimehistogram. 

### ResettingTimer API

Back when ResettingTImer was implemented, as part of https://github.com/ethereum/go-ethereum/pull/15910, Anton implemented a `Percentiles` on the new type. However, the method did not conform to the other existing types which also had a `Percentiles`. 

1. The existing ones, on input, took `0.5` to mean `50%`. Anton used `50` to mean `50%`. 
2. The existing ones returned `float64` outputs, thus interpolating between values. A value-set of `0, 10`, at `50%` would return `5`, whereas Anton's would return either `0` or `10`. 

This PR removes the 'new' version, and uses only the 'legacy' percentiles, also for the ResettingTimer type. 

The resetting timer snapshot was also defined so that it would expose the internal values. This has been removed, and getters for `Max, Min, Mean` have been added instead. 

### Unexport types

A lot of types were exported, but do not need to be. This PR unexports quite a lot of them.
2023-09-13 13:13:47 -04:00
rjl493456442 a7842c9cae
core, trie: cleanup trie database (#28062) 2023-09-07 21:17:14 +08:00
rjl493456442 0acc0a1f86
core/state: simplify storage trie update and commit (#28030)
This change improves function description and simplifies logic in statedb update and commit operations.
2023-08-31 14:33:18 -04:00
Martin Holst Swende d4e345c7d4
core/state: fix missing import (#28010) 2023-08-26 04:43:36 -04:00
rjl493456442 3ff6b3c31e
core/state: implement fast storage deletion (#27955)
This changes implements faster post-selfdestruct iteration of storage slots for deletion, by using snapshot-storage+stacktrie to recover the trienodes to be deleted. This mechanism is only implemented for path-based schema. 

For hash-based schema, the entire post-selfdestruct storage iteration is skipped, with this change, since hash-based does not actually perform deletion anyway. 

---------

Co-authored-by: Martin Holst Swende <martin@swende.se>
2023-08-26 04:13:22 -04:00
Martin Holst Swende eb6cbe37e1
core/state: remove public method ForEachStorage (#27986)
Co-authored-by: Felix Lange <fjl@twurst.com>
2023-08-23 21:53:31 +02:00
Guillaume Ballet 67979022aa
core/state: move UpdateContractCode before the trie hash is computed (#27853)
Context: The UpdateContractCode method was introduced for the state storage commitment
schemes that include the whole code for their commitment computation. It must therefore be called
before the root hash is computed at the end of IntermediateRoot.

This should have no impact on the MPT since, in this context, the method is a no-op.
2023-08-09 18:02:45 +02:00
rjl493456442 7de748d3f6
all: implement path-based state scheme (#25963)
* all: implement path-based state scheme

* all: edits from review

* core/rawdb, trie/triedb/pathdb: review changes

* core, light, trie, eth, tests: reimplement pbss history

* core, trie/triedb/pathdb: track block number in state history

* trie/triedb/pathdb: add history documentation

* core, trie/triedb/pathdb: address comments from Peter's review

Important changes to list:

- Cache trie nodes by path in clean cache
- Remove root->id mappings when history is truncated

* trie/triedb/pathdb: fallback to disk if unexpect node in clean cache

* core/rawdb: fix tests

* trie/triedb/pathdb: rename metrics, change clean cache key

* trie/triedb: manage the clean cache inside of disk layer

* trie/triedb/pathdb: move journal function

* trie/triedb/path: fix tests

* trie/triedb/pathdb: fix journal

* trie/triedb/pathdb: fix history

* trie/triedb/pathdb: try to fix tests on windows

* core, trie: address comments

* trie/triedb/pathdb: fix test issues

---------

Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
2023-08-01 15:17:32 +03:00
rjl493456442 817553cc28
core, trie: track state change set with account address (#27815) 2023-07-31 15:07:51 +03:00
rjl493456442 88f3d61468
all: expose block number information to statedb (#27753)
* core/state: clean up

* all: add block number infomration to statedb

* core, trie: rename blockNumber to block
2023-07-24 13:22:09 +03:00
jwasinger 988d84aa7c
core/state, core/vm: implement EIP 6780 (#27189)
EIP-6780: SELFDESTRUCT only in same transaction

>     SELFDESTRUCT will recover all funds to the caller but not delete the account, except when called in the same transaction as creation

---------

Co-authored-by: Martin Holst Swende <martin@swende.se>
2023-07-17 13:02:18 -04:00
jwasinger d233b6b23a
core: replace instances of 'suicide' with 'selfdestruct' to improve code consistency. (#27716)
---------

Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com>
2023-07-15 10:35:30 -04:00
rjl493456442 4b06e4f25e
core/state: value diff tracking in StateDB (#27349)
This change makes the StateDB track the state key value diff of a block transition.
We already tracked current account and storage values for the purpose of updating
the state snapshot. With this PR, we now also track the original (pre-transition) values
of accounts and storage slots.
2023-07-11 15:43:23 +02:00
Guillaume Ballet c7b099b2ea
trie, core/state: revert error removal in (*state.Trie).Commit (#27544)
* trie, core/state: revert error removal in (*state.Trie).Commit

* Gary's nitpick :)

Co-Authored-By:  rjl493456442 <garyrong0905@gmail.com>

---------

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
2023-06-27 15:36:38 +03:00
Guillaume Ballet 699243f8ae
core/state, light, trie: add UpdateContractCode to the Trie interface (#27476)
Verkle trees store the code inside the trie. This PR changes the interface to pass the code, as well as the dirty flag to tell the trie package if the code is dirty and needs to be updated. This is a no-op for the MPT and the odr trie.
2023-06-22 08:52:52 -04:00
rjl493456442 6d2aeb43d5
cmd, core/state, eth, tests, trie: improve state reader (#27428)
The state availability is checked during the creation of a state reader.

-    In hash-based database, if the specified root node does not exist on disk disk, then
    the state reader won't be created and an error will be returned.

-    In path-based database, if the specified state layer is not available, then the
    state reader won't be created and an error will be returned.

This change also contains a stricter semantics regarding the `Commit` operation: once it has been performed, the trie is no longer usable, and certain operations will return an error.
2023-06-20 15:31:45 -04:00
rjl493456442 ceca4578ca
trie: remove parameter 'fromLevel' in Prove (#27512)
This removes the feature where top nodes of the proof can be elided.
It was intended to be used by the LES server, to save bandwidth 
when the client had already fetched parts of the state and only needed
some extra nodes to complete the proof. Alas, it never got implemented
in the client.
2023-06-19 16:28:40 +02:00
rjl493456442 0e5d2c7c53
core/state/snapshot, core/types, eth: move account definition to type (#27323)
* core/state/snapshot, core/types, eth: move account definition to type

* core, eth: revert snapshot Account API change
2023-06-06 11:17:39 +03:00
rjl493456442 380fb4e249
core/state: clear out cached state data when reset occurs (#27376)
* core/state: remove cached snap data if reset occurs

* core/state: address comment from peter

* core/state: skip revert in case data is nil
2023-06-05 16:25:57 +03:00
rjl493456442 15bd21f3c8
core/state: mark account as dirty when resetObject occurs (#27339)
This changes the journal logic to mark the state object dirty immediately when it
is reset. 

We're mostly adding this change to appease the fuzzer. Marking it dirty immediately
makes no difference in practice because accounts will always be modified by EVM
right after creation.
2023-06-01 11:09:32 +02:00
rjl493456442 5021d36d35
all: port boring changes from pbss (#27176)
* all: port boring changes from pbss

* core, trie: address comments from martin

* trie: minor fixes

* core/rawdb: update comment

* core, eth, tests, trie: address comments

* tests, trie: add extra check when update trie database

* trie/triedb/hashdb: degrade the error to warning
2023-05-09 10:11:04 +03:00
ucwong cc8d40c65f
core/state: initialize maps with known size (#27222)
* core/state : fix map size avoid resizing

* core/state : fixed size
2023-05-08 09:59:14 +03:00
Guillaume Ballet 41f89ca944
core/state, trie: remove Try prefix in Trie accessors (#26975)
This change renames StateTrie methods to remove the Try* prefix. 

We added the Trie methods with prefix 'Try' a long time ago, working
around the problem that most existing methods of Trie did not return the
database error. This weird naming convention has persisted until now.

Co-authored-by: Gary Rong <garyrong0905@gmail.com>
2023-03-27 10:48:46 +02:00
Guillaume Ballet 8990c92aea
core/state: add account address to Trie slot accessors (#26934)
This changes the Trie interface to add the plain account address as a
parameter to all storage-related methods.

After the introduction of the TryAccount* functions, TryGet, TryUpdate and
TryDelete are now only meant to read an account's storage. In their current
form, they assume that an account storage is stored in a separate trie, and
that the hashing of the slot is independent of its account's address.

The proposed structure for a stateless storage breaks these two
assumptions: the hashing of a slot key requires the address and all slots
and accounts are stored in a single trie.

This PR therefore adds an address parameter to the interface. It is ignored
in the MPT version, so this change has no functional impact, however it
will reduce the diff size when merging verkle trees.
2023-03-23 11:52:22 +01:00
rjl493456442 bba2a1bac5
core: show db error-info in case of mismatched hash root (#26870)
When a database failure occurs, bubble it up a into statedb, and report it in suitable places, such as during a 'bad block' report.
2023-03-16 03:12:34 -04:00
rjl493456442 c8a6b7100c
core/state, trie: port changes from PBSS (#26763) 2023-03-14 04:50:53 -04:00
rjl493456442 fe01a2f63b
all: use unified emptyRootHash and emptyCodeHash (#26718)
The EmptyRootHash and EmptyCodeHash are defined everywhere in the codebase, this PR replaces all of them with unified one defined in core/types package, and also defines constants for TxRoot, WithdrawalsRoot and UncleRoot
2023-02-21 06:12:27 -05:00
rjl493456442 13ef21d467
Revert "core/trie: remove trie tracer (#26665)" (#26732)
This reverts commit 7c749c947a.
2023-02-20 09:54:52 -05:00
Martin Holst Swende 7c749c947a
core/trie: remove trie tracer (#26665)
This PR contains a small portion of the full pbss PR, namely

    Remove the tracer from trie (and comitter), and instead using an accessList.
    Related changes to the Nodeset.


---------

Co-authored-by: Gary Rong <garyrong0905@gmail.com>
2023-02-19 14:10:19 -05:00