Commit Graph

149 Commits

Author SHA1 Message Date
Felix Lange bc6569462d
p2p: use netip.Addr where possible (#29891)
enode.Node was recently changed to store a cache of endpoint information. The IP address in the cache is a netip.Addr. I chose that type over net.IP because it is just better. netip.Addr is meant to be used as a value type. Copying it does not allocate, it can be compared with ==, and can be used as a map key.

This PR changes most uses of Node.IP() into Node.IPAddr(), which returns the cached value directly without allocating.
While there are still some public APIs left where net.IP is used, I have converted all code used internally by p2p/discover to the new types. So this does change some public Go API, but hopefully not APIs any external code actually uses.

There weren't supposed to be any semantic differences resulting from this refactoring, however it does introduce one: In package p2p/netutil we treated the 0.0.0.0/8 network (addresses 0.x.y.z) as LAN, but netip.Addr.IsPrivate() doesn't. The treatment of this particular IP address range is controversial, with some software supporting it and others not. IANA lists it as special-purpose and invalid as a destination for a long time, so I don't know why I put it into the LAN list. It has now been marked as special in p2p/netutil as well.
2024-06-05 19:31:04 +02:00
hattizai 2f06c1e854
cmd/devp2p: fix node.TCP -> node.UDP (#29879) 2024-05-29 15:55:51 +02:00
Felix Lange 94a8b296e4
p2p/discover: refactor node and endpoint representation (#29844)
Here we clean up internal uses of type discover.node, converting most code to use
enode.Node instead. The discover.node type used to be the canonical representation of
network hosts before ENR was introduced. Most code worked with *node to avoid conversions
when interacting with Table methods. Since *node also contains internal state of Table and
is a mutable type, using *node outside of Table code is prone to data races. It's also
cleaner not having to wrap/unwrap *enode.Node all the time.

discover.node has been renamed to tableNode to clarify its purpose.

While here, we also change most uses of net.UDPAddr into netip.AddrPort. While this is
technically a separate refactoring from the *node -> *enode.Node change, it is more
convenient because *enode.Node handles IP addresses as netip.Addr. The switch to package
netip in discovery would've happened very soon anyway.

The change to netip.AddrPort stops at certain interface points. For example, since package
p2p/netutil has not been converted to use netip.Addr yet, we still have to convert to
net.IP/net.UDPAddr in a few places.
2024-05-29 15:02:26 +02:00
Felix Lange 6a9158bb1b
p2p/discover: improved node revalidation (#29572)
Node discovery periodically revalidates the nodes in its table by sending PING, checking
if they are still alive. I recently noticed some issues with the implementation of this
process, which can cause strange results such as nodes dropping unexpectedly, certain
nodes not getting revalidated often enough, and bad results being returned to incoming
FINDNODE queries.

In this change, the revalidation process is improved with the following logic:

- We maintain two 'revalidation lists' containing the table nodes, named 'fast' and 'slow'.
- The process chooses random nodes from each list on a randomized interval, the interval being
  faster for the 'fast' list, and performs revalidation for the chosen node.
- Whenever a node is newly inserted into the table, it goes into the 'fast' list.
  Once validation passes, it transfers to the 'slow' list. If a request fails, or the
  node changes endpoint, it transfers back into 'fast'.
- livenessChecks is incremented by one for successful checks. Unlike the old implementation,
  we will not drop the node on the first failing check. We instead quickly decay the
  livenessChecks give it another chance.
- Order of nodes in bucket doesn't matter anymore.

I am also adding a debug API endpoint to dump the node table content.

Co-authored-by: Martin HS <martin@swende.se>
2024-05-23 14:26:09 +02:00
rjl493456442 9f96e07c1c
core/rawdb, trie: improve db APIs for accessing trie nodes (#29362)
* core/rawdb, trie: improve db APIs for accessing trie nodes

* triedb/pathdb: fix
2024-04-30 16:25:35 +02:00
Aaron Chen ed4bc7f27b
all: replace fmt.Errorf() with errors.New() if no param required (#29472) 2024-04-08 12:59:17 +02:00
Aaron Chen 74995bf8a1
all: use slices.Contains (#29459)
Co-authored-by: Felix Lange <fjl@twurst.com>
2024-04-06 12:05:06 +02:00
Martin HS 14cc967d19
all: remove dependency on golang.org/exp (#29314)
This change includes a leftovers from https://github.com/ethereum/go-ethereum/pull/29307
- using the [new `slices` package](https://go.dev/doc/go1.21#slices) and
- using the [new `cmp.Ordered`](https://go.dev/doc/go1.21#cmp) instead of exp `constraints.Ordered`
2024-03-25 07:50:18 +01:00
Ng Wei Han 20d3e0ac06
cmd/devp2p: fix decoding of raw RLP ENR attributes (#29257) 2024-03-14 10:32:49 +01:00
Martin HS d5bacfa4de
crypto/kz4844: pass blobs by ref (#29050)
This change makes use of the following underlying changes to the kzg-libraries in order to avoid passing large things on the stack:

- c-kzg: https://github.com/ethereum/c-kzg-4844/pull/393 and
- go-kzg: https://github.com/crate-crypto/go-kzg-4844/pull/63
2024-03-13 07:51:46 +01:00
Bin 89cefe240f
cmd: use package filepath over path for file system operations (#29227)
Package filepath implements utility routines for manipulating filename paths in a way compatible with the target operating system-defined file paths.

Package path implements utility routines for manipulating slash-separated paths.

The path package should only be used for paths separated by forward slashes, such as the paths in URLs
2024-03-12 10:00:34 +01:00
Felix Lange a97d622588
cmd/devp2p: fix commandHasFlag (#29091)
It got broken in some update of the cli library, and thus bootnodes weren't 
being configured automatically for some of the discovery commands.
2024-03-04 14:07:41 +01:00
buddho 679a27a2b3
all: use EmptyUncleHash, EmptyCodeHash instead of raw value (#29134) 2024-03-04 10:31:18 +01:00
Felix Lange 1bdf8b9b2d
cmd/devp2p/internal/ethtest: some fixes for the eth test suite (#28996)
Improving two things here:

On hive, where we look at these tests, the Go code comment above the test
is not visible. When there is a failure, it's not obvious what the test is actually
expecting. I have converted the comments in to printed log messages to
explain the test more.

Second, I noticed that besu is failing some tests because it happens to request
a header when we want it to send transactions. Trying the minimal fix here to
serve the headers.

Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com>
2024-02-15 19:43:37 +01:00
colin 2a1d94bd1d
cmd/devp2p: fix modulo in makeBlobTxs (#28970) 2024-02-15 10:22:03 +01:00
Péter Szilágyi 8a76a814a2
cmd/devp2p, eth: drop support for eth/67 (#28956) 2024-02-08 15:49:19 +02:00
Sina Mahmoodi a608c0ac84
cmd/devp2p/internal/ethtest: skip large tx test on github build (#28794)
This test was failling consistently on the github 32-bit build probably due to slow IO. Skipping it for that green check.
2024-01-12 15:14:03 +01:00
vuittont60 ae4ea047e3
cmd: fix typos (#28798) 2024-01-12 10:40:00 +02:00
vuittont60 f29520ffdf
cmd/devp2p/internal/ethtest: fix typos in comments (#28772) 2024-01-08 20:31:22 +01:00
lightclient 577be37e0e
cmd/devp2p: update eth/snap protocol test suites for PoS (#28340)
Here we update the eth and snap protocol test suites with a new test chain,
created by the hivechain tool. The new test chain uses proof-of-stake. As such,
tests using PoW block propagation in the eth protocol are removed. The test suite
now connects to the node under test using the engine API in order to make it
accept transactions. 

The snap protocol test suite has been rewritten to output test descriptions and
log requests more verbosely.

---------

Co-authored-by: Felix Lange <fjl@twurst.com>
2023-12-20 17:23:48 +01:00
jwasinger 28e7371701
all: replace log15 with slog (#28187)
This PR replaces Geth's logger package (a fork of [log15](https://github.com/inconshreveable/log15)) with an implementation using slog, a logging library included as part of the Go standard library as of Go1.21.

Main changes are as follows:
* removes any log handlers that were unused in the Geth codebase.
* Json, logfmt, and terminal formatters are now slog handlers.
* Verbosity level constants are changed to match slog constant values.  Internal translation is done to make this opaque to the user and backwards compatible with existing `--verbosity` and `--vmodule` options.
* `--log.backtraceat` and `--log.debug` are removed.

The external-facing API is largely the same as the existing Geth logger.  Logger method signatures remain unchanged.

A small semantic difference is that a `Handler` can only be set once per `Logger` and not changed dynamically.  This just means that a new logger must be instantiated every time the handler of the root logger is changed.

----
For users of the `go-ethereum/log` module. If you were using this module for your own project, you will need to change the initialization. If you previously did 
```golang
log.Root().SetHandler(log.LvlFilterHandler(log.LvlInfo, log.StreamHandler(os.Stderr, log.TerminalFormat(true))))
```
You now instead need to do 
```golang
log.SetDefault(log.NewLogger(log.NewTerminalHandlerWithLevel(os.Stderr, log.LevelInfo, true)))
```
See more about reasoning here: https://github.com/ethereum/go-ethereum/issues/28558#issuecomment-1820606613
2023-11-29 08:33:50 +01:00
Martin Holst Swende eec37e3b71
cmd/devp2p/internal/ethtest: undo debug-hack (#28588)
cmd/devp2p/internal/ethtest: remove a debug-hack flaw which prevented certain tests from running
2023-11-23 09:22:09 +01:00
Håvard Anda Estensen 460cc1673e
cmd: run tests in parallel (#28546) 2023-11-20 10:52:14 +01:00
Zoro 2814ee0547
accounts,cmd,console,les,metrics: refactor some errors checked by (ST1005) go-staticcheck (#28532)
fix: fix some (ST1005)go-staticcheck
2023-11-15 14:36:57 +02:00
Péter Szilágyi 425cb6f65d
go.mod: pull in the latest cloudflare API libs (#28336) 2023-10-13 17:08:38 +03:00
Martin Holst Swende f62c58f8de
trie: make rhs-proof align with last key in range proofs (#28311)
During snap-sync, we request ranges of values: either a range of accounts or a range of storage values. For any large trie, e.g. the main account trie or a large storage trie, we cannot fetch everything at once.

Short version; we split it up and request in multiple stages. To do so, we use an origin field, to say "Give me all storage key/values where key > 0x20000000000000000". When the server fulfils this, the server provides the first key after origin, let's say 0x2e030000000000000 -- never providing the exact origin. However, the client-side needs to be able to verify that the 0x2e03.. indeed is the first one after 0x2000.., and therefore the attached proof concerns the origin, not the first key.

So, short-short version: the left-hand side of the proof relates to the origin, and is free-standing from the first leaf.

On the other hand, (pun intended), the right-hand side, there's no such 'gap' between "along what path does the proof walk" and the last provided leaf. The proof must prove the last element (unless there are no elements).

Therefore, we can simplify the semantics for trie.VerifyRangeProof by removing an argument. This doesn't make much difference in practice, but makes it so that we can remove some tests. The reason I am raising this is that the upcoming stacktrie-based verifier does not support such fancy features as standalone right-hand borders.
2023-10-13 16:05:29 +02:00
rjl493456442 1cb3b6aee4
eth/protocols/snap: fix snap sync failure on empty storage range (#28306)
This change addresses an issue in snap sync, specifically when the entire sync process can be halted due to an encountered empty storage range.

Currently, on the snap sync client side, the response to an empty (partial) storage range is discarded as a non-delivery. However, this response can be a valid response, when the particular range requested does not contain any slots.

For instance, consider a large contract where the entire key space is divided into 16 chunks, and there are no available slots in the last chunk [0xf] -> [end]. When the node receives a request for this particular range, the response includes:

    The proof with origin [0xf]
    A nil storage slot set

If we simply discard this response, the finalization of the last range will be skipped, halting the entire sync process indefinitely. The test case TestSyncWithUnevenStorage can reproduce the scenario described above.

In addition, this change also defines the common variables MaxAddress and MaxHash.
2023-10-13 09:08:26 +02:00
Martin Holst Swende 6b1e4f4211
all: move light.NodeSet to trienode.ProofSet (#28287)
This is a minor refactor in preparation of changes to range verifier. This PR contains no intentional functional changes but moves (and renames) the light.NodeSet
2023-10-10 10:30:47 +02:00
Péter Szilágyi bc6d184872
cmd/devp2p, eth: drop eth/66 (#28239)
* cmd/devp2p, eth: drop eth/66

* eth/protocols/eth: yes sir, linter
2023-10-03 15:03:19 +03:00
rjl493456442 73f5bcb75b
core, accounts, eth, trie: handle genesis state missing (#28171)
* core, accounts, eth, trie: handle genesis state missing

* core, eth, trie: polish

* core: manage txpool subscription in mainpool

* eth/backend: fix test

* cmd, eth: fix test

* core/rawdb, trie/triedb/pathdb: address comments

* eth, trie: address comments

* eth: inline the function

* eth: use synced flag

* core/txpool: revert changes in txpool

* core, eth, trie: rename functions
2023-09-28 10:00:53 +03:00
Delweng 41a0ad9f03
cmd/devp2p: use bootnodes as crawl input (#28139)
This PR makes the tool use the --bootnodes list as the input to devp2p crawl.
The flag will take effect if the input/output.json file is missing or empty.
2023-09-19 14:18:29 +02:00
lightclient eff7c3bda0
core/forkid: skip genesis forks by time (#28034)
* core/forkid: skip genesis forks by time

* core/forkid: add comment about skipping non-zero fork times

* core/forkid: skip all time based forks in genesis using loop

* core/forkid: simplify logic for dropping time-based forks
2023-09-04 09:32:14 -04:00
Péter Szilágyi 6b98d18789
cmd, core, params: add support for the Holesky testnet (#28007)
* cmd, core, params: add support for the Holesky testnet

* cmd/devp2p: add support for holesky for the dns crawler
2023-08-25 18:11:40 +03:00
lightclient 32fde3f838
core/forkid: correctly compute forkid when timestamp fork is activated in genesis (#27895)
This changes the forkID calculation to ignore time-based forks that occurred before the
genesis block. It's supposed to be done this way because the spec says:

> If a chain is configured to start with a non-Frontier ruleset already in its genesis, that is NOT considered a fork.
2023-08-16 23:31:02 +02:00
Péter Szilágyi be65b47645
all: update golang/x/ext and fix slice sorting fallout (#27909)
The Go authors updated golang/x/ext to change the function signature of the slices sort method. 
It's an entire shitshow now because x/ext is not tagged, so everyone's codebase just 
picked a new version that some other dep depends on, causing our code to fail building.

This PR updates the dep on our code too and does all the refactorings to follow upstream...
2023-08-12 00:04:12 +02:00
ucwong a196f3e8a2
cmd/devp2p: atomic types used (#27755) 2023-07-24 13:23:38 +03:00
rjl493456442 59f7b289c3
cmd, core, eth, graphql, trie: no persisted clean trie cache file (#27525)
The clean trie cache is persisted periodically, therefore Geth can
quickly warmup the cache in next restart.

However it will reduce the robustness of system. The assumption is
held in Geth that if the parent trie node is present, then the entire
sub-trie associated with the parent are all prensent.

Imagine the scenario that Geth rewinds itself to a past block and
restart, but Geth finds the root node of "future state" in clean
cache then regard this state is present in disk, while is not in fact.

Another example is offline pruning tool. Whenever an offline pruning
is performed, the clean cache file has to be removed to aviod hitting
the root node of "deleted states" in clean cache.

All in all, compare with the minor performance gain, system robustness
is something we care more.
2023-07-04 10:21:06 +03:00
Dan Laine 4544dc5f32
cmd/devp2p: use slices package for sorting (#27487) 2023-06-19 08:42:49 +02:00
Péter Szilágyi c7c84ca16c
all: remove the Rinkeby testnet (#27406) 2023-06-02 14:03:21 +03:00
Delweng e9c3183c52
cmd: use errrors.New instead of empty fmt.Errorf (#27329)
Signed-off-by: jsvisa <delweng@gmail.com>
2023-05-24 12:21:29 +02:00
Péter Szilágyi dde2da0efb
all: remove ethash pow, only retain shims needed for consensus and tests (#27178)
* all: remove ethash pow, only retain shims needed for consensus and tests

* all: thank you linter

* all: disallow launching Geth in legacy PoW mode

* cmd/env/internal/t8ntool: remove dangling ethash flag
2023-05-03 12:58:39 +03:00
Martin Holst Swende 2b0a34bea6
cmd/devp2p: make crawler-route53-updater less verbose (#27116)
Follow-up to #26697, makes the crawler less verbose on route53-based scenarios.

It also changes the loglevel from debug to info on Updates, which are typically the root, and can be interesting to see.
2023-04-19 06:46:56 -04:00
noel 8fe807c8f2
cmd/devp2p: fix erroneous log output in crawler (#27089)
cmd/devp2p: fix log of ignored recent nodes counter
2023-04-17 10:29:27 -04:00
Felix Lange f86913bc3e
cmd/devp2p, cmd/geth: add version in --help output (#26895)
Not sure why this was removed, it's pretty useful to see the version
also in --help.
2023-03-15 14:34:36 +01:00
Martin Holst Swende c155c8e179
cmd/devp2p: faster crawling + less verbose dns updates (#26697)
This improves the speed of DHT crawling by using concurrent requests.
It also removes logging of individual DNS updates.
2023-02-27 11:36:26 +01:00
rjl493456442 fe01a2f63b
all: use unified emptyRootHash and emptyCodeHash (#26718)
The EmptyRootHash and EmptyCodeHash are defined everywhere in the codebase, this PR replaces all of them with unified one defined in core/types package, and also defines constants for TxRoot, WithdrawalsRoot and UncleRoot
2023-02-21 06:12:27 -05:00
Martin Holst Swende 1c5fa40a78
cmd/devp2p: reduce output of node crawler (#26674)
Our discovery crawler spits out a huge amount of logs, most of which is pretty non-interesting. This change moves the very verbose output to Debug, and adds a 8-second status log message giving the general idea about what's going on.
2023-02-14 03:08:06 -05:00
Péter Szilágyi 095e365fac
all: remove support for Ropsten (#26644) 2023-02-09 10:03:00 +02:00
Pascal Marco Caversaccio 3ff3d07e2c
cmd/devp2p: fix broken link in readme(#26576)
fix broken link to DNS discovery tutorial
2023-01-30 09:12:55 -05:00
Péter Szilágyi 2189773093
Merge pull request #25878 from MariusVanDerWijden/shanghai-by-time
params: core: enable shanghai based on timestamps
2023-01-06 16:10:12 +02:00