Here we update the eth and snap protocol test suites with a new test chain,
created by the hivechain tool. The new test chain uses proof-of-stake. As such,
tests using PoW block propagation in the eth protocol are removed. The test suite
now connects to the node under test using the engine API in order to make it
accept transactions.
The snap protocol test suite has been rewritten to output test descriptions and
log requests more verbosely.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
This is primarily to make lint work again on macOS 14. The older version of golangci-lint kept crashing.
Also included is a fix for a goroutine leak in the recently-introduced function MustRunCommandWithOutput.
Certain flags, such as `--rpc.txfeecap` currently do not have an env-var auto-generated for them. This change adds three missing cli flag types to the auto env-var helper function to fix this.
This change
- Removes interface `log.Format`,
- Removes method `log.FormatFunc`,
- unexports `TerminalHandler.TerminalFormat` formatting methods (renamed to `TerminalHandler.format`)
- removes the notion of `log.Lazy` values
The lazy handler was useful in the old log package, since it
could defer the evaluation of costly attributes until later in the
log pipeline: thus, if the logging was done at 'Trace', we could
skip evaluation if logging only was set to 'Info'.
With the move to slog, this way of deferring evaluation is no longer
needed, since slog introduced 'Enabled': the caller can thus do
the evaluate-or-not decision at the callsite, which is much more
straight-forward than dealing with lazy reflect-based evaluation.
Also, lazy evaluation would not work with 'native' slog, as in, these
two statements would be evaluated differently:
```golang
log.Info("foo", "my lazy", lazyObj)
slog.Info("foo", "my lazy", lazyObj)
```
These changes improves the performance of the non-coloured terminal formatting, _quite a lot_.
```
name old time/op new time/op delta
TerminalHandler-8 10.2µs ±15% 5.4µs ± 9% -47.02% (p=0.008 n=5+5)
name old alloc/op new alloc/op delta
TerminalHandler-8 2.17kB ± 0% 0.40kB ± 0% -81.46% (p=0.008 n=5+5)
name old allocs/op new allocs/op delta
TerminalHandler-8 33.0 ± 0% 5.0 ± 0% -84.85% (p=0.008 n=5+5)
```
I tried to _somewhat_ organize the commits, but the it might still be a bit chaotic. Some core insights:
- The function `terminalHandler.Handl` uses a mutex, and writes all output immediately to 'upstream'. Thus, it can reuse a scratch-buffer every time.
- This buffer can be propagated internally, making all the internal formatters either write directly to it,
- OR, make use of the `tmp := buf.AvailableBuffer()` in some cases, where a byte buffer "extra capacity" can be temporarily used.
- The `slog` package uses `Attr` by value. It makes sense to minimize operating on them, since iterating / collecting into a new slice, iterating again etc causes copy-on-heap. Better to operate on them only once.
- If we want to do padding, it's better to copy from a constant `space`-buffer than to invoke `bytes.Repeat` every single time.
This PR replaces Geth's logger package (a fork of [log15](https://github.com/inconshreveable/log15)) with an implementation using slog, a logging library included as part of the Go standard library as of Go1.21.
Main changes are as follows:
* removes any log handlers that were unused in the Geth codebase.
* Json, logfmt, and terminal formatters are now slog handlers.
* Verbosity level constants are changed to match slog constant values. Internal translation is done to make this opaque to the user and backwards compatible with existing `--verbosity` and `--vmodule` options.
* `--log.backtraceat` and `--log.debug` are removed.
The external-facing API is largely the same as the existing Geth logger. Logger method signatures remain unchanged.
A small semantic difference is that a `Handler` can only be set once per `Logger` and not changed dynamically. This just means that a new logger must be instantiated every time the handler of the root logger is changed.
----
For users of the `go-ethereum/log` module. If you were using this module for your own project, you will need to change the initialization. If you previously did
```golang
log.Root().SetHandler(log.LvlFilterHandler(log.LvlInfo, log.StreamHandler(os.Stderr, log.TerminalFormat(true))))
```
You now instead need to do
```golang
log.SetDefault(log.NewLogger(log.NewTerminalHandlerWithLevel(os.Stderr, log.LevelInfo, true)))
```
See more about reasoning here: https://github.com/ethereum/go-ethereum/issues/28558#issuecomment-1820606613
* eth/gasestimator: early exit for plain transfer and error allowance
* core, eth/gasestimator: hard guess at a possible required gas
* internal/ethapi: update estimation tests with the error ratio
* eth/gasestimator: I hate you linter
* graphql: fix gas estimation test
---------
Co-authored-by: Oren <orenyomtov@users.noreply.github.com>
* core/vm: set basefee to 0 internally on eth_call
* core: nicer 0-basefee, make it work for blob fees too
* internal/ethapi: make tests a bit more complex
* core: fix blob fee checker
* core: make code a bit more readable
* core: fix some test error strings
* core/vm: Get rid of weird comment
* core: dict wrong typo
a little copying is better than a little dependency
-- go proverb
We have this dependency on docker, a.k.a moby: a gigantic library, and we only need ~70 LOC,
so here I tried moving it inline instead.
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR is a bit in preparation for the slog work in #28187 .
Our current test re logging mostly test the internals, but we have no real end-to-end test of the logging output. This PR introduces a simple reexec-based log tester. This also relies upon a special mode in geth, which can be made to eject a set of predefined log messages (only available if the build-tag `integrationtests` is used
e.g. go run --tags=integrationtests ./cmd/geth --log.format terminal logtest
While working on this, I also noticed a quirk in the setup: when geth was configured to use a file output, then two separate handlers were used (one handler for the file, one handler for the console). Using two separate handlers means that two formatters are used, thus the formatting of any/all records happened twice. This PR changes the mechanism to use two separate io.Writers instead, which is both more optimal and fixes a bug which occurs due to a global statefulness in the formatter.
This change fixes#28355, where eth_getProof failed to return the correct codehash under certain conditions. This PR changes the logic to unconditionally look up the codehash, and also adds some more tests.
* build: upgrade to golang 1.21.2
* build: verify checksums via tool
* deps: upgrade go to 1.21.3
* build: move more build metadata into checksum file
* build: move gobootsrc to checksums
So apparently in the spec the base block parameter of eth_call is optional.
I agree that "latest" is a sane default for this that most people would use.
This PR makes EIP-4788 work in the engine API and miner. It also fixes some bugs related to
EIP-4844 block processing and mining. Changes in detail:
- Header.BeaconRoot has been renamed to ParentBeaconRoot.
- The engine API now implements forkchoiceUpdatedV3
- newPayloadV3 method has been updated with the parentBeaconBlockRoot parameter
- beacon root is now applied to new blocks in miner
- For EIP-4844, block creation now updates the blobGasUsed field of the header
Optimizations:
- Previously, if a transaction was reverting, EstimateGas would exhibit worst-case behavior and binary search up to the max gas limit (~40 state-clone + tx executions). This change allows EstimateGas to return after only a single unconstrained execution in this scenario.
- Uses the gas used from the unconstrained execution to bias the remaining binary search towards the likely solution in a simple way that doesn't impact the worst case. For a typical contract-invoking transaction, this reduces the median number of state-clone+executions from 25 to 18 (28% reduction).
Cleanup:
- added & improved function + code comments
- correct the EstimateGas documentation to clarify the gas limit determination is at latest block, not pending, if the blockNr is unspecified.
The Go authors updated golang/x/ext to change the function signature of the slices sort method.
It's an entire shitshow now because x/ext is not tagged, so everyone's codebase just
picked a new version that some other dep depends on, causing our code to fail building.
This PR updates the dep on our code too and does all the refactorings to follow upstream...
This adds support for the "yParity" field in transaction objects returned by RPC
APIs. We somehow forgot to add this field even though it has been in the spec for
a long time.
This change adds back the 'geth --dev' mode of operation, using a cl-mocker.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com>
This changes the eth_getProof method implementation to re-encode the requested
storage keys, canonicalizing them in the response. For backwards-compatibility reasons,
go-ethereum accepts non-canonical hex keys. Accepting them is fine, but we should
not mirror invalid inputs into the output.
Closes#27306
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
This removes the feature where top nodes of the proof can be elided.
It was intended to be used by the LES server, to save bandwidth
when the client had already fetched parts of the state and only needed
some extra nodes to complete the proof. Alas, it never got implemented
in the client.
* all: move main transaction pool into a subpool
* go.mod: remove superfluous updates
* core/txpool: review fixes, handle txs rejected by all subpools
* core/txpool: typos
Deserialize hex keys early to shortcut on invalid input, and re-use the account storageTrie for each proof for each proof in the account, preventing repeated deep-copying of the trie.
Closes#27308
--------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
RPC methods `eth_getHeaderBy*` returned a size value which was meant for internal
processes. Please instead use `size` field returned by `eth_getBlockBy*` if you're interested
in the RLP encoded storage size of the block.
Signed-off-by: jsvisa <delweng@gmail.com>
* EstimateGas should use LatestBlockNumber by default
* graphql: default to use latest for gas estimation
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
* cryto/kzg4844: pull in the C and Go libs for KZG cryptography
* go.mod: pull in the KZG libraries
* crypto/kzg4844: add basic becnhmarks for ballpark numbers
* cmd, crypto: integrate both CKZG and GoKZG all the time, add flag
* cmd/utils, crypto/kzg4844: run library init on startup
* crypto/kzg4844: make linter happy
* crypto/kzg4844: push missing file
* crypto/kzg4844: fully disable CKZG but leave in the sources
* build, crypto/kzg4844, internal: link CKZG by default and with portable mode
* crypto/kzg4844: drop verifying the trusted setup in gokzg
* internal/build: yolo until it works?
* cmd/utils: make flag description friendlier
Co-authored-by: Martin Holst Swende <martin@swende.se>
* crypto/ckzg: no need for double availability check
* build: tiny flag cleanup nitpick
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Adds an optional config parameter to eth_call which allows users to override block context fields (same functionality that was added to traceCall in #24871)
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This PR removes the Debug field from vmconfig, making it so that if a tracer is set, debug=true is implied.
---------
Co-authored-by: 0xTylerHolmes <tyler@ethereum.org>
Co-authored-by: Sina Mahmoodi <1591639+s1na@users.noreply.github.com>
This change enables log rotation, which can be activated using the flag --log.rotate. Additional parameters that can be given are:
- log.maxsize to set maximum size before files are rotated,
- log.maxbackups to set how many files are retailed,
- log.maxage to configure max age of rotated files,
- log.compress whether to compress rotated files
The way to configure location of the logfile(s) is left unchanged, via the `log.logfile` parameter.
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
Removes the new --log.logfmt directive and hides --log.json, replacing both with log.format=(json|logfmt|terminal). The hidden log.json option is still respected if log.format is not specified for backwards compatibility.
Co-authored-by: Martin Holst Swende <martin@swende.se>
Here, the core.Message interface turns into a plain struct and
types.Message gets removed.
This is a breaking change to packages core and core/types. While we do
not promise API stability for package core, we do for core/types. An
exception can be made for types.Message, since it doesn't have any
purpose apart from invoking the state transition in package core.
types.Message was also marked deprecated by the same commit it
got added in, 4dca5d4db7 (November 2016).
The core.Message interface was added in December 2014, in commit
db494170dc, for the purpose of 'testing' state transitions. It's the
same change that made transaction struct fields private. Before that,
the state transition used *types.Transaction directly.
Over time, multiple implementations of the interface accrued across
different packages, since constructing a Message is required whenever
one wants to invoke the state transition. These implementations all
looked very similar, a struct with private fields exposing the fields
as accessor methods.
By changing Message into a struct with public fields we can remove all
these useless interface implementations. It will also hopefully
simplify future changes to the type with less updates to apply across
all of go-ethereum when a field is added to Message.
---------
Co-authored-by: Felix Lange <fjl@twurst.com>
Checks that Transaction.MarshalJSON and newRPCTransaction JSON output can be parsed by Transaction.UnmarshalJSON
---------
Co-authored-by: Martin Holst Swende <martin@swende.se>
This change adds a struct field EffectiveGasPrice in types.Receipt. The field is present
in RPC responses, but not in the Go struct, and thus can't easily be accessed via ethclient.
Co-authored-by: PulsarAI <dev@pulsar-systems.fi>
Logs stored on disk have minimal information. Contextual information such as block
number, index of log in block, index of transaction in block are filled in upon request.
We can fill in all these fields only having the block header and list of receipts.
But determining the transaction hash of a log requires the block body.
The goal of this PR is postponing this retrieval until we are sure we the transaction hash.
It happens often that the header bloom filter signals there might be matches in a block,
but after actually checking them reveals the logs do not match. We want to avoid fetching
the body in this case.
Note that this changes the semantics of Backend.GetLogs. Downstream callers of
GetLogs now assume log context fields have not been derived, and need to call
DeriveFields on the logs if necessary.
The execution-apis specification says that the full list of withdrawals should always be returned when requesting a block over RPC:
378c4304f7/src/schemas/block.yaml (L90-L94)
This change adopts the expected behavior.
This change implements withdrawals as specified in EIP-4895.
Co-authored-by: lightclient@protonmail.com <lightclient@protonmail.com>
Co-authored-by: marioevz <marioevz@gmail.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR changes the API so that uint64 is used for fork timestamps.
It's a good choice because types.Header also uses uint64 for time.
Co-authored-by: Felix Lange <fjl@twurst.com>
Fixes#26505 where the console crashed when a property getter
raised an exception during autocompletion. I also noticed while fixing this
issue that autocomplete wasn't working for objects/fields with numbers in
them (most importantly web3.<tab><tab>) which is also now fixed.
* internal/flags: use filepath.Clean instead of path.Clean
* internal/flags: fix windows pipe issue
* internal/flags: modify test for windows
* internal/flags: use backticks, fix test
This PR removes the notion of fakeStorage from the state objects, and instead, for any state modifications that are needed, it simply makes the changes.
This changes the StorageTrie method to return an error when the trie
is not available. It used to return an 'empty trie' in this case, but that's
not possible anymore under PBSS.
This PR makes it possible to modify the flush interval time via RPC. On one extreme, `0s`, it would act as an archive node. If set to `1h`, means that after one hour of effective block processing time, the trie would be flushed. If one block takes 200ms, this means that a flush would occur every `5*3600=18000` blocks -- however, if the memory size of the cached states grows too large, it will flush sooner.
Essentially, this makes it possible to configure the node to be more or less "archive:ish", and without restarting the node while reconfiguring it.
This removes an RPC test which takes > 90s to execute, and updates the
internal/guide tests to use lighter scrypt parameters.
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR changes the pending tx subscription to return RPCTransaction types instead of normal Transaction objects. This will fix the inconsistencies with other tx returning API methods (i.e. getTransactionByHash), and also fill in the sender value for the tx.
co-authored by @s1na
This adds an option to direct log output to a file. This feature has been
requested a lot. It's sometimes useful to have this available when running
geth in an environment that doesn't easily allow redirecting the output.
Notably, there is no support for log rotation with this change. The --log.file option
opens the file once on startup and then keeps writing to the file handle.
This can become an issue when external log rotation tools are involved, so it's
best not to use them with this option for now.
This changes the CI build to store the git commit and date into package
internal/version instead of package main. Doing this essentially merges our
two ways of tracking the go-ethereum version into a single place, achieving
two objectives:
- Bad block reports, which use version.Info(), will now have the git commit
information even when geth is built in an environment such as
launchpad.net where git access is unavailable.
- For geth builds created by `go build ./cmd/geth` (i.e. not using `go run
build/ci.go install`), git information stored by the go tool is now used
in the p2p node name as well as in `geth version` and `geth
version-check`.
This change makes eth_getProof and eth_getStorageAt return an error when
the argument contains invalid hex in storage keys.
Co-authored-by: Felix Lange <fjl@twurst.com>
This changes the CI / release builds to use the latest Go version. It also
upgrades golangci-lint to a newer version compatible with Go 1.19.
In Go 1.19, godoc has gained official support for links and lists. The
syntax for code blocks in doc comments has changed and now requires a
leading tab character. gofmt adapts comments to the new syntax
automatically, so there are a lot of comment re-formatting changes in this
PR. We need to apply the new format in order to pass the CI lint stage with
Go 1.19.
With the linter upgrade, I have decided to disable 'gosec' - it produces
too many false-positive warnings. The 'deadcode' and 'varcheck' linters
have also been removed because golangci-lint warns about them being
unmaintained. 'unused' provides similar coverage and we already have it
enabled, so we don't lose much with this change.
Some small fixes to get the existing debug methods to conform to the spec. Mainly dropping the encoding information from the method name as it should be deduced from the debug context and allowing the method to be invoked by either block number or block hash. It also adds the method debug_getTransaction which returns the raw tx bytes by tx hash. This is pretty much equivalent to the eth_getRawTransactionByHash method.
This adds a cache for block logs which is shared by all filters. The cache
size of is configurable using the `--cache.blocklogs` flag.
Co-authored-by: Felix Lange <fjl@twurst.com>
This adds support for building statically-linked executables using ci.go.
Static linking is enabled by default in Docker builds, making it possible to
use the geth executable in any Docker image, regardless of the Linux
distribution the Dockerfile is based on.
Co-authored-by: Felix Lange <fjl@twurst.com>
Because the goal of eth_createAccessList is providing the caller with the largest-possible
access list, it's generally not important that the gas limit used by the tracer will match the usage
of the call exactly. Avoiding the gas estimation step is a performance improvement. As long as the
call does not branch based on gas limit, the returned access list will be accurate.
* internal/ethapi: error if tx args includes chain id that doesn't match local
* internal/ethapi: simplify code a bit
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
This change updates our urfave/cli dependency to the v2 branch of the library.
There are some Go API changes in cli v2:
- Flag values can now be accessed using the methods ctx.Bool,
ctx.Int, ctx.String, ... regardless of whether the flag is 'local' or
'global'.
- v2 has built-in support for flag categories. Our home-grown category
system is removed and the categories of flags are assigned as part of
the flag definition.
For users, there is only one observable difference with cli v2: flags must now
strictly appear before regular arguments. For example, the following command is
now invalid:
geth account import mykey.json --password file.txt
Instead, the command must be invoked as follows:
geth account import --password file.txt mykey.json
Marius Kjærstad
lightclient
Felix Lange
Martin HS
Delweng
Sebastian Stammler
jwasinger
Péter Szilágyi
aliening
Brandon Liu
Marius van der Wijden
Sina Mahmoodi
phenix3443
cam-schultz
rjl493456442
Roberto Bayardo
Amin Talebi
ucwong
Christopher Harrison
Curith
James Prestwich
Dan Laine
John Chase
hero5512
jin
Alex Mylonas
makcandrov
zhiqiangxu
David Dzhalaev
Adrian Sutton
sudeep
Adrian Sutton
aaronbuchwald
Darioush Jalali
Roberto Bayardo
s7v7nislands
TY
Justin Traglia
0xe3b0c4
Seungbae.yu
willian.eth
lightclient@protonmail.com
Kosuke Taniguchi
lmittmann