Commit Graph

177 Commits

Author SHA1 Message Date
Marius van der Wijden ddeea1e0c6
core: types: less allocations when hashing and tx handling (#21265)
* core, crypto: various allocation savings regarding tx handling

* core: reduce allocs for gas price comparison

This change reduces the allocations needed for comparing different transactions to each other.
A call to `tx.GasPrice()` copies the gas price as it has to be safe against modifications and
also needs to be threadsafe. For comparing and ordering different transactions we don't need
these guarantees

* core: added tx.GasPriceIntCmp for comparison without allocation

adds a method to remove unneeded allocation in comparison to tx.gasPrice

* core/types: pool legacykeccak256 objects in rlpHash

rlpHash is by far the most used function in core that allocates a legacyKeccak256 object on each call.
Since it is so widely used it makes sense to add pooling here so we relieve the GC.
On my machine these changes result in > 100 MILLION less allocations and > 30 GB less allocated memory.

* reverted some changes

* reverted some changes

* trie: use crypto.KeccakState instead of replicating code

Co-authored-by: Martin Holst Swende <martin@swende.se>
2020-06-30 11:59:06 +02:00
Marius van der Wijden b482423e61
trie: reduce allocs in insertPreimage (#21261) 2020-06-24 21:56:27 +02:00
gary rong 389da6aa48
trie: enforce monotonic range in prover and return end marker (#21130)
* trie: add hasRightElement indicator

* trie: ensure the range is monotonic increasing

* trie: address comment and fix lint

* trie: address comment

* trie: make linter happy

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-05-27 17:37:37 +03:00
gary rong 070a5e1252
trie: fix for range proof (#21107)
* trie: fix for range proof

* trie: fix typo
2020-05-26 13:11:29 +03:00
Martin Michlmayr 4f2784b38f
all: fix typos in comments (#21118) 2020-05-25 10:21:28 +02:00
gary rong 65ce550b37
trie: extend range proofs with non-existence (#21000)
* trie: implement range proof with non-existent edge proof

* trie: fix cornercase

* trie: consider empty range

* trie: add singleSide test

* trie: support all-elements range proof

* trie: fix typo

* trie: tiny typos and formulations

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-05-20 15:45:38 +03:00
Guillaume Ballet 234cc8e77f
eth/downloader: minor typo fixes in comments (#21035) 2020-05-06 15:35:04 +02:00
gary rong 5cdc2dffda
trie: fix TestBadRangeProof unit test (#21034) 2020-05-06 15:33:57 +02:00
gary rong 44ff3f3dc9
trie: initial implementation for range proof (#20908)
* trie: initial implementation for range proof

* trie: add benchmark

* trie: fix lint

* trie: fix minor issue

* trie: unset the edge valuenode as well

* trie: unset the edge valuenode as nilValuenode
2020-04-24 14:37:56 +03:00
Péter Szilágyi 79b68dd78d
Merge pull request #20923 from holiman/fix_seckeybuf
trie: fix concurrent usage of secKeyBuf, ref #20920
2020-04-20 16:52:04 +03:00
Martin Holst Swende 6402c42b67
all: simplify and fix database iteration with prefix/start (#20808)
* core/state/snapshot: start fixing disk iterator seek

* ethdb, rawdb, leveldb, memorydb: implement iterators with prefix and start

* les, core/state/snapshot: iterator fixes

* all: remove two iterator methods

* all: rename Iteratee.NewIteratorWith -> NewIterator

* ethdb: fix review concerns
2020-04-15 14:08:53 +03:00
Martin Holst Swende af4080b4b7
trie: fix concurrent usage of secKeyBuf, ref #20920 2020-04-15 11:07:29 +02:00
Martin Holst Swende a1313b5b1e
trie: make hasher parallel when number of changes are large (#20488)
* trie: make hasher parallel when number of changes are large

* trie: remove unused field dirtyCount

* trie: rename unhashedCount/unhashed
2020-02-04 14:02:38 +02:00
Martin Holst Swende 5a9c96454e
trie: separate hashes and committer, collapse on commit
* trie:  make db insert use size instead of full data

* core/state: minor optimization in state onleaf allocation

* trie: implement dedicated committer and hasher

* trie: use dedicated committer/hasher

* trie: linter nitpicks

* core/state, trie: avoid unnecessary storage trie load+commit

* trie: review feedback, mainly docs + minor changes

* trie: start deprecating old hasher

* trie: fix misspell+lint

* trie: deprecate hasher.go, make proof framework use new hasher

* trie: rename pure_committer/hasher to committer/hasher

* trie, core/state: fix review concerns

* trie: more review concerns

* trie: make commit collapse into hashnode, don't touch dirtyness

* trie: goimports fixes

* trie: remove panics
2020-02-03 17:28:30 +02:00
Martin Holst Swende 9b09c0fc83 * trie: utilize callbacks instead of amassing lists in ref/unref (#20529)
* trie/tests: add benchmarks and update trie tests

* trie: update benchmark tests

* trie: utilize callbacks instead of amassing lists of hashes in database ref/unref

* trie: replace remaining non-callback based accesses
2020-01-17 13:59:45 +02:00
me020523 4f7b7f84ae add node.go unit test file node_test.go (#20028)
* add node.go unit test file node_test.go

* add node_test.go file license and rollback trie_test.go

* fix unused variable v

* trie: fix license year

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-01-07 10:31:20 +01:00
Péter Szilágyi 13ccf6016e
trie: track dirty cache metrics, track clean writes on commit 2019-12-02 12:23:35 +02:00
gary rong 08611cfd75 trie: remove dead code (#20405) 2019-11-28 12:47:35 +02:00
gary rong 9d67222f4e trie: replace bigcache with fastcache (#19971) 2019-11-25 10:58:15 +02:00
Felix Lange 0abcf03fde trie: remove unused code (#20366) 2019-11-22 17:24:48 +02:00
gary rong ecdbb402ee trie: remove node ordering slice in sync batch (#19929)
When we flush a batch of trie nodes into database during the state
sync, we should guarantee that all children should be flushed before
parent.

Actually the trie nodes commit order is strict by: children -> parent.
But when we flush all ready nodes into db, we don't need the order
anymore since

    (1) they are all ready nodes (no more dependency)
    (2) underlying database provides write atomicity
2019-10-28 18:50:11 +01:00
gary rong df6c08a485 core, trie: decode the value for storage dump (#19943)
* core, trie: decode the value for storage dump

* core/state: address comment
2019-08-12 17:14:40 +03:00
Christian Muehlhaeuser 5183483c53 core/state, p2p/discover, trie, whisper: avoid unnecessary conversions (#19870)
No need to convert these types.
2019-07-22 10:30:09 +03:00
Guillaume Ballet dcc4adfcd7 cmd/geth: wrong memory size sanitizing on OpenBSD (#19793) 2019-07-05 13:13:21 +03:00
Péter Szilágyi 536b3b416c
cosensus, core, eth, params, trie: fixes + clique history cap 2019-05-16 10:39:35 +03:00
gary rong 80469bea0c
all: integrate the freezer with fast sync
* all: freezer style syncing

core, eth, les, light: clean up freezer relative APIs

core, eth, les, trie, ethdb, light: clean a bit

core, eth, les, light: add unit tests

core, light: rewrite setHead function

core, eth: fix downloader unit tests

core: add receipt chain insertion test

core: use constant instead of hardcoding table name

core: fix rollback

core: fix setHead

core/rawdb: remove canonical block first and then iterate side chain

core/rawdb, ethdb: add hasAncient interface

eth/downloader: calculate ancient limit via cht first

core, eth, ethdb: lots of fixes

* eth/downloader: print ancient disable log only for fast sync
2019-05-16 10:39:32 +03:00
Péter Szilágyi 9effd64290
core, eth, trie: bloom filter for trie node dedup during fast sync (#19489)
* core, eth, trie: bloom filter for trie node dedup during fast sync

* eth/downloader, trie: address review comments

* core, ethdb, trie: restart fast-sync bloom construction now and again

* eth/downloader: initialize fast sync bloom on startup

* eth: reenable eth/62 until we properly remove it
2019-05-13 15:28:01 +03:00
gary rong 14868a37fb trie: clarify why verifyProof doesn't check hashes (#19530)
* trie: fix merkle proof

* trie: use hasher instead of allocate keccack256 every time

* trie: add comments
2019-05-07 15:06:07 +03:00
Péter Szilágyi 4a4abc41d4
trie: approximate the wasted cache metaspace closer 2019-04-12 11:43:16 +03:00
Péter Szilágyi 4bf0d11e7c
trie: there's no point in retrieving the metaroot 2019-04-05 13:09:28 +03:00
Martin Holst Swende 59e1953246 core, ethdb, trie: mode dirty data to clean cache on flush (#19307)
This PR is a more advanced form of the dirty-to-clean cacher (#18995),
where we reuse previous database write batches as datasets to uncache,
saving a dirty-trie-iteration and a dirty-trie-rlp-reencoding per block.
2019-03-26 15:48:31 +01:00
Martin Holst Swende 876f357364 trie: disable fnv64a hashing of hashes for bigcache (#19314)
* trie: disable fnv64a hashing of hashes for bigcache

* trie/database: add very important period
2019-03-22 17:13:28 +02:00
Péter Szilágyi 91eec1251c
cmd, core, eth, trie: get rid of trie cache generations (#19262)
* cmd, core, eth, trie: get rid of trie cache generations

* core, trie: get rid of remainder of cache gen boilerplate
2019-03-14 15:25:12 +02:00
Péter Szilágyi 8111b9dda5
ethdb, trie: tiny API tidy-up from the database rework pr 2019-03-12 12:32:02 +02:00
Péter Szilágyi 054412e335
all: clean up and proerly abstract database access 2019-03-06 13:35:03 +02:00
Matthew Halpern 514a9472ad trie: prefer nil slices over zero-length slices (#19084) 2019-02-19 14:50:11 +01:00
Martin Holst Swende 4f85c2b88b
trie: fix error in node decoding (#19111) 2019-02-16 16:16:12 +01:00
HackyMiner bb7c786b09 trie: add missing unlock call in error case (#18985) 2019-02-04 12:42:46 +01:00
Dave McGregor 33d233d3e1
vendor, crypto, swarm: switch over to upstream sha3 package 2019-01-04 09:26:07 +02:00
Martin Holst Swende 2843001ac2 trie: fix overflow in write cache parent tracking (#18165)
trie/database: fix overflow in parent tracking
2018-11-22 15:14:31 +02:00
Péter Szilágyi d136e985e8
trie: go fmt package 2018-11-16 16:35:39 +02:00
Łukasz Kurowski 68be45e5f8 trie: return hasher to pool (#18116)
* trie: return hasher to pool

* trie: minor code formatting fix
2018-11-16 11:50:48 +02:00
Péter Szilágyi 434dd5bc00
cmd, core, eth, light, trie: add trie read caching layer 2018-11-15 12:22:13 +02:00
Péter Szilágyi 53634f1e04 trie: remove unused originalRoot field (#17862) 2018-10-08 13:16:16 +02:00
Wenbiao Zheng 6a33954731 core, eth, trie: use common/prque (#17508) 2018-09-03 17:33:21 +02:00
Mymskmkt 70398d300d trie: fix typo (#17498) 2018-08-24 21:08:48 +03:00
Péter Szilágyi 11bbc66082
eth, trie: fix tracer GC which accidentally pruned the metaroot 2018-08-09 12:33:30 +03:00
Mymskmkt 8051a0768a trie: fix comment typo (#17350) 2018-08-08 16:08:40 +03:00
Oleg Kovalov cf05ef9106 p2p, swarm, trie: avoid copying slices in loops (#17265) 2018-08-07 13:56:40 +03:00
Péter Szilágyi 8a9c31a307
trie: handle removing the freshest node too 2018-07-30 16:31:17 +03:00