Commit Graph

287 Commits

Author SHA1 Message Date
Guillaume Ballet 234cc8e77f
eth/downloader: minor typo fixes in comments (#21035) 2020-05-06 15:35:04 +02:00
Péter Szilágyi b0bbd47185
eth: fix shutdown regression to abort downloads, not just cancel 2020-04-27 11:22:15 +03:00
Martin Holst Swende 3eca7b5d27
eth/downloader: fix data race in downloader 2019-10-29 14:32:45 +01:00
Christian Muehlhaeuser a32a2b933a cmd, contracts, eth, p2p, signer, whisper: fixed ineffectual assignments (#19869)
Fixed assigning values to variables we don't end up using.
2019-07-22 13:34:41 +03:00
gary rong 7641bbe535 eth/downloader: make syncing error more obvious (#19413) 2019-06-05 14:00:46 +02:00
Péter Szilágyi f35975ea21
core/rawdb, eth/downloader: align 64bit atomic fields 2019-05-17 01:45:05 +03:00
Péter Szilágyi 536b3b416c
cosensus, core, eth, params, trie: fixes + clique history cap 2019-05-16 10:39:35 +03:00
gary rong 37d280da41
core, cmd, vendor: fixes and database inspection tool (#15)
* core, eth: some fixes for freezer

* vendor, core/rawdb, cmd/geth: add db inspector

* core, cmd/utils: check ancient store path forceily

* cmd/geth, common, core/rawdb: a few fixes

* cmd/geth: support windows file rename and fix rename error

* core: support ancient plugin

* core, cmd: streaming file copy

* cmd, consensus, core, tests: keep genesis in leveldb

* core: write txlookup during ancient init

* core: bump database version
2019-05-16 10:39:34 +03:00
gary rong 80469bea0c
all: integrate the freezer with fast sync
* all: freezer style syncing

core, eth, les, light: clean up freezer relative APIs

core, eth, les, trie, ethdb, light: clean a bit

core, eth, les, light: add unit tests

core, light: rewrite setHead function

core, eth: fix downloader unit tests

core: add receipt chain insertion test

core: use constant instead of hardcoding table name

core: fix rollback

core: fix setHead

core/rawdb: remove canonical block first and then iterate side chain

core/rawdb, ethdb: add hasAncient interface

eth/downloader: calculate ancient limit via cht first

core, eth, ethdb: lots of fixes

* eth/downloader: print ancient disable log only for fast sync
2019-05-16 10:39:32 +03:00
Péter Szilágyi 9effd64290
core, eth, trie: bloom filter for trie node dedup during fast sync (#19489)
* core, eth, trie: bloom filter for trie node dedup during fast sync

* eth/downloader, trie: address review comments

* core, ethdb, trie: restart fast-sync bloom construction now and again

* eth/downloader: initialize fast sync bloom on startup

* eth: reenable eth/62 until we properly remove it
2019-05-13 15:28:01 +03:00
Péter Szilágyi 85726fdb02
eth/downloader: fix header delays during chain dedup 2019-05-07 16:09:11 +03:00
gary rong 749ccab9a4 eth/downloader: enable unsync-protection for light client (#19496)
* eth/downloader: enable unsync-protection for light client

* eth/downloader: fix tests
2019-04-25 11:31:23 +03:00
Péter Szilágyi 38f6b85638
eth, les, light: enforce CHT checkpoints on fast-sync too 2019-04-17 13:16:15 +03:00
gary rong 6c312a24b6 eth/downloader: fix ancestor searching for light syncing (#19136) 2019-03-14 12:19:03 +02:00
Matthew Halpern 3c62f965e7 eth: remove redundant parentheses (#19108) 2019-02-18 17:42:22 +02:00
lhendre d88441025f cmd,eth: 16400 Add an option to stop geth once in sync. WIP for light mode (#17321)
* cmd, eth: Added in the flag to step geth once sync based on input

* cmd, eth: 16400 Add an option to stop geth once in sync.

* cmd: 16400 Add an option to stop geth once in sync. WIP

* cmd/geth/main, les/fletcher: added in light mode support

* cmd/geth/main, les/fletcher: Cleaned Comments and code for light mode

* cmd: 16400 Fixed formatting issue and cleaned code

* cmd, eth, les: 16400 Fixed formatting issues

* cmd, eth, les: Performed gofmt to update formatting

* cmd, eth, les: Fixed bugs resulting formatting

* cmd/geth, eth/, les: switched to downloader event

* eth: Fixed styling and gen_config

* eth/: Fix nil error in config file

* cmd/geth: Updated countdown log

* les/fetcher.go: Removed depcreated channel

* eth/downloader.go: Removed deprecated select

* cmd/geth, cmd/utils: Fixed minor issues

* eth: Reverted config files to proper format

* eth: Fixed typo in config file

* cmd/geth, eth/down: Updated code to use header time stamp

* eth/downloader: Changed the time threshold to 10 minutes

* cmd/geth, eth/downloader: Updated downloading event to pass latest header

* cmd/geth: Updated main to use right timer object

* cmd/geth: Removed unused failed event

* cmd/geth: added in correct time field with type assertion

* cmd/geth, cmd/utils: Updated flag to use boolean

* cmd/geth, cmd/utils, eth/downloader: Cleaned up code based on recommendations

* cmd/geth: Removed unneeded import

* cmd/geth, eth/downloader: fixed event field and suggested changes

* cmd/geth, cmd/utils: Updated flag and linting issue
2019-01-30 08:40:36 +01:00
Martin Holst Swende 5f251a6448
downloader: fix edgecase where returned index is OOB for downloader (#18335)
* downloader: fix edgecase where returned index is OOB for downloader

* downloader: documentation

Co-Authored-By: holiman <martin@swende.se>
2018-12-20 10:46:08 +01:00
Péter Szilágyi 174083c3ae
eth/downloader: fix light client cht binary search issue 2018-11-28 13:46:13 +02:00
Péter Szilágyi accc0fab4f
core, eth/downloader: fix ancestor lookup for fast sync 2018-11-16 13:21:20 +02:00
Martin Holst Swende 51b2f1620c
downloader: different sync strategy 2018-11-16 11:54:36 +02:00
Martin Holst Swende eea3ae42a3 core, eth/downloader: fix validation flaw, fix downloader printout flaw (#17974) 2018-11-07 14:47:11 +02:00
Wenbiao Zheng 3088c122d8 eth/downloader: fix comment typos (#17956) 2018-10-23 13:21:16 +02:00
Péter Szilágyi 6ee3b26f44
eth/downloader: fix invalid hash chain error due to head mini reorg 2018-10-05 10:45:02 +03:00
Wenbiao Zheng 024b22c30e eth/downloader: use intermediate variable for better readability (#17510) 2018-09-29 22:13:39 +02:00
williambannas 9402f96597 eth: conform better to the golint standards (#16783)
* eth: made changes to conform better to the golint standards

* eth: fix comment nit
2018-06-14 13:14:52 +03:00
kiel barry 84f8c0cc1f common: improve documentation comments (#16701)
This commit adds many comments and removes unused code.
It also removes the EmptyHash function, which had some uses
but was silly.
2018-05-29 12:42:21 +02:00
Péter Szilágyi 6cf0ab38bd
core/rawdb: separate raw database access to own package (#16666) 2018-05-07 14:35:06 +03:00
Domino Valdano 7cf83cee52 eth/downloader: fix for Issue #16539 (#16546) 2018-04-23 10:01:21 +03:00
gary rong de2a7bb764 eth/downloader: wait for all fetcher goroutines to exit before terminating (#16509) 2018-04-16 11:37:48 +03:00
Yusup 7aad81f881 eth: fix typos (#16414) 2018-04-04 12:25:02 +02:00
gary rong 77da203547 eth: update higest block we know during the sync if a higher was found (#16283)
* eth: update higest block we know during the sync if a higher was found

* eth: avoid useless sync in fast sync
2018-03-09 11:51:30 +02:00
Péter Szilágyi ca64a122d3 eth/downloader: save and load trie sync progress (#16224) 2018-03-03 00:52:39 +01:00
Anton Evangelatov ae9f97221a metrics: pull library and introduce ResettingTimer and InfluxDB reporter (#15910)
* go-metrics: fork library and introduce ResettingTimer and InfluxDB reporter.

* vendor: change nonsense/go-metrics to ethersphere/go-metrics

* go-metrics: add tests. move ResettingTimer logic from reporter to type.

* all, metrics: pull in metrics package in go-ethereum

* metrics/test: make sure metrics are enabled for tests

* metrics: apply gosimple rules

* metrics/exp, internal/debug: init expvar endpoint when starting pprof server

* internal/debug: tiny comment formatting fix
2018-02-23 11:56:08 +02:00
Péter Szilágyi 96dad6b6f6
eth/downloader: don't require state for ancestor lookups 2018-02-11 14:43:56 +02:00
Péter Szilágyi eb07dbb079
eth, light: minor light client startup cleanups 2018-02-08 07:49:23 +02:00
Péter Szilágyi 55599ee95d core, trie: intermediate mempool between trie and database (#15857)
This commit reduces database I/O by not writing every state trie to disk.
2018-02-05 17:40:32 +01:00
Martin Holst Swende bc0666fb27 eth/downloader: fix #15858 by checking if downloader dropPeer function is set (#15992) 2018-02-05 15:38:06 +02:00
Péter Szilágyi 566d5c0777
core, eth, les, light: get rid of redundant methods 2018-01-30 18:42:00 +02:00
jtakalai 5aa3eac22d eth/downloader: minor comments cleanup (#15495)
it's -> its

pet peeve, and I like to imagine I'm not alone.
2017-11-16 13:14:51 +02:00
ferhat elmas 86f6568f66 build: enable unconvert linter (#15456)
* build: enable unconvert linter

 - fixes #15453
 - update code base for failing cases

* cmd/puppeth: replace syscall.Stdin with os.Stdin.Fd() for unconvert linter
2017-11-10 19:06:45 +02:00
Péter Szilágyi c8e70186a6 Merge pull request #14973 from rjl493456442/fix_downloader
eth/downloader: exit loop when there is no more available task
2017-09-11 14:02:02 +03:00
Felix Lange 10181b57a9 core, eth/downloader: commit block data using batches (#15115)
* ethdb: add Putter interface and Has method

* ethdb: improve docs and add IdealBatchSize

* ethdb: remove memory batch lock

Batches are not safe for concurrent use.

* core: use ethdb.Putter for Write* functions

This covers the easy cases.

* core/state: simplify StateSync

* trie: optimize local node check

* ethdb: add ValueSize to Batch

* core: optimize HasHeader check

This avoids one random database read get the block number. For many uses
of HasHeader, the expectation is that it's actually there. Using Has
avoids a load + decode of the value.

* core: write fast sync block data in batches

Collect writes into batches up to the ideal size instead of issuing many
small, concurrent writes.

* eth/downloader: commit larger state batches

Collect nodes into a batch up to the ideal size instead of committing
whenever a node is received.

* core: optimize HasBlock check

This avoids a random database read to get the number.

* core: use numberCache in HasHeader

numberCache has higher capacity, increasing the odds of finding the
header without a database lookup.

* core: write imported block data using a batch

Restore batch writes of state and add blocks, tx entries, receipts to
the same batch. The change also simplifies the miner.

This commit also removes posting of logs when a forked block is imported.

* core: fix DB write error handling

* ethdb: use RLock for Has

* core: fix HasBlock comment
2017-09-09 19:03:07 +03:00
Péter Szilágyi f30179d62e
eth: disable fast sync after pivot is committed 2017-09-06 15:02:44 +03:00
rjl493456442 02aa86e659 eth/downloader: exit loop when there is no more available task 2017-08-14 13:51:37 +08:00
Nick Johnson fe13949d9d eth/downloader: Doc fixes 2017-07-05 11:13:16 +01:00
Nick Johnson be8f8409bc eth/downloader, les, light: Changes in response to review 2017-07-03 15:17:12 +01:00
Nick Johnson ae11545bc5 eth, les: Refactor downloader peer to use structs 2017-06-29 12:49:18 +01:00
Nick Johnson 0550957989 eth, les, light: Refactor downloader to use blockchain interface 2017-06-28 15:58:41 +01:00
Felix Lange 0042f13d47 eth/downloader: separate state sync from queue (#14460)
* eth/downloader: separate state sync from queue

Scheduling of state node downloads hogged the downloader queue lock when
new requests were scheduled. This caused timeouts for other requests.
With this change, state sync is fully independent of all other downloads
and doesn't involve the queue at all.

State sync is started and checked on in processContent. This is slightly
awkward because processContent doesn't have a select loop. Instead, the
queue is closed by an auxiliary goroutine when state sync fails. We
tried several alternatives to this but settled on the current approach
because it's the least amount of change overall.

Handling of the pivot block has changed slightly: the queue previously
prevented import of pivot block receipts before the state of the pivot
block was available. In this commit, the receipt will be imported before
the state. This causes an annoyance where the pivot block is committed
as fast block head even when state downloads fail. Stay tuned for more
updates in this area ;)

* eth/downloader: remove cancelTimeout channel

* eth/downloader: retry state requests on timeout

* eth/downloader: improve comment

* eth/downloader: mark peers idle when state sync is done

* eth/downloader: move pivot block splitting to processContent

This change also ensures that pivot block receipts aren't imported
before the pivot block itself.

* eth/downloader: limit state node retries

* eth/downloader: improve state node error handling and retry check

* eth/downloader: remove maxStateNodeRetries

It fails the sync too much.

* eth/downloader: remove last use of cancelCh in statesync.go

Fixes TestDeliverHeadersHang*Fast and (hopefully)
the weird cancellation behaviour at the end of fast sync.

* eth/downloader: fix leak in runStateSync

* eth/downloader: don't run processFullSyncContent in LightSync mode

* eth/downloader: improve comments

* eth/downloader: fix vet, megacheck

* eth/downloader: remove unrequested tasks anyway

* eth/downloader, trie: various polishes around duplicate items

This commit explicitly tracks duplicate and unexpected state
delieveries done against a trie Sync structure, also adding
there to import info logs.

The commit moves the db batch used to commit trie changes one
level deeper so its flushed after every node insertion. This
is needed to avoid a lot of duplicate retrievals caused by
inconsistencies between Sync internals and database. A better
approach is to track not-yet-written states in trie.Sync and
flush on commit, but I'm focuing on correctness first now.

The commit fixes a regression around pivot block fail count.
The counter previously was reset to 1 if and only if a sync
cycle progressed (inserted at least 1 entry to the database).
The current code reset it already if a node was delivered,
which is not stong enough, because unless it ends up written
to disk, an attacker can just loop and attack ad infinitum.

The commit also fixes a regression around state deliveries
and timeouts. The old downloader tracked if a delivery is
stale (none of the deliveries were requestedt), in which
case it didn't mark the node idle and did not send further
requests, since it signals a past timeout. The current code
did mark it idle even on stale deliveries, which eventually
caused two requests to be in flight at the same time, making
the deliveries always stale and mass duplicating retrievals
between multiple peers.

* eth/downloader: fix state request leak

This commit fixes the hang seen sometimes while doing the state
sync. The cause of the hang was a rare combination of events:
request state data from peer, peer drops and reconnects almost
immediately. This caused a new download task to be assigned to
the peer, overwriting the old one still waiting for a timeout,
which in turned leaked the requests out, never to be retried.
The fix is to ensure that a task assignment moves any pending
one back into the retry queue.

The commit also fixes a regression with peer dropping due to
stalls. The current code considered a peer stalling if they
timed out delivering 1 item. However, the downloader never
requests only one, the minimum is 2 (attempt to fine tune
estimated latency/bandwidth). The fix is simply to drop if
a timeout is detected at 2 items.

Apart from the above bugfixes, the commit contains some code
polishes I made while debugging the hang.

* core, eth, trie: support batched trie sync db writes

* trie: rename SyncMemCache to syncMemBatch
2017-06-22 15:26:03 +03:00
Péter Szilágyi 96c2ab22e0
eth/downloader: fix a potential issue against future refactors 2017-05-02 16:14:35 +03:00
Péter Szilágyi 9b84caf3a5 core, eth, les: support resuming fast sync on heavy rollback (#3743) 2017-03-22 01:37:24 +01:00
Péter Szilágyi 9184249b39 Logger updates 3 (#3730)
* accounts, cmd, eth, ethdb: port logs over to new system

* ethdb: drop concept of cache distribution between dbs

* eth: fix some log nitpicks to make them nicer
2017-03-02 14:06:16 +01:00
Péter Szilágyi 2f28a12cdb
common, eth/downloader, log: support terminal log formatting 2017-02-27 19:15:18 +02:00
Péter Szilágyi 8676aeb798
eth/downloader: review fixes 2017-02-27 13:22:33 +02:00
Péter Szilágyi 46eea4d105
accounts, eth/downloader: use "err" instead of "error" in logs 2017-02-27 13:17:58 +02:00
Péter Szilágyi 0a63c3e362
eth/downloader: port over old logs from glog to log15 2017-02-27 13:16:40 +02:00
Péter Szilágyi 61e6bb1247
eth, les, swarm: fix go vet issues sufraced by log15 2017-02-23 12:16:45 +02:00
Péter Szilágyi d4fd06c3dc
all: blidly swap out glog to our log15, logs need rework 2017-02-23 12:16:44 +02:00
Jeffrey Wilcke c12f4df910 params: core, core/vm, miner: 64bit gas instructions
Reworked the EVM gas instructions to use 64bit integers rather than
arbitrary size big ints. All gas operations, be it additions,
multiplications or divisions, are checked and guarded against 64 bit
integer overflows.

In additon, most of the protocol paramaters in the params package have
been converted to uint64 and are now constants rather than variables.

* common/math: added overflow check ops
* core: vmenv, env renamed to evm
* eth, internal/ethapi, les: unmetered eth_call and cancel methods
* core/vm: implemented big.Int pool for evm instructions
* core/vm: unexported intPool methods & verification methods
* core/vm: added memoryGasCost overflow check and test
2017-02-13 21:44:25 +01:00
Jeffrey Wilcke 57f4e90257 Revert "params: core, core/vm, miner: 64bit gas instructions (#3514)"
This reverts commit 8b57c49490.
2017-02-13 15:15:12 +01:00
Jeffrey Wilcke 8b57c49490 params: core, core/vm, miner: 64bit gas instructions (#3514)
Reworked the EVM gas instructions to use 64bit integers rather than
arbitrary size big ints. All gas operations, be it additions,
multiplications or divisions, are checked and guarded against 64 bit
integer overflows.

In additon, most of the protocol paramaters in the params package have
been converted to uint64 and are now constants rather than variables.

* common/math: added overflow check ops
* core: vmenv, env renamed to evm
* eth, internal/ethapi, les: unmetered eth_call and cancel methods
* core/vm: implemented big.Int pool for evm instructions
* core/vm: unexported intPool methods & verification methods
* core/vm: added memoryGasCost overflow check and test
2017-02-02 15:25:42 +01:00
Péter Szilágyi 18c77744ff
all: fix spelling errors 2017-01-06 19:44:35 +02:00
zsfelfoldi 7db7109a5b cmd, eth: added light client and light server modes 2016-11-09 02:12:53 +01:00
Péter Szilágyi 8ecee175f2
eth/downloader: lower a state sync log level (users freak out) 2016-10-31 18:28:03 +02:00
Péter Szilágyi 8b10617bba
eth/downloader: reduce fast sync block requirements, fix test 2016-10-31 17:26:44 +02:00
Péter Szilágyi 90b16a3e85
core/state, eth/downloader, trie: reset fast-failure on progress 2016-10-31 14:19:14 +02:00
Péter Szilágyi 53db80da89
eth/downloader: fix a data race in a log output 2016-10-21 13:09:27 +03:00
Péter Szilágyi 1291778032
cmd/geth, code, eth/downloader: tune import logs and mem stats 2016-10-21 12:23:39 +03:00
Péter Szilágyi 64500ab0fa
common, core, eth/downloader: adjust import log formatting 2016-10-18 13:16:36 +03:00
Péter Szilágyi 2924fdfcf7 ethereum, ethclient: add SyncProgress API endpoint 2016-09-06 13:41:43 +03:00
Péter Szilágyi 89a3fbc0fb Merge pull request #2866 from karalabe/downloader-future-ancestors
eth/downloader: fewer headers and futures too in ancestor lookup
2016-08-16 10:01:52 +03:00
Péter Szilágyi b46b36729f Merge pull request #2868 from karalabe/downloader-abort-master-drop
eth/downloader: abort sync if master drops (timeout prev)
2016-08-09 11:39:24 +03:00
Péter Szilágyi 893fabd336 Merge pull request #2867 from karalabe/dao-challenge-finish
eth, eth/downloader: don't forward the DAO challenge header
2016-08-09 11:39:13 +03:00
Péter Szilágyi 44ea0da2b0 Merge pull request #2861 from karalabe/track-peer-heads-properly
eth, eth/downloader: better remote head tracking
2016-08-09 11:38:34 +03:00
Péter Szilágyi 8f0a4a25f8 eth/downloader: abort sync if master drops (timeout prev) 2016-07-26 13:07:12 +03:00
Péter Szilágyi 071af57bcf eth, eth/downloader: don't forward the DAO challenge header 2016-07-26 12:26:41 +03:00
Péter Szilágyi d68865f3b1 eth/downloader: fewer headers and futures too un ancestor lookup 2016-07-26 11:15:38 +03:00
Péter Szilágyi 1dd272080d eth, eth/downloader: better remote head tracking 2016-07-25 15:14:14 +03:00
Felix Lange 3e3a79ea13 Merge pull request #2855 from karalabe/downloader-fix-stall-drop
eth/downloader: fix the stall checks/drops during sync
2016-07-25 12:56:11 +02:00
Péter Szilágyi c7c82f1b44 eth/downloader: fix the stall checks/drops during sync 2016-07-22 17:55:46 +03:00
Felix Lange 016007bd25 eth, eth/downloader, eth/fetcher: delete eth/61 code
The eth/61 protocol was disabled in #2776, this commit removes its
message handlers and hash-chain sync logic.
2016-07-22 13:17:19 +02:00
Péter Szilágyi a691aa2a13 eth/downloader: return invalid chain (peer drop) on import fails 2016-07-11 19:00:49 +03:00
Péter Szilágyi 88f174a014 eth/downloader: adaptive quality of service tuning 2016-06-06 14:20:57 +03:00
Péter Szilágyi 61ee9f299d eth/downloader: make fast sync resilient to critical section fails 2016-06-02 12:37:14 +03:00
Péter Szilágyi 16a23ff740 Merge pull request #2637 from karalabe/downloader-always-cancel
eth/downloader: ensure cancel channel is closed post sync
2016-05-31 14:48:52 +03:00
Péter Szilágyi 4496a44f68 eth/downloader: ensure cancel channel is closed post sync 2016-05-30 12:01:50 +03:00
Péter Szilágyi 4f1d92b332 eth/downloader, trie: pull head state concurrently with chain 2016-05-27 14:26:00 +03:00
Péter Szilágyi 8906b2fe09 eth/downloader: fix reviewer comments 2016-05-20 10:56:01 +03:00
Péter Szilágyi e86619e75d eth/downloader: stream partial skeleton filling to processor 2016-05-17 10:03:34 +03:00
Péter Szilágyi b40dc8a1da eth/downloader: implement concurrent header downloads 2016-05-17 10:03:34 +03:00
Péter Szilágyi 39ce85cf5d eth/downloader: bound fork ancestry and allow heavy short forks 2016-05-13 12:12:13 +02:00
Felix Lange 6fdd0893c3 all: fix go vet warnings 2016-04-15 11:17:27 +02:00
Leif Jurvetson 434e4b31d8 core, eth: replace reorganiz with reorganis 2016-03-15 12:03:17 -07:00
Leif Jurvetson bbbe2360d0 eth: various typos 2016-03-15 11:27:49 -07:00
Péter Szilágyi 64ee5763ee eth/downloader: fix premature exit before notifying all part fetchers 2016-02-29 14:22:28 +02:00
Felix Lange 26e72b2ccd eth/downloader: fix header download limiting
Fixes #2201
2016-02-26 15:24:51 +01:00
Péter Szilágyi 216729009b eth/downloader: fix partial rollback and ancestor lookup 2016-02-19 11:17:02 +02:00
Péter Szilágyi 59cd60b266 eth, eth/downloader, jsre: surface state sync progress through the API 2016-02-10 14:30:41 +02:00
Péter Szilágyi f1ec226d80 eth/downloader: raise pending state limit that prevented concurrency 2016-02-08 14:57:02 +02:00
Péter Szilágyi 9e011ff1cd core, eth/downloader: ensure state presence in ancestor lookup 2016-01-04 16:27:23 +02:00