Commit Graph

246 Commits

Author SHA1 Message Date
Martin Holst Swende 9f7b79af00
eth/downloader: fix rollback issue on short chains 2020-08-21 13:36:08 +02:00
Péter Szilágyi 8cbdc8638f
core: define and test chain rewind corner cases (#21409)
* core: define and test chain reparation cornercases

* core: write up a variety of set-head tests

* core, eth: unify chain rollbacks, handle all the cases

* core: make linter smile

* core: remove commented out legacy code

* core, eth/downloader: fix review comments

* core: revert a removed recovery mechanism
2020-08-20 13:01:24 +03:00
rene 1976bb3df0
eth/downloader: remove eth62 (#21378)
* init

notes

removed some mentions of eth62, bumped protocol err too old to >=63

* remove sanity checks and bump supported protocol version up to 63

* remove 62 tests, still need to add 65

* remove 65 tests
2020-07-29 13:47:19 +03:00
Martin Holst Swende 105922180f
eth/downloader: refactor downloader + queue (#21263)
* eth/downloader: refactor downloader + queue

downloader, fetcher: throttle-metrics, fetcher filter improvements, standalone resultcache

downloader: more accurate deliverytime calculation, less mem overhead in state requests

downloader/queue: increase underlying buffer of results, new throttle mechanism

eth/downloader: updates to tests

eth/downloader: fix up some review concerns

eth/downloader/queue: minor fixes

eth/downloader: minor fixes after review call

eth/downloader: testcases for queue.go

eth/downloader: minor change, don't set progress unless progress...

eth/downloader: fix flaw which prevented useless peers from being dropped

eth/downloader: try to fix tests

eth/downloader: verify non-deliveries against advertised remote head

eth/downloader: fix flaw with checking closed-status causing hang

eth/downloader: hashing avoidance

eth/downloader: review concerns + simplify resultcache and queue

eth/downloader: add back some locks, address review concerns

downloader/queue: fix remaining lock flaw

* eth/downloader: nitpick fixes

* eth/downloader: remove the *2*3/4 throttling threshold dance

* eth/downloader: print correct throttle threshold in stats

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
2020-07-24 10:46:26 +03:00
gary rong 6eef141aef
les: historical data garbage collection (#19570)
This change introduces garbage collection for the light client. Historical
chain data is deleted periodically. If you want to disable the GC, use
the --light.nopruning flag.
2020-07-13 11:02:54 +02:00
Martin Holst Swende 967d8de77a
eth/downloader: fix peer idleness tracking when restarting state sync (#21260)
This fixes two issues with state sync restarts:

When sync restarts with a new root, some peers can have in-flight requests.
Since all peers with active requests were marked idle when exiting sync,
the new sync would schedule more requests for those peers. When the
response for the earlier request arrived, the new sync would reject it and
mark the peer idle again, rendering the peer useless until it disconnected.

The other issue was that peers would not be marked idle when they had
delivered a response, but the response hadn't been processed before
restarting the state sync. This also made the peer useless because it
would be permanently marked busy.

Co-authored-by: Felix Lange <fjl@twurst.com>
2020-07-08 23:08:08 +02:00
Marius van der Wijden d671dbd5b7
eth/downloader: fixes data race between synchronize and other methods (#21201)
* eth/downloaded: fixed datarace between synchronize and Progress

There was a race condition between `downloader.synchronize()` and `Progress` `syncWithPeer` `fetchHeight` `findAncestors` and `processHeaders`
This PR changes the behavior of the downloader a bit.
Previously the functions `Progress` `syncWithPeer` `fetchHeight` `findAncestors` and `processHeaders` read the syncMode anew within their loops. Now they read the syncMode at the start of their function and don't change it during their runtime.

* eth/downloaded: comment

* eth/downloader: added comment
2020-06-30 19:43:29 +02:00
Martin Holst Swende a5eee8d1dc
eth/downloader: more context in errors (#21067)
This PR makes use of go 1.13 error handling, wrapping errors and using
errors.Is to check a wrapped root-cause. It also removes the travis
builders for go 1.11 and go 1.12.
2020-05-29 11:12:43 +02:00
Martin Michlmayr 4f2784b38f
all: fix typos in comments (#21118) 2020-05-25 10:21:28 +02:00
Guillaume Ballet 234cc8e77f
eth/downloader: minor typo fixes in comments (#21035) 2020-05-06 15:35:04 +02:00
Péter Szilágyi b0bbd47185
eth: fix shutdown regression to abort downloads, not just cancel 2020-04-27 11:22:15 +03:00
Martin Holst Swende 3eca7b5d27
eth/downloader: fix data race in downloader 2019-10-29 14:32:45 +01:00
Christian Muehlhaeuser a32a2b933a cmd, contracts, eth, p2p, signer, whisper: fixed ineffectual assignments (#19869)
Fixed assigning values to variables we don't end up using.
2019-07-22 13:34:41 +03:00
gary rong 7641bbe535 eth/downloader: make syncing error more obvious (#19413) 2019-06-05 14:00:46 +02:00
Péter Szilágyi f35975ea21
core/rawdb, eth/downloader: align 64bit atomic fields 2019-05-17 01:45:05 +03:00
Péter Szilágyi 536b3b416c
cosensus, core, eth, params, trie: fixes + clique history cap 2019-05-16 10:39:35 +03:00
gary rong 37d280da41
core, cmd, vendor: fixes and database inspection tool (#15)
* core, eth: some fixes for freezer

* vendor, core/rawdb, cmd/geth: add db inspector

* core, cmd/utils: check ancient store path forceily

* cmd/geth, common, core/rawdb: a few fixes

* cmd/geth: support windows file rename and fix rename error

* core: support ancient plugin

* core, cmd: streaming file copy

* cmd, consensus, core, tests: keep genesis in leveldb

* core: write txlookup during ancient init

* core: bump database version
2019-05-16 10:39:34 +03:00
gary rong 80469bea0c
all: integrate the freezer with fast sync
* all: freezer style syncing

core, eth, les, light: clean up freezer relative APIs

core, eth, les, trie, ethdb, light: clean a bit

core, eth, les, light: add unit tests

core, light: rewrite setHead function

core, eth: fix downloader unit tests

core: add receipt chain insertion test

core: use constant instead of hardcoding table name

core: fix rollback

core: fix setHead

core/rawdb: remove canonical block first and then iterate side chain

core/rawdb, ethdb: add hasAncient interface

eth/downloader: calculate ancient limit via cht first

core, eth, ethdb: lots of fixes

* eth/downloader: print ancient disable log only for fast sync
2019-05-16 10:39:32 +03:00
Péter Szilágyi 9effd64290
core, eth, trie: bloom filter for trie node dedup during fast sync (#19489)
* core, eth, trie: bloom filter for trie node dedup during fast sync

* eth/downloader, trie: address review comments

* core, ethdb, trie: restart fast-sync bloom construction now and again

* eth/downloader: initialize fast sync bloom on startup

* eth: reenable eth/62 until we properly remove it
2019-05-13 15:28:01 +03:00
Péter Szilágyi 85726fdb02
eth/downloader: fix header delays during chain dedup 2019-05-07 16:09:11 +03:00
gary rong 749ccab9a4 eth/downloader: enable unsync-protection for light client (#19496)
* eth/downloader: enable unsync-protection for light client

* eth/downloader: fix tests
2019-04-25 11:31:23 +03:00
Péter Szilágyi 38f6b85638
eth, les, light: enforce CHT checkpoints on fast-sync too 2019-04-17 13:16:15 +03:00
gary rong 6c312a24b6 eth/downloader: fix ancestor searching for light syncing (#19136) 2019-03-14 12:19:03 +02:00
Matthew Halpern 3c62f965e7 eth: remove redundant parentheses (#19108) 2019-02-18 17:42:22 +02:00
lhendre d88441025f cmd,eth: 16400 Add an option to stop geth once in sync. WIP for light mode (#17321)
* cmd, eth: Added in the flag to step geth once sync based on input

* cmd, eth: 16400 Add an option to stop geth once in sync.

* cmd: 16400 Add an option to stop geth once in sync. WIP

* cmd/geth/main, les/fletcher: added in light mode support

* cmd/geth/main, les/fletcher: Cleaned Comments and code for light mode

* cmd: 16400 Fixed formatting issue and cleaned code

* cmd, eth, les: 16400 Fixed formatting issues

* cmd, eth, les: Performed gofmt to update formatting

* cmd, eth, les: Fixed bugs resulting formatting

* cmd/geth, eth/, les: switched to downloader event

* eth: Fixed styling and gen_config

* eth/: Fix nil error in config file

* cmd/geth: Updated countdown log

* les/fetcher.go: Removed depcreated channel

* eth/downloader.go: Removed deprecated select

* cmd/geth, cmd/utils: Fixed minor issues

* eth: Reverted config files to proper format

* eth: Fixed typo in config file

* cmd/geth, eth/down: Updated code to use header time stamp

* eth/downloader: Changed the time threshold to 10 minutes

* cmd/geth, eth/downloader: Updated downloading event to pass latest header

* cmd/geth: Updated main to use right timer object

* cmd/geth: Removed unused failed event

* cmd/geth: added in correct time field with type assertion

* cmd/geth, cmd/utils: Updated flag to use boolean

* cmd/geth, cmd/utils, eth/downloader: Cleaned up code based on recommendations

* cmd/geth: Removed unneeded import

* cmd/geth, eth/downloader: fixed event field and suggested changes

* cmd/geth, cmd/utils: Updated flag and linting issue
2019-01-30 08:40:36 +01:00
Martin Holst Swende 5f251a6448
downloader: fix edgecase where returned index is OOB for downloader (#18335)
* downloader: fix edgecase where returned index is OOB for downloader

* downloader: documentation

Co-Authored-By: holiman <martin@swende.se>
2018-12-20 10:46:08 +01:00
Péter Szilágyi 174083c3ae
eth/downloader: fix light client cht binary search issue 2018-11-28 13:46:13 +02:00
Péter Szilágyi accc0fab4f
core, eth/downloader: fix ancestor lookup for fast sync 2018-11-16 13:21:20 +02:00
Martin Holst Swende 51b2f1620c
downloader: different sync strategy 2018-11-16 11:54:36 +02:00
Martin Holst Swende eea3ae42a3 core, eth/downloader: fix validation flaw, fix downloader printout flaw (#17974) 2018-11-07 14:47:11 +02:00
Wenbiao Zheng 3088c122d8 eth/downloader: fix comment typos (#17956) 2018-10-23 13:21:16 +02:00
Péter Szilágyi 6ee3b26f44
eth/downloader: fix invalid hash chain error due to head mini reorg 2018-10-05 10:45:02 +03:00
Wenbiao Zheng 024b22c30e eth/downloader: use intermediate variable for better readability (#17510) 2018-09-29 22:13:39 +02:00
williambannas 9402f96597 eth: conform better to the golint standards (#16783)
* eth: made changes to conform better to the golint standards

* eth: fix comment nit
2018-06-14 13:14:52 +03:00
kiel barry 84f8c0cc1f common: improve documentation comments (#16701)
This commit adds many comments and removes unused code.
It also removes the EmptyHash function, which had some uses
but was silly.
2018-05-29 12:42:21 +02:00
Péter Szilágyi 6cf0ab38bd
core/rawdb: separate raw database access to own package (#16666) 2018-05-07 14:35:06 +03:00
Domino Valdano 7cf83cee52 eth/downloader: fix for Issue #16539 (#16546) 2018-04-23 10:01:21 +03:00
gary rong de2a7bb764 eth/downloader: wait for all fetcher goroutines to exit before terminating (#16509) 2018-04-16 11:37:48 +03:00
Yusup 7aad81f881 eth: fix typos (#16414) 2018-04-04 12:25:02 +02:00
gary rong 77da203547 eth: update higest block we know during the sync if a higher was found (#16283)
* eth: update higest block we know during the sync if a higher was found

* eth: avoid useless sync in fast sync
2018-03-09 11:51:30 +02:00
Péter Szilágyi ca64a122d3 eth/downloader: save and load trie sync progress (#16224) 2018-03-03 00:52:39 +01:00
Anton Evangelatov ae9f97221a metrics: pull library and introduce ResettingTimer and InfluxDB reporter (#15910)
* go-metrics: fork library and introduce ResettingTimer and InfluxDB reporter.

* vendor: change nonsense/go-metrics to ethersphere/go-metrics

* go-metrics: add tests. move ResettingTimer logic from reporter to type.

* all, metrics: pull in metrics package in go-ethereum

* metrics/test: make sure metrics are enabled for tests

* metrics: apply gosimple rules

* metrics/exp, internal/debug: init expvar endpoint when starting pprof server

* internal/debug: tiny comment formatting fix
2018-02-23 11:56:08 +02:00
Péter Szilágyi 96dad6b6f6
eth/downloader: don't require state for ancestor lookups 2018-02-11 14:43:56 +02:00
Péter Szilágyi eb07dbb079
eth, light: minor light client startup cleanups 2018-02-08 07:49:23 +02:00
Péter Szilágyi 55599ee95d core, trie: intermediate mempool between trie and database (#15857)
This commit reduces database I/O by not writing every state trie to disk.
2018-02-05 17:40:32 +01:00
Martin Holst Swende bc0666fb27 eth/downloader: fix #15858 by checking if downloader dropPeer function is set (#15992) 2018-02-05 15:38:06 +02:00
Péter Szilágyi 566d5c0777
core, eth, les, light: get rid of redundant methods 2018-01-30 18:42:00 +02:00
jtakalai 5aa3eac22d eth/downloader: minor comments cleanup (#15495)
it's -> its

pet peeve, and I like to imagine I'm not alone.
2017-11-16 13:14:51 +02:00
ferhat elmas 86f6568f66 build: enable unconvert linter (#15456)
* build: enable unconvert linter

 - fixes #15453
 - update code base for failing cases

* cmd/puppeth: replace syscall.Stdin with os.Stdin.Fd() for unconvert linter
2017-11-10 19:06:45 +02:00
Péter Szilágyi c8e70186a6 Merge pull request #14973 from rjl493456442/fix_downloader
eth/downloader: exit loop when there is no more available task
2017-09-11 14:02:02 +03:00