Commit Graph

10978 Commits

Author SHA1 Message Date
Ferenc Szabo 50b872bf05 p2p, swarm: fix node up races by granular locking (#18976)
* swarm/network: DRY out repeated giga comment

I not necessarily agree with the way we wait for event propagation.
But I truly disagree with having duplicated giga comments.

* p2p/simulations: encapsulate Node.Up field so we avoid data races

The Node.Up field was accessed concurrently without "proper" locking.
There was a lock on Network and that was used sometimes to access
the  field. Other times the locking was missed and we had
a data race.

For example: https://github.com/ethereum/go-ethereum/pull/18464
The case above was solved, but there were still intermittent/hard to
reproduce races. So let's solve the issue permanently.

resolves: ethersphere/go-ethereum#1146

* p2p/simulations: fix unmarshal of simulations.Node

Making Node.Up field private in 13292ee897
broke TestHTTPNetwork and TestHTTPSnapshot. Because the default
UnmarshalJSON does not handle unexported fields.

Important: The fix is partial and not proper to my taste. But I cut
scope as I think the fix may require a change to the current
serialization format. New ticket:
https://github.com/ethersphere/go-ethereum/issues/1177

* p2p/simulations: Add a sanity test case for Node.Config UnmarshalJSON

* p2p/simulations: revert back to defer Unlock() pattern for Network

It's a good patten to call `defer Unlock()` right after `Lock()` so
(new) error cases won't miss to unlock. Let's get back to that pattern.

The patten was abandoned in 85a79b3ad3,
while fixing a data race. That data race does not exist anymore,
since the Node.Up field got hidden behind its own lock.

* p2p/simulations: consistent naming for test providers Node.UnmarshalJSON

* p2p/simulations: remove JSON annotation from private fields of Node

As unexported fields are not serialized.

* p2p/simulations: fix deadlock in Network.GetRandomDownNode()

Problem: GetRandomDownNode() locks -> getDownNodeIDs() ->
GetNodes() tries to lock -> deadlock

On Network type, unexported functions must assume that `net.lock`
is already acquired and should not call exported functions which
might try to lock again.

* p2p/simulations: ensure method conformity for Network

Connect* methods were moved to p2p/simulations.Network from
swarm/network/simulation. However these new methods did not follow
the pattern of Network methods, i.e., all exported method locks
the whole Network either for read or write.

* p2p/simulations: fix deadlock during network shutdown

`TestDiscoveryPersistenceSimulationSimAdapter` often got into deadlock.
The execution was stuck on two locks, i.e, `Kademlia.lock` and
`p2p/simulations.Network.lock`. Usually the test got stuck once in each
20 executions with high confidence.

`Kademlia` was stuck in `Kademlia.EachAddr()` and `Network` in
`Network.Stop()`.

Solution: in `Network.Stop()` `net.lock` must be released before
calling `node.Stop()` as stopping a node (somehow - I did not find
the exact code path) causes `Network.InitConn()` to be called from
`Kademlia.SuggestPeer()` and that blocks on `net.lock`.

Related ticket: https://github.com/ethersphere/go-ethereum/issues/1223

* swarm/state: simplify if statement in DBStore.Put()

* p2p/simulations: remove faulty godoc from private function

The comment started with the wrong method name.

The method is simple and self explanatory. Also, it's private.
=> Let's just remove the comment.
2019-02-18 07:38:14 +01:00
gluk256 12ca3b172a swarm/pss: refactoring (#19110)
* swarm/pss: split pss and keystore

* swarm/pss: moved whisper to keystore

* swarm/pss: goimports fixed
2019-02-17 06:29:41 +01:00
Martin Holst Swende 4f85c2b88b
trie: fix error in node decoding (#19111) 2019-02-16 16:16:12 +01:00
Elad 5b8ae7885e swarm/storage: fix influxdb gc metrics report (#19102) 2019-02-15 07:41:42 +01:00
Péter Szilágyi 7f6b05aa43
Merge pull request #19087 from karalabe/ios-fix-take-2
vendor: pull in upstream syscall fixes for non-linux/arm64
2019-02-15 02:01:28 +02:00
Matthew Halpern fa87929a2f cmd: prefer nil slices over zero-length slices (#19077) 2019-02-15 01:02:11 +02:00
Matthew Halpern e26a119c9b console: prefer nil slices over zero-length slices (#19076) 2019-02-15 00:59:54 +02:00
Péter Szilágyi 9d3ea8df1c
vendor: pull in upstream syscall fixes for non-linux/arm64 2019-02-15 00:51:34 +02:00
Matthew Halpern 2b75fa9d61 core: enforce camel case variable names (#19058) 2019-02-14 21:14:05 +02:00
holisticode 2af24724dd swarm/network: Saturation check for healthy networks (#19071)
* swarm/network: new saturation for  implementation

* swarm/network: re-added saturation func in Kademlia as it is used elsewhere

* swarm/network: saturation with higher MinBinSize

* swarm/network: PeersPerBin with depth check

* swarm/network: edited tests to pass new saturated check

* swarm/network: minor fix saturated check

* swarm/network/simulations/discovery: fixed renamed RPC call

* swarm/network: renamed to isSaturated and returns bool

* swarm/network: early depth check
2019-02-14 19:01:50 +01:00
Péter Szilágyi fab8c5a1cd
Merge pull request #19072 from karalabe/update-syscalls
vendor: update syscalls dependency
2019-02-14 18:42:02 +02:00
Péter Szilágyi dcc045f03c
vendor: update syscalls dependency 2019-02-14 18:14:28 +02:00
Felix Lange ba90a4aaa4 common/fdlimit: fix windows build (#19068) 2019-02-14 17:32:22 +02:00
Matthew Halpern 325334f61a light: enforce camel case variable names (#19054) 2019-02-14 17:18:32 +02:00
Felix Lange a8ddf7ad83 build: avoid dput and upload with sftp directly (#19067) 2019-02-14 17:10:09 +02:00
Matthew Halpern 7d24a73192 eth/tracers: enforce camel case variable names (#19057) 2019-02-14 16:38:55 +02:00
Matthew Halpern e6c06a1da8 console, internal: enforce camel case variable names (#19059) 2019-02-14 16:38:07 +02:00
Elad 3ee09ba035 swarm/storage/netstore: add fetcher cancellation on shutdown (#19049)
swarm/network/stream: remove netstore internal wg
swarm/network/stream: run individual tests with t.Run
2019-02-14 07:51:57 +01:00
Martin Holst Swende e9f70c9064
clef: documentation generator + docs (#19020)
* clef: implement documentation generation + remove unused struct

* clef: formatting + spelling

* clef: updates to doc
2019-02-13 21:37:59 +01:00
Janoš Guljaš 3fd6db2bf6 swarm: fix network/stream data races (#19051)
* swarm/network/stream: newStreamerTester cleanup only if err is nil

* swarm/network/stream: raise newStreamerTester waitForPeers timeout

* swarm/network/stream: fix data races in GetPeerSubscriptions

* swarm/storage: prevent data race on LDBStore.batchesC

https://github.com/ethersphere/go-ethereum/issues/1198#issuecomment-461775049

* swarm/network/stream: fix TestGetSubscriptionsRPC data race

https://github.com/ethersphere/go-ethereum/issues/1198#issuecomment-461768477

* swarm/network/stream: correctly use Simulation.Run callback

https://github.com/ethersphere/go-ethereum/issues/1198#issuecomment-461783804

* swarm/network: protect addrCountC in Kademlia.AddrCountC function

https://github.com/ethersphere/go-ethereum/issues/1198#issuecomment-462273444

* p2p/simulations: fix a deadlock calling getRandomNode with lock

https://github.com/ethersphere/go-ethereum/issues/1198#issuecomment-462317407

* swarm/network/stream: terminate disconnect goruotines in tests

* swarm/network/stream: reduce memory consumption when testing data races

* swarm/network/stream: add watchDisconnections helper function

* swarm/network/stream: add concurrent counter for tests

* swarm/network/stream: rename race/norace test files and use const

* swarm/network/stream: remove watchSim and its panic

* swarm/network/stream: pass context in watchDisconnections

* swarm/network/stream: add concurrent safe bool for watchDisconnections

* swarm/storage: fix LDBStore.batchesC data race by not closing it
2019-02-13 13:03:23 +01:00
Elad d596bea2d5 swarm: fix uptime gauge update goroutine leak by introducing cleanup functions (#19040) 2019-02-13 08:15:03 +01:00
Dan Kinsley 555b3652dc accounts/abi/bind/backends: add TransactionByHash to SimulatedBackend (#19026) 2019-02-13 00:23:10 +01:00
holisticode 3d22a46c94 swarm/storage: fix HashExplore concurrency bug ethersphere#1211 (#19028)
* swarm/storage: fix HashExplore concurrency bug ethersphere#1211

*  swarm/storage: lock as value not pointer

* swarm/storage: wait for  to complete

* swarm/storage: fix linter problems

* swarm/storage: append to nil slice
2019-02-13 00:17:44 +01:00
gluk256 b30109df3c swarm/pss: mutex lifecycle fixed (#19045) 2019-02-13 00:12:41 +01:00
Martin Holst Swende 8771fbf3c8
rpc: make stdio usable over custom channels (#19046) 2019-02-12 18:05:28 +01:00
Martin Holst Swende b5d471a739
clef: bidirectional communication with UI (#19018)
* clef: initial implementation of bidirectional RPC communication for the UI

* signer: fix tests to pass + formatting

* clef: fix unused import + formatting

* signer: gosimple nitpicks
2019-02-12 17:38:46 +01:00
Martin Holst Swende 75d292bcf6
clef: external signing fixes + signing data (#19003)
* signer/clef: make use of json-rpc notification

* signer: tidy up output of OnApprovedTx

* accounts/external, signer: implement remote signing of text, make accounts_sign take hexdata

* clef: added basic testscript

* signer, external, api: add clique signing test to debug rpc, fix clique signing in clef

* signer: fix clique interoperability between geth and clef

* clef: rename networkid switch to chainid

* clef: enable chainid flag

* clef, signer: minor changes from review

* clef: more tests for signer
2019-02-12 14:00:02 +01:00
Felix Lange edf976ee8e
.travis.yml: fix upload destination (#19043) 2019-02-12 13:02:40 +01:00
Martin Holst Swende f48da43bae common/fdlimit: cap on MacOS file limits, fixes #18994 (#19035)
* common/fdlimit: cap on MacOS file limits, fixes #18994

* common/fdlimit: fix Maximum-check to respect OPEN_MAX

* common/fdlimit: return error if OPEN_MAX is exceeded in Raise()

* common/fdlimit: goimports

* common/fdlimit: check value after setting fdlimit

* common/fdlimit: make comment a bit more descriptive

* cmd/utils: make fdlimit happy path a bit cleaner
2019-02-12 12:29:05 +02:00
Felix Lange 3de19c8b31 build: use SFTP for launchpad uploads (#19037)
* build: use sftp for launchpad uploads

* .travis.yml: configure sftp export

* build: update CI docs
2019-02-12 11:55:25 +02:00
Rafael Matias 6cb7d52a29 swarm/docker: add global-store and split docker images (#19038) 2019-02-12 08:34:08 +01:00
Ferenc Szabo 27e3f96819 swarm: CI race detector test adjustments (#19017) 2019-02-08 17:07:11 +01:00
gluk256 cde02e017e swarm/pss: transition to whisper v6 (#19023) 2019-02-08 17:05:10 +01:00
lash 0c10d37606 swarm/network, swarm/storage: Preserve opentracing contexts (#19022) 2019-02-08 16:57:48 +01:00
Péter Szilágyi 0436412412
Merge pull request #18988 from holiman/repro18977
core: repro #18977
2019-02-08 12:27:15 +02:00
Péter Szilágyi 940e317094
core: fix pruner panic when importing low-diff-large-sidechain 2019-02-08 11:56:25 +02:00
Martin Holst Swende da1efdae0c
core: repro #18977 2019-02-08 10:29:51 +02:00
Janoš Guljaš 4f3d22f06c swarm/storage/localstore: new localstore package (#19015) 2019-02-07 18:40:26 +01:00
holisticode 41597c2856 swarm: Debug API and HasChunks() API endpoint (#18980) 2019-02-07 15:49:19 +01:00
Janoš Guljaš 33d0a0efa6 cmd/swarm/global-store: global store cmd (#19014) 2019-02-07 15:46:58 +01:00
Péter Szilágyi 685eec3128
Merge pull request #19012 from holiman/default155
ethapi: default to use eip-155 protected transactions
2019-02-07 16:19:28 +02:00
Péter Szilágyi 9fa4c3ce94
Merge pull request #18991 from karalabe/archive-write-cache
cmd/utils, eth: relinquish GC cache to read cache in archive mode
2019-02-07 16:03:43 +02:00
Anton Evangelatov d212535ddd
cmd/swarm/swarm-smoke: refactor generateEndpoints (#19006) 2019-02-07 14:38:32 +01:00
holisticode 7f55b0cbd8 cmd/swarm: hashes command (#19008) 2019-02-07 13:51:24 +01:00
Péter Szilágyi d6225ab846
cmd/utils, eth: relinquish GC cache to read cache in archive mode 2019-02-07 14:36:25 +02:00
Péter Szilágyi 85b3b1c8d6
Merge pull request #16619 from kielbarry/contractsgolint
contracts/*: golint updates for this or self warning
2019-02-07 14:07:03 +02:00
Péter Szilágyi 159e991196
contracts/chequebook: polishes and naked return removals 2019-02-07 13:16:55 +02:00
Kiel barry 53b823afc8
contracts/*: golint updates for this or self warning 2019-02-07 13:15:14 +02:00
Martin Holst Swende 2ac61a9914
ethapi: default to use eip-155 protected transactions 2019-02-07 11:59:17 +01:00
Matthew Halpern 6f714ed73e light: make chain receiver names consistent (#18997) 2019-02-07 12:53:45 +02:00