Commit Graph

376 Commits

Author SHA1 Message Date
Kurkó Mihály 16e4d0e005 p2p: meter peer traffic, emit metered peer events (#17695)
This change extends the peer metrics collection:

- traces the life-cycle of the peers
- meters the peer traffic separately for every peer
- creates event feed for the peer events
- emits the peer events
2018-10-16 00:40:51 +02:00
Felix Lange 6f607de5d5
p2p, p2p/discover: add signed ENR generation (#17753)
This PR adds enode.LocalNode and integrates it into the p2p
subsystem. This new object is the keeper of the local node
record. For now, a new version of the record is produced every
time the client restarts. We'll make it smarter to avoid that in
the future.

There are a couple of other changes in this commit: discovery now
waits for all of its goroutines at shutdown and the p2p server
now closes the node database after discovery has shut down. This
fixes a leveldb crash in tests. p2p server startup is faster
because it doesn't need to wait for the external IP query
anymore.
2018-10-12 11:47:24 +02:00
Felix Lange dcae0d348b
p2p/simulations: fix a deadlock and clean up adapters (#17891)
This fixes a rare deadlock with the inproc adapter:

- A node is stopped, which acquires Network.lock.
- The protocol code being simulated (swarm/network in my case)
  waits for its goroutines to shut down.
- One of those goroutines calls into the simulation to add a peer,
  which waits for Network.lock.

The fix for the deadlock is really simple, just release the lock
before stopping the simulation node.

Other changes in this PR clean up the exec adapter so it reports
node startup errors better and remove the docker adapter because
it just adds overhead.

In the exec adapter, node information is now posted to a one-shot
server. This avoids log parsing and allows reporting startup
errors to the simulation host.

A small change in package node was needed because simulation
nodes use port zero. Node.{HTTP,WS}Endpoint now return the live
endpoints after startup by checking the TCP listener.
2018-10-11 20:32:14 +02:00
holisticode 11d0ff6578 Fix retrieval tests and simulation backends (#17723)
* swarm/network/stream: introduced visualized snapshot sync test

* swarm/network/stream: non-existing hash visualization sim

* swarm/network/stream: fixed retrieval tests; new backend for visualization

* swarm/network/stream: cleanup of visualized_snapshot_sync_sim_test.go

* swarm/network/stream: rebased PR on master

* swarm/network/stream: fixed loop logic in retrieval tests

* swarm/network/stream: fixed iterations for snapshot tests

* swarm/network/stream: address PR comments

* swarm/network/stream: addressed PR comments
2018-10-08 20:28:44 +02:00
Felix Lange 1895059119 p2p: add enode URL to PeerInfo (#17838) 2018-10-04 18:13:21 +03:00
Liang ZOU 6663e5da10 all: fix various comment typos (#17748) 2018-09-25 12:26:35 +02:00
Felix Lange 30cd5c1854
all: new p2p node representation (#17643)
Package p2p/enode provides a generalized representation of p2p nodes
which can contain arbitrary information in key/value pairs. It is also
the new home for the node database. The "v4" identity scheme is also
moved here from p2p/enr to remove the dependency on Ethereum crypto from
that package.

Record signature handling is changed significantly. The identity scheme
registry is removed and acceptable schemes must be passed to any method
that needs identity. This means records must now be validated explicitly
after decoding.

The enode API is designed to make signature handling easy and safe: most
APIs around the codebase work with enode.Node, which is a wrapper around
a valid record. Going from enr.Record to enode.Node requires a valid
signature.

* p2p/discover: port to p2p/enode

This ports the discovery code to the new node representation in
p2p/enode. The wire protocol is unchanged, this can be considered a
refactoring change. The Kademlia table can now deal with nodes using an
arbitrary identity scheme. This requires a few incompatible API changes:

  - Table.Lookup is not available anymore. It used to take a public key
    as argument because v4 protocol requires one. Its replacement is
    LookupRandom.
  - Table.Resolve takes *enode.Node instead of NodeID. This is also for
    v4 protocol compatibility because nodes cannot be looked up by ID
    alone.
  - Types Node and NodeID are gone. Further commits in the series will be
    fixes all over the the codebase to deal with those removals.

* p2p: port to p2p/enode and discovery changes

This adapts package p2p to the changes in p2p/discover. All uses of
discover.Node and discover.NodeID are replaced by their equivalents from
p2p/enode.

New API is added to retrieve the enode.Node instance of a peer. The
behavior of Server.Self with discovery disabled is improved. It now
tries much harder to report a working IP address, falling back to
127.0.0.1 if no suitable address can be determined through other means.
These changes were needed for tests of other packages later in the
series.

* p2p/simulations, p2p/testing: port to p2p/enode

No surprises here, mostly replacements of discover.Node, discover.NodeID
with their new equivalents. The 'interesting' API changes are:

 - testing.ProtocolSession tracks complete nodes, not just their IDs.
 - adapters.NodeConfig has a new method to create a complete node.

These changes were needed to make swarm tests work.

Note that the NodeID change makes the code incompatible with old
simulation snapshots.

* whisper/whisperv5, whisper/whisperv6: port to p2p/enode

This port was easy because whisper uses []byte for node IDs and
URL strings in the API.

* eth: port to p2p/enode

Again, easy to port because eth uses strings for node IDs and doesn't
care about node information in any way.

* les: port to p2p/enode

Apart from replacing discover.NodeID with enode.ID, most changes are in
the server pool code. It now deals with complete nodes instead
of (Pubkey, IP, Port) triples. The database format is unchanged for now,
but we should probably change it to use the node database later.

* node: port to p2p/enode

This change simply replaces discover.Node and discover.NodeID with their
new equivalents.

* swarm/network: port to p2p/enode

Swarm has its own node address representation, BzzAddr, containing both
an overlay address (the hash of a secp256k1 public key) and an underlay
address (enode:// URL).

There are no changes to the BzzAddr format in this commit, but certain
operations such as creating a BzzAddr from a node ID are now impossible
because node IDs aren't public keys anymore.

Most swarm-related changes in the series remove uses of
NewAddrFromNodeID, replacing it with NewAddr which takes a complete node
as argument. ToOverlayAddr is removed because we can just use the node
ID directly.
2018-09-25 00:59:00 +02:00
Emil 86a03f97d3 all: simplify s[:] to s where s is a slice (#17673) 2018-09-14 22:07:13 +02:00
HAOYUatHZ 5c0954afff p2p/discv5: make idx bounds checking more sound (#17571) 2018-09-03 16:47:20 +02:00
Wenbiao Zheng d1aa605f1e all: remove the duplicate 'the' in annotations (#17509) 2018-08-27 11:49:29 +03:00
Mymskmkt 1df1187d83 p2p: fix comment typo (#17491) 2018-08-23 11:47:43 +03:00
Wuxiang d3488c1aff p2p: fix typo (#17446) 2018-08-20 15:07:21 +03:00
Felföldi Zsolt b52bb31b76 p2p/discv5: add delay to refresh cycle when no seed nodes are found (#16994) 2018-08-14 22:59:18 +02:00
libotony 834057592f p2p/discv5: fix negative index after uint convert to int (#17274) 2018-08-09 10:03:42 +03:00
Oleg Kovalov cf05ef9106 p2p, swarm, trie: avoid copying slices in loops (#17265) 2018-08-07 13:56:40 +03:00
Felföldi Zsolt eef65b20fc p2p: use safe atomic operations when changing connFlags (#17325) 2018-08-06 15:46:30 +03:00
Felföldi Zsolt c4df67461f
Merge pull request #16333 from shazow/addremovetrustedpeer
rpc: Add admin_addTrustedPeer and admin_removeTrustedPeer.
2018-08-06 13:30:04 +02:00
Janoš Guljaš 8f4c4fea20 p2p: fix rare deadlock in Stop (#17260) 2018-07-30 12:44:17 +03:00
Oleg Kovalov d42ce0f2c1 all: simplify switches (#17267)
* all: simplify switches

* silly mistake
2018-07-30 12:30:09 +03:00
Viktor Trón b536460f8e
Merge pull request #17231 from ethersphere/develop
swarm: client-side MRU signatures ; BMT fixes ; network simulation tests
2018-07-24 08:44:43 +02:00
Wenbiao Zheng fe6a9473dc p2p: token is useless in xxxEncHandshake (#17230) 2018-07-23 17:36:08 +02:00
Janoš Guljaš dcaaa3c804 swarm: network simulation for swarm tests (#769)
* cmd/swarm: minor cli flag text adjustments

* cmd/swarm, swarm/storage, swarm: fix  mingw on windows test issues

* cmd/swarm: support for smoke tests on the production swarm cluster

* cmd/swarm/swarm-smoke: simplify cluster logic as per suggestion

* changed colour of landing page

* landing page reacts to enter keypress

* swarm/api/http: sticky footer for swarm landing page using flex

* swarm/api/http: sticky footer for error pages and fix for multiple choices

* swarm: propagate ctx to internal apis (#754)

* swarm/simnet: add basic node/service functions

* swarm/netsim: add buckets for global state and kademlia health check

* swarm/netsim: Use sync.Map as bucket and provide cleanup function for...

* swarm, swarm/netsim: adjust SwarmNetworkTest

* swarm/netsim: fix tests

* swarm: added visualization option to sim net redesign

* swarm/netsim: support multiple services per node

* swarm/netsim: remove redundant return statement

* swarm/netsim: add comments

* swarm: shutdown HTTP in Simulation.Close

* swarm: sim HTTP server timeout

* swarm/netsim: add more simulation methods and peer events examples

* swarm/netsim: add WaitKademlia example

* swarm/netsim: fix comments

* swarm/netsim: terminate peer events goroutines on simulation done

* swarm, swarm/netsim: naming updates

* swarm/netsim: return not healthy kademlias on WaitTillHealthy

* swarm: fix WaitTillHealthy call in testSwarmNetwork

* swarm/netsim: allow bucket to have any type for a key

* swarm: Added snapshots to new netsim

* swarm/netsim: add more tests for bucket

* swarm/netsim: move http related things into separate files

* swarm/netsim: add AddNodeWithService option

* swarm/netsim: add more tests and Start* methods

* swarm/netsim: add peer events and kademlia tests

* swarm/netsim: fix some tests flakiness

* swarm/netsim: improve random nodes selection, fix TestStartStop* tests

* swarm/netsim: remove time measurement from TestClose to avoid flakiness

* swarm/netsim: builder pattern for netsim HTTP server (#773)

* swarm/netsim: add connect related tests

* swarm/netsim: add comment for TestPeerEvents

* swarm: rename netsim package to network/simulation
2018-07-23 15:33:25 +02:00
jkcomment 65c91ad5e7 p2p: correct comments typo (#17184) 2018-07-18 10:41:18 +03:00
Anton Evangelatov 7c9314f231 swarm: integrate OpenTracing; propagate ctx to internal APIs (#17169)
* swarm: propagate ctx, enable opentracing

* swarm/tracing: log error when tracing is misconfigured
2018-07-13 17:40:28 +02:00
Felix Lange c73b654fd1 p2p/discover: move bond logic from table to transport (#17048)
* p2p/discover: move bond logic from table to transport

This commit moves node endpoint verification (bonding) from the table to
the UDP transport implementation. Previously, adding a node to the table
entailed pinging the node if needed. With this change, the ping-back
logic is embedded in the packet handler at a lower level.

It is easy to verify that the basic protocol is unchanged: we still
require a valid pong reply from the node before findnode is accepted.

The node database tracked the time of last ping sent to the node and
time of last valid pong received from the node. Node endpoints are
considered verified when a valid pong is received and the time of last
pong was called 'bond time'. The time of last ping sent was unused. In
this commit, the last ping database entry is repurposed to mean last
ping _received_. This entry is now used to track whether the node needs
to be pinged back.

The other big change is how nodes are added to the table. We used to add
nodes in Table.bond, which ran when a remote node pinged us or when we
encountered the node in a neighbors reply. The transport now adds to the
table directly after the endpoint is verified through ping. To ensure
that the Table can't be filled just by pinging the node repeatedly, we
retain the isInitDone check. During init, only nodes from neighbors
replies are added.

* p2p/discover: reduce findnode failure counter on success

* p2p/discover: remove unused parameter of loadSeedNodes

* p2p/discover: improve ping-back check and comments

* p2p/discover: add neighbors reply nodes always, not just during init
2018-07-03 16:24:12 +03:00
ethersphere e187711c65 swarm: network rewrite merge 2018-06-21 21:10:31 +02:00
Andrey Petrov 6209545083 p2p: Wrap conn.flags ops with atomic.Load/Store 2018-06-21 12:22:47 -04:00
Andrey Petrov 193a402cc0 p2p: Test for peer.rw.flags race conditions 2018-06-21 12:22:47 -04:00
Andrey Petrov dcca66bce8 p2p: Cache inbound flag on Peer.isInbound to avoid a race 2018-06-21 12:22:47 -04:00
Andrey Petrov 399aa710d5 p2p: Attempt to race check peer.Inbound() in TestServerDial 2018-06-21 12:22:47 -04:00
Andrey Petrov 699794d88d p2p: More tests for AddTrustedPeer/RemoveTrustedPeer 2018-06-21 12:22:47 -04:00
Andrey Petrov 773857a524 p2p: Test for MaxPeers=0 and TrustedPeer override 2018-06-21 12:21:48 -04:00
Andrey Petrov 2a75fe3308 rpc: Add admin_addTrustedPeer and admin_removeTrustedPeer.
These RPC calls are analogous to Parity's parity_addReservedPeer and
parity_removeReservedPeer.

They are useful for adjusting the trusted peer set during runtime,
without requiring restarting the server.
2018-06-21 12:21:48 -04:00
Elad 1836366ac1 all: library changes for swarm-network-rewrite (#16898)
This commit adds all changes needed for the merge of swarm-network-rewrite.
The changes:

- build: increase linter timeout
- contracts/ens: export ensNode
- log: add Output method and enable fractional seconds in format
- metrics: relax test timeout
- p2p: reduced some log levels, updates to simulation packages
- rpc: increased maxClientSubscriptionBuffer to 20000
2018-06-14 11:21:17 +02:00
Felix Lange 0255951587 crypto: replace ToECDSAPub with error-checking func UnmarshalPubkey (#16932)
ToECDSAPub was unsafe because it returned a non-nil key with nil X, Y in
case of invalid input. This change replaces ToECDSAPub with
UnmarshalPubkey across the codebase.
2018-06-12 15:26:08 +02:00
Dmitry Shulyak 7677ec1f34 p2p/discv5: add egress/ingress traffic metrics to discv5 udp transport (#16369) 2018-05-29 13:46:09 +02:00
Felix Lange 6286c255f1
p2p/enr: updates for discovery v4 compatibility (#16679)
This applies spec changes from ethereum/EIPs#1049 and adds support for
pluggable identity schemes.

Some care has been taken to make the "v4" scheme standalone. It uses
public APIs only and could be moved out of package enr at any time.

A couple of minor changes were needed to make identity schemes work:

- The sequence number is now updated in Set instead of when signing.
- Record is now copy-safe, i.e. calling Set on a shallow copy doesn't
  modify the record it was copied from.
2018-05-17 15:11:27 +02:00
Guilherme Salgado c60f6f6214 p2p: don't discard reason set by Disconnect (#16559)
Peer.run was discarding the reason for disconnection sent to the disc
channel by Disconnect.
2018-05-09 01:20:20 +02:00
Ivan Daniluk c4a4613d95 p2p/simulations/adapters: fix websocket log line parsing in exec adapter (#16667) 2018-05-08 17:05:27 +02:00
kiel barry 864e80a48f p2p: fix some golint warnings (#16577) 2018-05-08 13:08:43 +02:00
GagziW f2447bd4c3 p2p: changed if-else blocks to conform with golint (#16660) 2018-05-03 11:33:39 +03:00
thomasmodeneis ba1030b6b8 build: enable goimports and varcheck linters (#16446) 2018-04-18 00:53:50 +02:00
David Huie 23ac783332 ecies: drop randomness parameter from `PrivateKey.Decrypt` (#16374)
The parameter `rand` is unused in `PrivateKey.Decrypt`. Decryption in
the ECIES encryption scheme is deterministic, so randomness isn't
needed.
2018-03-26 13:46:18 +03:00
JU HYEONG PARK 61c9730b2d p2p: fix doEncHandshake documentation (#16184) 2018-02-26 17:22:46 +01:00
Anton Evangelatov ae9f97221a metrics: pull library and introduce ResettingTimer and InfluxDB reporter (#15910)
* go-metrics: fork library and introduce ResettingTimer and InfluxDB reporter.

* vendor: change nonsense/go-metrics to ethersphere/go-metrics

* go-metrics: add tests. move ResettingTimer logic from reporter to type.

* all, metrics: pull in metrics package in go-ethereum

* metrics/test: make sure metrics are enabled for tests

* metrics: apply gosimple rules

* metrics/exp, internal/debug: init expvar endpoint when starting pprof server

* internal/debug: tiny comment formatting fix
2018-02-23 11:56:08 +02:00
Ivan Daniluk 8522b31221 p2p: remove unused code (#16158)
* p2p: remove unused code

* p2p: remove unused imports
2018-02-22 19:20:28 +02:00
Balint Gabor 221486a291
Merge pull request #15919 from ethersphere/p2p-protocols-pr
p2p/protocols, p2p/testing: protocol abstraction and testing
2018-02-22 15:02:51 +01:00
Anton Evangelatov 1e457b6599 p2p: don't send DiscReason when using net.Pipe (#16004) 2018-02-22 11:41:06 +01:00
Felix Lange 28b20cff4b p2p/protocols: gofmt -w -s 2018-02-22 11:37:57 +01:00
Dmitry Shulyak 14c76371ba p2p: when peer is removed remove it also from dial history (#16060)
This change removes a peer information from dialing history
when peer is removed from static list. It allows to force a
server to re-dial concrete peer if it is needed.

In our case we are running geth node on mobile devices, and
it is common for a network connection to flap on mobile.
Almost every time it flaps or network connection is changed
from cellular to wifi peers are disconnected with read
timeout. And usually it takes 30 seconds (default expiration
timeout) to recover connection with static peers after
connectivity is restored.

This change allows us to reconnect with peers almost
immediately and it seems harmless enough.
2018-02-21 15:03:26 +01:00