diff --git a/assets/node_architecture.png b/assets/node_architecture.png new file mode 100644 index 0000000000..5b0bd1baad Binary files /dev/null and b/assets/node_architecture.png differ diff --git a/assets/node_basic.png b/assets/node_basic.png new file mode 100644 index 0000000000..2add22296c Binary files /dev/null and b/assets/node_basic.png differ diff --git a/content/about-geth/about_geth.md b/content/about-geth/about_geth.md new file mode 100644 index 0000000000..c66810c594 --- /dev/null +++ b/content/about-geth/about_geth.md @@ -0,0 +1,86 @@ +--- +title: What is Geth +root: .. +--- + +## What is Geth? + +Geth (go-ethereum) is a [Go](https://go.dev/) implementation of [Ethereum](http://ethereum.org) - a +gateway into the decentralized web. + +Running Geth alongside a consensus client turns a computer into an Ethereum node. +Nodes communicate with one another, agreeing on the data they should each add to their local databases. +Ethereum itself is the network of connected nodes running Ethereum software. + + +## Why run a node? + +Running your own node enables you to use Ethereum in a truly private, self-sufficient and trustless +manner. You don't need to trust information you receive because you can verify the data yourself +using your Geth instance. + +**"Don't trust, verify"** + +![node basic](/assets/node-basic.png) + +Your node verifies all changes to its database by itself. This means: + +- you don’t have to trust any other nodes in the network. +- You never have to leak your addresses and balances to other nodes. +- You can use Ethereum securely and privately. Most wallet software can be pointed to your own local node. +- You can program your own custom RPC endpoints and make your own modifications to the source code. +- You get low latency, fast access to Ethereum. + +A large and diverse set of nodes independently verifying new information is critical for Ethereum’s health, +security and operational resiliency. + +**If you run a full node, the whole Ethereum network benefits.** + + +## Node architecture + +Geth is an [execution client](https://ethereum.org/en/developers/docs/nodes-and-clients/#execution-clients). +Originally, an execution client alone was enough to run a full Ethereum node. +However, ever since Ethereum turned off proof-of-work and implemented proof-of-stake, +Geth must to be coupled to another piece of software called a +[“consensus client”](https://ethereum.org/en/developers/docs/nodes-and-clients/#consensus-clients). + +The execution client is responsible for transaction handling, transaction gossip, state management and +the Ethereum Virtual Machine (EVM). However, Geth is **not** responsible for block building, block gossiping +or handling consensus logic. These are in the remit of the consensus client. + +The relationship between the two Ethereum clients is shown in the schematic below. The two clients each +connect to their own respective peer-to-peer (P2P) networks. This is because the execution clients gossip +transactions over their P2P network enabling them to manage their local transaction pool. The consensus clients +gossip blocks over their P2P network, enabling consensus and chain growth. + +![node-architecture](/assets/node_architecture.png) + +For this two-client structure to work, consensus clients must be able to pass bundles of transactions to +Geth to be executed. Executing the transactions locally is how the client validates that the transactions +do not violate any Ethereum rules and that the proposed update to Ethereum’s state is correct. Likewise, +when the node is selected to be a block producer the consensus client must be able to request bundles of +transactions from Geth to include in the new block. This inter-client communication is handled by a local +RPC connection using the engine API which is part of the JSON-RPC API exposed by Geth. + + + +## What does Geth do? + +As an execution client, Geth is responsible for creating the execution payloads - the bundles of transactions - +that consensus clients include in their blocks. Geth is also responsible for re-executing transactions that arrive +in new blocks to ensure they are valid. Executing transactions is done on Geth's embedded computer, known as the +Ethereum Virtual Machine (EVM). + +Geth also offers a user-interface to Ethereum by exposing a set of RPC methods that enable users to query the +Ethereum blockchain, submit transactions and deploy smart contracts using the command line, programmatically +using Geth's built-in console, web3 development frameworks such as Hardhat and Truffle or via web-apps and wallets. + +In summary, Geth is: + - a user gateway to Ethereum + - home to the Ethereum Virtual Machine, Ethereum's state and transaction pool. + + + + + diff --git a/content/about-geth/about_us.md b/content/about-geth/about_us.md new file mode 100644 index 0000000000..e69de29bb2 diff --git a/content/about-geth/contributing.md b/content/about-geth/contributing.md new file mode 100644 index 0000000000..f01d2c1a59 --- /dev/null +++ b/content/about-geth/contributing.md @@ -0,0 +1,34 @@ +--- +title: Contributing +--- + +We welcome contributions from anyone on the internet, and are grateful for even the smallest of fixes! + +## Contributing to the Geth source code + +If you'd like to contribute to the Geth source code, please fork the +[Github repository](https://github.com/ethereum/go-ethereum), fix, commit and send a pull request for the +maintainers to review and merge into the main code base. If you wish to submit more complex changes +though, please check up with the core devs first on our Discord Server to ensure those changes are in +line with the general philosophy of the project and/or get some early feedback which can make both your +efforts much lighter as well as our review and merge procedures quick and simple. + +Please make sure your contributions adhere to our coding guidelines: + +* Code must adhere to the official Go formatting guidelines (i.e. uses gofmt). +* Code must be documented adhering to the official Go commentary guidelines. +* Pull requests need to be based on and opened against the master branch. +* Commit messages should be prefixed with the package(s) they modify. + E.g. "eth, rpc: make trace configs optional" + + +## Contributing to the Geth website + +The Geth website is hosted separately from Geth itself. The contribution guidelines are the same. Please +for the Geth website Github repository and raise pull requests for the maintainers to review and merge. + +## License + +The go-ethereum library (i.e. all code outside of the cmd directory) is licensed under the GNU Lesser General Public License v3.0, also included in our repository in the COPYING.LESSER file. + +The go-ethereum binaries (i.e. all code inside of the cmd directory) is licensed under the GNU General Public License v3.0, also included in our repository in the COPYING file. diff --git a/content/about-geth/ethereum.md b/content/about-geth/ethereum.md new file mode 100644 index 0000000000..7515053e4c --- /dev/null +++ b/content/about-geth/ethereum.md @@ -0,0 +1,56 @@ +--- +title: Intro to Ethereum +description: A brief introduction to Ethereum. +--- + + +Ethereum is a technology for building apps and organizations, holding assets, transacting and +communicating without being controlled by a central authority. There is no need to hand over all +your personal details to use Ethereum - you keep control of your own data and what is being shared. +Ethereum has its own cryptocurrency, Ether (ETH), which is used to pay for certain activities on +the Ethereum network. In essence, Ethereum is a blockchain with an embedded computer. + + +## What is a blockchain? + +A blockchain is a database of transactions that is updated and shared across many computers in a +network. Every time a new set of transactions is added, its called a “block” - hence the name +blockchain. Most blockchains are public and immutable, and you can only add data, not remove. If someone +wanted to alter any of the information or cheat the system, they’d need to do so in such a way that the +majority of computers on the network accept. There are very strong crypto-economic defenses against this +on Ethereum. This makes established blockchains like Ethereum highly secure base-layers for organizations +and applications. + + +## What are smart contracts? + +Smart contracts are computer programs living on the Ethereum blockchain. They only execute when +triggered by a transaction from a user (or another contract). They make Ethereum very flexible in what +it can do and distinguish it from other cryptocurrencies. These programs are what we now call +decentralized apps, or dapps. + +Once a smart contract is published to Ethereum, it will be online and operational for as long as Ethereum +exists. Not even the author can take it down. Since smart contracts are automated, they do not discriminate +against any user and are always ready to use. + +Popular examples of smart contracts are lending apps, decentralized trading exchanges, insurance, +crowdfunding apps - basically anything you can think of. + + +## Who runs Ethereum? + +Ethereum is not controlled by any one entity. It exists solely through the decentralized participation +and cooperation of the community. Ethereum makes use of nodes (a computer with a copy of the Ethereum +blockchain data) run by volunteers to replace individual server and cloud systems owned by major +internet providers and services. + +These distributed nodes, run by individuals and businesses all over the world, provide resiliency to +the Ethereum network infrastructure. It is therefore much less vulnerable to hacks or shutdowns. +Since its launch in 2015, Ethereum has never suffered downtime. There are thousands of individual nodes +running Ethereum network. + + +## Learn more about Ethereum + +[ethereum.org](https://ethereum.org/) + diff --git a/content/about-geth/faq.md b/content/about-geth/faq.md new file mode 100644 index 0000000000..8b04b29e3d --- /dev/null +++ b/content/about-geth/faq.md @@ -0,0 +1,74 @@ +--- +title: FAQ +permalink: docs/faq +sort_key: C +--- + +* TOC +{:toc} + +#### I noticed my peercount slowly decreasing, and now it is at 0. Restarting doesn't get any peers. + +Check and sync your clock with ntp. For example, you can [force a clock update using ntp](https://askubuntu.com/questions/254826/how-to-force-a-clock-update-using-ntp) like so: + +```sh +sudo ntpdate -s time.nist.gov +``` + +#### I would like to run multiple geth instances but got the error "Fatal: blockchain db err: resource temporarily unavailable". + +Geth uses a datadir to store the blockchain, accounts and some additional information. This directory cannot be shared between running instances. If you would like to run multiple instances follow [these](getting-started/private-net) instructions. + +#### When I try to use the --password command line flag, I get the error "Could not decrypt key with given passphrase" but the password is correct. Why does this error appear? + +Especially if the password file was created on Windows, it may have a Byte Order Mark or other special encoding that the go-ethereum client doesn't currently recognize. You can change this behavior with a PowerShell command like: + +```sh +echo "mypasswordhere" | out-file test.txt -encoding ASCII +``` + +Additional details and/or any updates on more robust handling are at . + +#### How does Ethereum syncing work? + +The current default syncing mode used by Geth is called [snap sync](https://github.com/ethereum/devp2p/blob/master/caps/snap.md). Instead of starting from the genesis block and processing all the transactions that ever occurred (which could take weeks), snap sync downloads the blocks, and only verifies the associated proof-of-works. Downloading all the blocks is a straightforward and fast procedure and will relatively quickly reassemble the entire chain. + +Many people falsely assume that because they have the blocks, they are in sync. Unfortunately this is not the case. Since no transaction was executed, so we do not have any account state available (ie. balances, nonces, smart contract code and data). These need to be downloaded separately and cross-checked with the latest blocks. This phase is called the state trie download phase. Snap sync tries to hasten this process by downloading contiguous chunks of useful state data, instead of doing so one-by-one, as in previous synchronization methods. + +#### So, what's the state trie? + +In the Ethereum mainnet, there are a ton of accounts already, which track the balance, nonce, etc of each user/contract. The accounts themselves are however insufficient to run a node, they need to be cryptographically linked to each block so that nodes can actually verify that the accounts are not tampered with. + +This cryptographic linking is done by creating a tree-like data structure, where each leaf corresponds to an account, and each intermediary level aggregates the layer below it into an ever smaller layer, until you reach a single root. This gigantic data structure containing all the accounts and the intermediate cryptographic proofs is called the state trie. + +#### Why does the state trie download phase require a special syncing mode? + +The trie data structure is an intricate interlink of hundreds of millions of tiny cryptographic proofs (trie nodes). To truly have a synchronized node, you need to download all the account data, as well as all the tiny cryptographic proofs to verify that no one in the network is trying to cheat you. This itself is already a crazy number of data items. + +The part where it gets even messier is that this data is constantly morphing: at every block (roughly 13s), about 1000 nodes are deleted from this trie and about 2000 new ones are added. This means your node needs to synchronize a dataset that is changing more than 200 times per second. Until you actually do gather all the data, your local node is not usable since it cannot cryptographically prove anything about any accounts. But while you're syncing the network is moving forward and most nodes on the network keep the state for only a limited number of recent blocks. Any sync algorithm needs to consider this fact. + +#### What happened to fast sync? + +Snap syncing was introduced by version [1.10.0](https://blog.ethereum.org/2021/03/03/geth-v1-10-0/) and was adopted as the default mode in version [1.10.4](https://github.com/ethereum/go-ethereum/releases/tag/v1.10.4). Before that, the default was the "fast" syncing mode, which was dropped in version [1.10.14](https://github.com/ethereum/go-ethereum/releases/tag/v1.10.14). Even though support for fast sync was dropped, Geth still serves the relevant `eth` requests to other client implementations still relying on it. The reason being that snap sync relies on an alternative data structure called the [snapshot](https://blog.ethereum.org/2020/07/17/ask-about-geth-snapshot-acceleration/) which not all clients implement. + +You can read more in the article posted above why snap sync replaced fast sync in Geth. Below is a table taken from the article summarising the benefits: + +![snap-fast](https://user-images.githubusercontent.com/129561/109820169-6ee0af00-7c3d-11eb-9721-d8484eee4709.png) + +#### When doing a fast sync, the node just hangs on importing state enties?! + +The node doesn’t hang, it just doesn’t know how large the state trie is in advance so it keeps on going and going and going until it discovers and downloads the entire thing. + +The reason is that a block in Ethereum only contains the state root, a single hash of the root node. When the node begins synchronizing, it knows about exactly 1 node and tries to download it. That node, can refer up to 16 new nodes, so in the next step, we’ll know about 16 new nodes and try to download those. As we go along the download, most of the nodes will reference new ones that we didn’t know about until then. This is why you might be tempted to think it’s stuck on the same numbers. It is not, rather it’s discovering and downloading the trie as it goes along. + +During this phase you might see that your node is 64 blocks behind mainnet. You aren't actually synchronized. That's a side-effect of how fast sync works and you need to wait out until all state entries are downloaded. + +#### I have good bandwidth, so why does downloading the state take so long when using fast sync? + +State sync is mostly limited by disk IO, not bandwidth. + +The state trie in Ethereum contains hundreds of millions of nodes, most of which take the form of a single hash referencing up to 16 other hashes. This is a horrible way to store data on a disk, because there's almost no structure in it, just random numbers referencing even more random numbers. This makes any underlying database weep, as it cannot optimize storing and looking up the data in any meaningful way. Snap sync solves this issue by adopting the Snapshot data structure. + +#### Wait, so I can't use fast sync on an HDD? + +Doing a "fast" sync on an HDD will take more time than you're willing to wait, because the data structures used are not optimized for HDDs. Even if you do wait it out, an HDD will not be able to keep up with the read/write requirements of transaction processing on mainnet. You however should be able to run a light client on an HDD with minimal impact on system resources. diff --git a/content/docs/developers/dapp-developer/custom-tracer.md b/content/docs/developers/dapp-developer/custom-tracer.md new file mode 100644 index 0000000000..4817ed4053 --- /dev/null +++ b/content/docs/developers/dapp-developer/custom-tracer.md @@ -0,0 +1,458 @@ +--- +title: Custom EVM tracer +sort_key: B +--- + +In addition to the default opcode tracer and the built-in tracers, Geth offers the possibility to write custom code +that hook to events in the EVM to process and return the data in a consumable format. Custom tracers can be +written either in Javascript or Go. JS tracers are good for quick prototyping and experimentation as well as for +less intensive applications. Go tracers are performant but require the tracer to be compiled together with the Geth source code. + +* TOC +{:toc} + +## Custom Javascript tracing + +Transaction traces include the complete status of the EVM at every point during the transaction execution, which +can be a very large amount of data. Often, users are only interested in a small subset of that data. Javascript trace +filters are available to isolate the useful information. Detailed information about `debug_traceTransaction` and its +component parts is available in the [reference documentation](/docs/rpc/ns-debug#debug_tracetransaction). + +### A simple filter + +Filters are Javascript functions that select information from the trace to persist and discard based on some +conditions. The following Javascript function returns only the sequence of opcodes executed by the transaction as a +comma-separated list. The function could be written directly in the Javascript console, but it is cleaner to +write it in a separate re-usable file and load it into the console. + +1. Create a file, `filterTrace_1.js`, with this content: + + ```javascript + + tracer = function(tx) { + return debug.traceTransaction(tx, {tracer: + '{' + + 'retVal: [],' + + 'step: function(log,db) {this.retVal.push(log.getPC() + ":" + log.op.toString())},' + + 'fault: function(log,db) {this.retVal.push("FAULT: " + JSON.stringify(log))},' + + 'result: function(ctx,db) {return this.retVal}' + + '}' + }) // return debug.traceTransaction ... + } // tracer = function ... + + ``` + +2. Run the [JavaScript console](https://geth.ethereum.org/docs/interface/javascript-console). + +3. Get the hash of a recent transaction from a node or block explorer. + +4. Run this command to run the script: + + ```javascript + loadScript("filterTrace_1.js") + ``` + +5. Run the tracer from the script. Be patient, it could take a long time. + + ```javascript + tracer("") + ``` + + The bottom of the output looks similar to: + ```sh + "3366:POP", "3367:JUMP", "1355:JUMPDEST", "1356:PUSH1", "1358:MLOAD", "1359:DUP1", "1360:DUP3", "1361:ISZERO", "1362:ISZERO", + "1363:ISZERO", "1364:ISZERO", "1365:DUP2", "1366:MSTORE", "1367:PUSH1", "1369:ADD", "1370:SWAP2", "1371:POP", "1372:POP", "1373:PUSH1", + "1375:MLOAD", "1376:DUP1", "1377:SWAP2", "1378:SUB", "1379:SWAP1", "1380:RETURN" + ``` + +6. Run this line to get a more readable output with each string in its own line. + + ```javascript + console.log(JSON.stringify(tracer(""), null, 2)) + ``` + +More information about the `JSON.stringify` function is available +[here](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/stringify). + +The commands above worked by calling the same `debug.traceTransaction` function that was previously +explained in [basic traces](https://geth.ethereum.org/docs/dapp/tracing), but with a new parameter, `tracer`. +This parameter takes the JavaScript object formated as a string. In the case of the trace above, it is: + +```javascript +{ + retVal: [], + step: function(log,db) {this.retVal.push(log.getPC() + ":" + log.op.toString())}, + fault: function(log,db) {this.retVal.push("FAULT: " + JSON.stringify(log))}, + result: function(ctx,db) {return this.retVal} +} +``` +This object has three member functions: + +- `step`, called for each opcode. +- `fault`, called if there is a problem in the execution. +- `result`, called to produce the results that are returned by `debug.traceTransaction` after the execution is done. + +In this case, `retVal` is used to store the list of strings to return in `result`. + +The `step` function adds to `retVal` the program counter and the name of the opcode there. Then, in `result`, this +list is returned to be sent to the caller. + + +### Filtering with conditions + +For actual filtered tracing we need an `if` statement to only log relevant information. For example, to isolate +the transaction's interaction with storage, the following tracer could be used: + +```javascript +tracer = function(tx) { + return debug.traceTransaction(tx, {tracer: + '{' + + 'retVal: [],' + + 'step: function(log,db) {' + + ' if(log.op.toNumber() == 0x54) ' + + ' this.retVal.push(log.getPC() + ": SLOAD");' + + ' if(log.op.toNumber() == 0x55) ' + + ' this.retVal.push(log.getPC() + ": SSTORE");' + + '},' + + 'fault: function(log,db) {this.retVal.push("FAULT: " + JSON.stringify(log))},' + + 'result: function(ctx,db) {return this.retVal}' + + '}' + }) // return debug.traceTransaction ... +} // tracer = function ... +``` + +The `step` function here looks at the opcode number of the op, and only pushes an entry if the opcode is +`SLOAD` or `SSTORE` ([here is a list of EVM opcodes and their numbers](https://github.com/wolflo/evm-opcodes)). +We could have used `log.op.toString()` instead, but it is faster to compare numbers rather than strings. + +The output looks similar to this: + +```javascript +[ + "5921: SLOAD", + . + . + . + "2413: SSTORE", + "2420: SLOAD", + "2475: SSTORE", + "6094: SSTORE" +] +``` + + +### Stack Information + +The trace above reports the program counter (PC) and whether the program read from storage or wrote to it. +That alone isn't particularly useful. To know more, the `log.stack.peek` function can be used to peek +into the stack. `log.stack.peek(0)` is the stack top, `log.stack.peek(1)` the entry below it, etc. + +The values returned by `log.stack.peek` are Go `big.Int` objects. By default they are converted to JavaScript +floating point numbers, so you need `toString(16)` to get them as hexadecimals, which is how 256-bit values such as +storage cells and their content are normally represented. + +#### Storage Information + +The function below provides a trace of all the storage operations and their parameters. This gives +a more complete picture of the program's interaction with storage. + +```javascript +tracer = function(tx) { + return debug.traceTransaction(tx, {tracer: + '{' + + 'retVal: [],' + + 'step: function(log,db) {' + + ' if(log.op.toNumber() == 0x54) ' + + ' this.retVal.push(log.getPC() + ": SLOAD " + ' + + ' log.stack.peek(0).toString(16));' + + ' if(log.op.toNumber() == 0x55) ' + + ' this.retVal.push(log.getPC() + ": SSTORE " +' + + ' log.stack.peek(0).toString(16) + " <- " +' + + ' log.stack.peek(1).toString(16));' + + '},' + + 'fault: function(log,db) {this.retVal.push("FAULT: " + JSON.stringify(log))},' + + 'result: function(ctx,db) {return this.retVal}' + + '}' + }) // return debug.traceTransaction ... +} // tracer = function ... + +``` + +The output is similar to: + +```javascript +[ + "5921: SLOAD 0", + . + . + . + "2413: SSTORE 3f0af0a7a3ed17f5ba6a93e0a2a05e766ed67bf82195d2dd15feead3749a575d <- fb8629ad13d9a12456", + "2420: SLOAD cc39b177dd3a7f50d4c09527584048378a692aed24d31d2eabeddb7f3c041870", + "2475: SSTORE cc39b177dd3a7f50d4c09527584048378a692aed24d31d2eabeddb7f3c041870 <- 358c3de691bd19", + "6094: SSTORE 0 <- 1" +] +``` + +#### Operation Results + +One piece of information missing from the function above is the result on an `SLOAD` operation. The +state we get inside `log` is the state prior to the execution of the opcode, so that value is not +known yet. For more operations we can figure it out for ourselves, but we don't have access to the +storage, so here we can't. + +The solution is to have a flag, `afterSload`, which is only true in the opcode right after an +`SLOAD`, when we can see the result at the top of the stack. + +```javascript +tracer = function(tx) { + return debug.traceTransaction(tx, {tracer: + '{' + + 'retVal: [],' + + 'afterSload: false,' + + 'step: function(log,db) {' + + ' if(this.afterSload) {' + + ' this.retVal.push(" Result: " + ' + + ' log.stack.peek(0).toString(16)); ' + + ' this.afterSload = false; ' + + ' } ' + + ' if(log.op.toNumber() == 0x54) {' + + ' this.retVal.push(log.getPC() + ": SLOAD " + ' + + ' log.stack.peek(0).toString(16));' + + ' this.afterSload = true; ' + + ' } ' + + ' if(log.op.toNumber() == 0x55) ' + + ' this.retVal.push(log.getPC() + ": SSTORE " +' + + ' log.stack.peek(0).toString(16) + " <- " +' + + ' log.stack.peek(1).toString(16));' + + '},' + + 'fault: function(log,db) {this.retVal.push("FAULT: " + JSON.stringify(log))},' + + 'result: function(ctx,db) {return this.retVal}' + + '}' + }) // return debug.traceTransaction ... +} // tracer = function ... +``` + +The output now contains the result in the line that follows the `SLOAD`. + +```javascript +[ + "5921: SLOAD 0", + " Result: 1", + . + . + . + "2413: SSTORE 3f0af0a7a3ed17f5ba6a93e0a2a05e766ed67bf82195d2dd15feead3749a575d <- fb8629ad13d9a12456", + "2420: SLOAD cc39b177dd3a7f50d4c09527584048378a692aed24d31d2eabeddb7f3c041870", + " Result: 0", + "2475: SSTORE cc39b177dd3a7f50d4c09527584048378a692aed24d31d2eabeddb7f3c041870 <- 358c3de691bd19", + "6094: SSTORE 0 <- 1" +] +``` + +### Dealing With Calls Between Contracts + +So the storage has been treated as if there are only 2256 cells. However, that is not true. +Contracts can call other contracts, and then the storage involved is the storage of the other contract. +We can see the address of the current contract in `log.contract.getAddress()`. This value is the execution +context - the contract whose storage we are using - even when code from another contract is executed (by using +[`CALLCODE` or `DELEGATECALL`][solidity-delcall]). + +However, `log.contract.getAddress()` returns an array of bytes. To convert this to the familiar hexadecimal +representation of Ethereum addresses, `this.byteHex()` and `array2Hex()` can be used. + +```javascript +tracer = function(tx) { + return debug.traceTransaction(tx, {tracer: + '{' + + 'retVal: [],' + + 'afterSload: false,' + + 'callStack: [],' + + + 'byte2Hex: function(byte) {' + + ' if (byte < 0x10) ' + + ' return "0" + byte.toString(16); ' + + ' return byte.toString(16); ' + + '},' + + + 'array2Hex: function(arr) {' + + ' var retVal = ""; ' + + ' for (var i=0; i 0 { + t.env.Cancel() + return + } + + name := op.String() + if _, ok := t.counts[name]; !ok { + t.counts[name] = 0 + } + t.counts[name]++ +} + +// CaptureEnter is called when EVM enters a new scope (via call, create or selfdestruct). +func (t *opcounter) CaptureEnter(op vm.OpCode, from common.Address, to common.Address, input []byte, gas uint64, value *big.Int) {} + +// CaptureExit is called when EVM exits a scope, even if the scope didn't +// execute any code. +func (t *opcounter) CaptureExit(output []byte, gasUsed uint64, err error) {} + +// CaptureFault implements the EVMLogger interface to trace an execution fault. +func (t *opcounter) CaptureFault(pc uint64, op vm.OpCode, gas, cost uint64, scope *vm.ScopeContext, depth int, err error) {} + +// CaptureEnd is called after the call finishes to finalize the tracing. +func (t *opcounter) CaptureEnd(output []byte, gasUsed uint64, _ time.Duration, err error) {} + +func (*opcounter) CaptureTxStart(gasLimit uint64) {} + +func (*opcounter) CaptureTxEnd(restGas uint64) {} + +// GetResult returns the json-encoded nested list of call traces, and any +// error arising from the encoding or forceful termination (via `Stop`). +func (t *opcounter) GetResult() (json.RawMessage, error) { + res, err := json.Marshal(t.counts) + if err != nil { + return nil, err + } + return res, t.reason +} + +// Stop terminates execution of the tracer at the first opportune moment. +func (t *opcounter) Stop(err error) { + t.reason = err + atomic.StoreUint32(&t.interrupt, 1) +} +``` + +As can be seen every method of the [EVMLogger interface](https://pkg.go.dev/github.com/ethereum/go-ethereum/core/vm#EVMLogger) needs to be implemented (even if empty). Key parts to notice are the `init()` function which registers the tracer in Geth, the `CaptureState` hook where the opcode counts are incremented and `GetResult` where the result is serialized and delivered. To test this out the source is first compiled with `make geth`. Then in the console it can be invoked through the usual API methods by passing in the name it was registered under: + +```console +> debug.traceTransaction('0x7ae446a7897c056023a8104d254237a8d97783a92900a7b0f7db668a9432f384', { tracer: 'opcounter' }) +{ + ADD: 4, + AND: 3, + CALLDATALOAD: 2, + ... +} +``` + +[solidity-delcall]:https://docs.soliditylang.org/en/v0.8.14/introduction-to-smart-contracts.html#delegatecall-callcode-and-libraries +[debug-docs]: /docs/rpc/ns-debug diff --git a/content/docs/developers/dapp-developer/mobile/mobile-accounts.md b/content/docs/developers/dapp-developer/mobile/mobile-accounts.md new file mode 100644 index 0000000000..9d4bd39219 --- /dev/null +++ b/content/docs/developers/dapp-developer/mobile/mobile-accounts.md @@ -0,0 +1,323 @@ +--- +title: Mobile Account Management +sort_key: G +--- + +To provide Ethereum integration for your mobile applications, the very first thing you +should be interested in doing is account management. + +Although all current leading Ethereum implementations provide account management built in, +it is ill advised to keep accounts in any location that is shared between multiple +applications and/or multiple people. The same way you do not entrust your ISP (who is +after all your gateway into the internet) with your login credentials; you should not +entrust an Ethereum node (who is your gateway into the Ethereum network) with your +credentials either. + +The proper way to handle user accounts in your mobile applications is to do client side +account management, everything self-contained within your own application. This way you +can ensure as fine grained (or as coarse) access permissions to the sensitive data as +deemed necessary, without relying on any third party application's functionality and/or +vulnerabilities. + +To support this, `go-ethereum` provides a simple, yet thorough accounts library that gives +you all the tools to do properly secured account management via encrypted keystores and +passphrase protected accounts. You can leverage all the security of the `go-ethereum` +crypto implementation while at the same time running everything in your own application. + +## Encrypted keystores + +Although handling your users' accounts locally on their own mobile device does provide +certain security guarantees, access keys to Ethereum accounts should never lay around in +clear-text form. As such, we provide an encrypted keystore that provides the proper +security guarantees for you without requiring a thorough understanding from your part of +the associated cryptographic primitives. + +The important thing to know when using the encrypted keystore is that the cryptographic +primitives used within can operate either in *standard* or *light* mode. The former +provides a higher level of security at the cost of increased computational burden and +resource consumption: + + * *standard* needs 256MB memory and 1 second processing on a modern CPU to access a key + * *light* needs 4MB memory and 100 millisecond processing on a modern CPU to access a key + +As such, *light* is more suitable for mobile applications, but you should be aware of the +trade-offs nonetheless. + +*For those interested in the cryptographic and/or implementation details, the key-store +uses the `secp256k1` elliptic curve as defined in the [Standards for Efficient +Cryptography](sec2), implemented by the [`libsecp256k`][secp256k1] library and wrapped by +[`github.com/ethereum/go-ethereum/accounts`][accounts-go]. Accounts are stored on disk in +the [Web3 Secret Storage][secstore] format.* + +### Keystores on Android (Java) + +The encrypted keystore on Android is implemented by the `KeyStore` class from the +`org.ethereum.geth` package. The configuration constants (for the *standard* or *light* +security modes described above) are located in the `Geth` abstract class, similarly from +the `org.ethereum.geth` package. Hence to do client side account management on Android, +you'll need to import two classes into your Java code: + +```java +import org.ethereum.geth.Geth; +import org.ethereum.geth.KeyStore; +``` + +Afterwards you can create a new encrypted keystore via: + +```java +KeyStore ks = new KeyStore("/path/to/keystore", Geth.LightScryptN, Geth.LightScryptP); +``` + +The path to the keystore folder needs to be a location that is writable by the local +mobile application but non-readable for other installed applications (for security reasons +obviously), so we'd recommend placing it inside your app's data directory. If you are +creating the `KeyStore` from within a class extending an Android object, you will most +probably have access to the `Context.getFilesDir()` method via `this.getFilesDir()`, so +you could set the keystore path to `this.getFilesDir() + "/keystore"`. + +The last two arguments of the `KeyStore` constructor are the crypto parameters defining +how resource-intensive the keystore encryption should be. You can choose between +`Geth.StandardScryptN, Geth.StandardScryptP`, `Geth.LightScryptN, Geth.LightScryptP` or +specify your own numbers (please make sure you understand the underlying cryptography for +this). We recommend using the *light* version. + +### Keystores on iOS (Swift 3) + +The encrypted keystore on iOS is implemented by the `GethKeyStore` class from the `Geth` +framework. The configuration constants (for the *standard* or *light* security modes +described above) are located in the same namespace as global variables. Hence to do client +side account management on iOS, you'll need to import the framework into your Swift code: + +```swift +import Geth +``` + +Afterwards you can create a new encrypted account manager via: + +```swift +let ks = GethNewKeyStore("/path/to/keystore", GethLightScryptN, GethLightScryptP); +``` + +The path to the keystore folder needs to be a location that is writable by the local +mobile application but non-readable for other installed applications (for security reasons +obviously), so we'd recommend placing it inside your app's document directory. You should +be able to retrieve the document directory via `let datadir = +NSSearchPathForDirectoriesInDomains(.documentDirectory, .userDomainMask, true)[0]`, so you +could set the keystore path to `datadir + "/keystore"`. + +The last two arguments of the `GethNewKeyStore` factory method are the crypto parameters +defining how resource-intensive the keystore encryption should be. You can choose between +`GethStandardScryptN, GethStandardScryptP`, `GethLightScryptN, GethLightScryptP` or +specify your own numbers (please make sure you understand the underlying cryptography for +this). We recommend using the *light* version. + +## Account lifecycle + +Having created an encrypted keystore for your Ethereum accounts, you can use this for the +entire account lifecycle requirements of your mobile application. This includes the basic +functionality of creating new accounts and deleting existing ones; as well as the more +advanced functionality of updating access credentials, exporting existing accounts, and +importing them on another device. + +Although the keystore defines the encryption strength it uses to store your accounts, +there is no global master password that can grant access to all of them. Rather each +account is maintained individually, and stored on disk in its [encrypted format][secstore] +individually, ensuring a much cleaner and stricter separation of credentials. + +This individuality however means that any operation requiring access to an account will +need to provide the necessary authentication credentials for that particular account in +the form of a passphrase: + + * When creating a new account, the caller must supply a passphrase to encrypt the account + with. This passphrase will be required for any subsequent access, the lack of which + will forever forfeit using the newly created account. + * When deleting an existing account, the caller must supply a passphrase to verify + ownership of the account. This isn't cryptographically necessary, rather a protective + measure against accidental loss of accounts. + * When updating an existing account, the caller must supply both current and new + passphrases. After completing the operation, the account will not be accessible via the + old passphrase any more. + * When exporting an existing account, the caller must supply both the current passphrase + to decrypt the account, as well as an export passphrase to re-encrypt it with before + returning the key-file to the user. This is required to allow moving accounts between + devices without sharing original credentials. + * When importing a new account, the caller must supply both the encryption passphrase of + the key-file being imported, as well as a new passhprase with which to store the + account. This is required to allow storing account with different credentials than used + for moving them around. + +*Please note, there is no recovery mechanisms for losing the passphrases. The +cryptographic properties of the encrypted keystore (if using the provided parameters) +guarantee that account credentials cannot be brute forced in any meaningful time.* + +### Accounts on Android (Java) + +An Ethereum account on Android is implemented by the `Account` class from the +`org.ethereum.geth` package. Assuming we already have an instance of a `KeyStore` called +`ks` from the previous section, we can easily execute all of the described lifecycle +operations with a handful of function calls. + +```java +// Create a new account with the specified encryption passphrase. +Account newAcc = ksm.newAccount("Creation password"); + +// Export the newly created account with a different passphrase. The returned +// data from this method invocation is a JSON encoded, encrypted key-file. +byte[] jsonAcc = ks.exportKey(newAcc, "Creation password", "Export password"); + +// Update the passphrase on the account created above inside the local keystore. +ks.updateAccount(newAcc, "Creation password", "Update password"); + +// Delete the account updated above from the local keystore. +ks.deleteAccount(newAcc, "Update password"); + +// Import back the account we've exported (and then deleted) above with yet +// again a fresh passphrase. +Account impAcc = ks.importKey(jsonAcc, "Export password", "Import password"); +``` + +*Although instances of `Account` can be used to access various information about specific +Ethereum accounts, they do not contain any sensitive data (such as passphrases or private +keys), rather act solely as identifiers for client code and the keystore.* + +### Accounts on iOS (Swift 3) + +An Ethereum account on iOS is implemented by the `GethAccount` class from the `Geth` +framework. Assuming we already have an instance of a `GethKeyStore` called `ks` from the +previous section, we can easily execute all of the described lifecycle operations with a +handful of function calls. + +```swift +// Create a new account with the specified encryption passphrase. +let newAcc = try! ks?.newAccount("Creation password") + +// Export the newly created account with a different passphrase. The returned +// data from this method invocation is a JSON encoded, encrypted key-file. +let jsonKey = try! ks?.exportKey(newAcc!, passphrase: "Creation password", newPassphrase: "Export password") + +// Update the passphrase on the account created above inside the local keystore. +try! ks?.update(newAcc, passphrase: "Creation password", newPassphrase: "Update password") + +// Delete the account updated above from the local keystore. +try! ks?.delete(newAcc, passphrase: "Update password") + +// Import back the account we've exported (and then deleted) above with yet +// again a fresh passphrase. +let impAcc = try! ks?.importKey(jsonKey, passphrase: "Export password", newPassphrase: "Import password") +``` + +*Although instances of `GethAccount` can be used to access various information about +specific Ethereum accounts, they do not contain any sensitive data (such as passphrases or +private keys), rather act solely as identifiers for client code and the keystore.* + +## Signing authorization + +As mentioned above, account objects do not hold the sensitive private keys of the +associated Ethereum accounts, but are merely placeholders to identify the cryptographic +keys with. All operations that require authorization (e.g. transaction signing) are +performed by the account manager after granting it access to the private keys. + +There are a few different ways one can authorize the account manager to execute signing +operations, each having its advantages and drawbacks. Since the different methods have +wildly different security guarantees, it is essential to be clear on how each works: + + * **Single authorization**: The simplest way to sign a transaction via the keystore is to + provide the passphrase of the account every time something needs to be signed, which + will ephemerally decrypt the private key, execute the signing operation and immediately + throw away the decrypted key. The drawbacks are that the passphrase needs to be queried + from the user every time, which can become annoying if done frequently; or the + application needs to keep the passphrase in memory, which can have security + consequences if not done properly; and depending on the keystore's configured strength, + constantly decrypting keys can result in non-negligible resource requirements. + * **Multiple authorizations**: A more complex way of signing transactions via the + keystore is to unlock the account via its passphrase once, and allow the account + manager to cache the decrypted private key, enabling all subsequent signing requests to + complete without the passphrase. The lifetime of the cached private key may be managed + manually (by explicitly locking the account back up) or automatically (by providing a + timeout during unlock). This mechanism is useful for scenarios where the user may need + to sign many transactions or the application would need to do so without requiring user + input. The crucial aspect to remember is that **anyone with access to the account + manager can sign transactions while a particular account is unlocked** (e.g. device + left unattended; application running untrusted code). + +*Note, creating transactions is out of scope here, so the remainder of this section will +assume we already have a transaction to sign, and will focus only on creating an +authorized version of it. Creating an actually meaningful transaction will be covered +later.* + +### Signing on Android (Java) + +Assuming we already have an instance of a `KeyStore` called `ks` from the previous +sections, we can create a new account to sign transactions with via it's already +demonstrated `newAccount` method; and to avoid going into transaction creation for now, we +can hard-code a random transaction to sign instead. + +```java +// Create a new account to sign transactions with +Account signer = ks.newAccount("Signer password"); +Transaction tx = new Transaction( + 1, new Address("0x0000000000000000000000000000000000000000"), + new BigInt(0), new BigInt(0), new BigInt(1), null); // Random empty transaction +BigInt chain = new BigInt(1); // Chain identifier of the main net +``` + +With the boilerplate out of the way, we can now sign transaction using the authorization +mechanisms described above: + +```java +// Sign a transaction with a single authorization +Transaction signed = ks.signTxPassphrase(signer, "Signer password", tx, chain); + +// Sign a transaction with multiple manually cancelled authorizations +ks.unlock(signer, "Signer password"); +signed = ks.signTx(signer, tx, chain); +ks.lock(signer.getAddress()); + +// Sign a transaction with multiple automatically cancelled authorizations +ks.timedUnlock(signer, "Signer password", 1000000000); +signed = ks.signTx(signer, tx, chain); +``` + +### Signing on iOS (Swift 3) + +Assuming we already have an instance of a `GethKeyStore` called `ks` from the previous +sections, we can create a new account to sign transactions with via it's already +demonstrated `newAccount` method; and to avoid going into transaction creation for now, we +can hard-code a random transaction to sign instead. + +```swift +// Create a new account to sign transactions with +var error: NSError? +let signer = try! ks?.newAccount("Signer password") + +let to = GethNewAddressFromHex("0x0000000000000000000000000000000000000000", &error) +let tx = GethNewTransaction(1, to, GethNewBigInt(0), GethNewBigInt(0), GethNewBigInt(0), nil) // Random empty transaction +let chain = GethNewBigInt(1) // Chain identifier of the main net +``` + +*Note, although Swift usually rewrites `NSError` returns to throws, this particular +instance seems to have been missed for some reason (possibly due to it being a +constructor). It will be fixed in a later version of the iOS bindings when the appropriate +fixed are implemented upstream in the `gomobile` project.* + +With the boilerplate out of the way, we can now sign transaction using the authorization +methods described above: + +```swift +// Sign a transaction with a single authorization +var signed = try! ks?.signTxPassphrase(signer, passphrase: "Signer password", tx: tx, chainID: chain) + +// Sign a transaction with multiple manually cancelled authorizations +try! ks?.unlock(signer, passphrase: "Signer password") +signed = try! ks?.signTx(signer, tx: tx, chainID: chain) +try! ks?.lock(signer?.getAddress()) + +// Sign a transaction with multiple automatically cancelled authorizations +try! ks?.timedUnlock(signer, passphrase: "Signer password", timeout: 1000000000) +signed = try! ks?.signTx(signer, tx: tx, chainID: chain) +``` + +[sec2]: https://www.secg.org/sec2-v2.pdf +[accounts-go]: https://godoc.org/github.com/ethereum/go-ethereum/accounts +[secp256k1]: https://github.com/bitcoin-core/secp256k1 +[secstore]: https://github.com/ethereum/wiki/wiki/Web3-Secret-Storage-Definition diff --git a/content/docs/developers/dapp-developer/mobile/mobile.md b/content/docs/developers/dapp-developer/mobile/mobile.md new file mode 100644 index 0000000000..efcc107451 --- /dev/null +++ b/content/docs/developers/dapp-developer/mobile/mobile.md @@ -0,0 +1,180 @@ +--- +title: Mobile API +sort_key: F +--- + +The Ethereum blockchain along with its two extension protocols Whisper and Swarm was +originally conceptualized to become the supporting pillar of web3, providing the +consensus, messaging and storage backbone for a new generation of distributed (actually, +decentralized) applications called DApps. + +The first incarnation towards this dream of web3 was a command line client providing an +RPC interface into the peer-to-peer protocols. The client was soon enough extended with a +web-browser-like graphical user interface, permitting developers to write DApps based on +the tried and proven HTML/CSS/JS technologies. + +As many DApps have more complex requirements than what a browser environment can handle, +it became apparent that providing programmatic access to the web3 pillars would open the +door towards a new class of applications. As such, the second incarnation of the web +dream is to open up all our technologies for other projects as reusable components. + +Starting with the 1.5 release family of `go-ethereum`, we transitioned away from providing +only a full blown Ethereum client and started shipping official Go packages that could be +embedded into third party desktop and server applications. It took only a small leap from +here to begin porting our code to mobile platforms. + +## Quick overview + +Similarly to our reusable Go libraries, the mobile wrappers also focus on four main usage +areas: + +- Simplified client side account management +- Remote node interfacing via different transports +- Contract interactions through auto-generated bindings +- In-process Ethereum, Whisper and Swarm peer-to-peer node + +You can watch a quick overview about these in Peter's (@karalabe) talk titled "Import +Geth: Ethereum from Go and beyond", presented at the Ethereum Devcon2 developer conference +in September, 2016 (Shanghai). Slides are [available +here](https://ethereum.karalabe.com/talks/2016-devcon.html). + +[![Peter's Devcon2 talk](https://img.youtube.com/vi/R0Ia1U9Gxjg/0.jpg)](https://www.youtube.com/watch?v=R0Ia1U9Gxjg) + +## Library bundles + +The `go-ethereum` mobile library is distributed either as an Android `.aar` archive +(containing binaries for `arm-7`, `arm64`, `x86` and `x64`); or as an iOS XCode framework +(containing binaries for `arm-7`, `arm64` and `x86`). We do not provide library bundles +for Windows phone the moment. + +### Android archive + +The simplest way to use `go-ethereum` in your Android project is through a Maven +dependency. We provide bundles of all our stable releases (starting from v1.5.0) through +Maven Central, and also provide the latest develop bundle through the Sonatype OSS +repository. + +#### Stable dependency (Maven Central) + +To add an Android dependency to the **stable** library release of `go-ethereum`, you'll +need to ensure that the Maven Central repository is enabled in your Android project, and +that the `go-ethereum` code is listed as a required dependency of your application. You +can do both of these by editing the `build.gradle` script in your Android app's folder: + +```gradle +repositories { + mavenCentral() +} + +dependencies { + // All your previous dependencies + compile 'org.ethereum:geth:1.5.2' // Change the version to the latest release +} +``` + +#### Develop dependency (Sonatype) + +To add an Android dependency to the current version of `go-ethereum`, you'll need to +ensure that the Sonatype snapshot repository is enabled in your Android project, and that +the `go-ethereum` code is listed as a required `SNAPSHOT` dependency of your application. +You can do both of these by editing the `build.gradle` script in your Android app's +folder: + +```gradle +repositories { + maven { + url "https://oss.sonatype.org/content/groups/public" + } +} + +dependencies { + // All your previous dependencies + compile 'org.ethereum:geth:1.5.3-SNAPSHOT' // Change the version to the latest release +} +``` + +#### Custom dependency + +If you prefer not to depend on Maven Central or Sonatype; or would like to access an older +develop build not available any more as an online dependency, you can download any bundle +directly from [our website](https://geth.ethereum.org/downloads/) and insert it into your +project in Android Studio via `File -> New -> New module... -> Import .JAR/.AAR Package`. + +You will also need to configure `gradle` to link the mobile library bundle to your +application. This can be done by adding a new entry to the `dependencies` section of your +`build.gradle` script, pointing it to the module you just added (named `geth` by default). + +```gradle +dependencies { + // All your previous dependencies + compile project(':geth') +} +``` + +#### Manual builds + +Lastly, if you would like to make modifications to the `go-ethereum` mobile code and/or +build it yourself locally instead of downloading a pre-built bundle, you can do so using a +`make` command. This will create an Android archive called `geth.aar` in the `build/bin` +folder that you can import into your Android Studio as described above. + +```bash +$ make android +[...] +Done building. +Import "build/bin/geth.aar" to use the library. +``` + +### iOS framework + +The simplest way to use `go-ethereum` in your iOS project is through a +[CocoaPods](https://cocoapods.org/) dependency. We provide bundles of all our stable +releases (starting from v1.5.3) and also latest develop versions. + +#### Automatic dependency + +To add an iOS dependency to the current stable or latest develop version of `go-ethereum`, +you'll need to ensure that your iOS XCode project is configured to use CocoaPods. +Detailing that is out of scope in this document, but you can find a guide in the upstream +[Using CocoaPods](https://guides.cocoapods.org/using/using-cocoapods.html) page. +Afterwards you can edit your `Podfile` to list `go-ethereum` as a dependency: + +```ruby +target 'MyApp' do + # All your previous dependencies + pod 'Geth', '1.5.4' # Change the version to the latest release +end +``` + +Alternatively, if you'd like to use the latest develop version, replace the package +version `1.5.4` with `~> 1.5.5-unstable` to switch to pre-releases and to always pull in +the latest bundle from a particular release family. + +#### Custom dependency + +If you prefer not to depend on CocoaPods; or would like to access an older develop build +not available any more as an online dependency, you can download any bundle directly from +[our website](https://geth.ethereum.org/downloads/) and insert it into your project in +XCode via `Project Settings -> Build Phases -> Link Binary With Libraries`. + +Do not forget to extract the framework from the compressed `.tar.gz` archive. You can do +that either using a GUI tool or from the command line via (replace the archive with your +downloaded file): + +``` +tar -zxvf geth-ios-all-1.5.3-unstable-e05d35e6.tar.gz +``` + +#### Manual builds + +Lastly, if you would like to make modifications to the `go-ethereum` mobile code and/or +build it yourself locally instead of downloading a pre-built bundle, you can do so using a +`make` command. This will create an iOS XCode framework called `Geth.framework` in the +`build/bin` folder that you can import into XCode as described above. + +```bash +$ make ios +[...] +Done building. +Import "build/bin/Geth.framework" to use the library. +``` diff --git a/content/docs/developers/dapp-developer/native-accounts.md b/content/docs/developers/dapp-developer/native-accounts.md new file mode 100644 index 0000000000..867a438018 --- /dev/null +++ b/content/docs/developers/dapp-developer/native-accounts.md @@ -0,0 +1,223 @@ +--- +title: Go Account Management +sort_key: D +--- + +Geth provides a simple, yet thorough accounts package that includes all the tools developers +need to leverage all the security of Geth's crypto implementation in a Go native application. +The account management is done client side with all sensitive data held inside the application. +This gives the user control over access permissions without relying on any third party. + +**Note Geth's built-in account management is convenient and straightforward to use, but +best practise is to use the external tool *Clef* for key management.** + +{:toc} + +- this will be removed by the toc + +## Encrypted keystores + +Access keys to Ethereum accounts should never be stored in plain-text. Instead, they should be +stored encrypted so that even if the mobile device is accessed by a malicious third party the +keys are still hidden under an additional layer of security. Geth provides a keystore that enables +developers to store keys securely. The Geth keystore uses [Scrypt][scrypt-docs] to store keys that are encoded +using the [`secp256k1`][secp256k1] elliptic curve. Accounts are stored on disk in the +[Web3 Secret Storage][wss] format. Developers should be aware of these implementation details +but are not required to deeply understand the cryptographic primitives in order to use the keystore. + +One thing that should be understood, though, is that the cryptographic primitives underpinning the +keystore can operate in light or standard mode. Light mode is computationally cheaper, while standard +mode has extra security. Light mode is appropriate for mobile devices, but developers should be +aware that there is a security trade-off. + +* standard needs 256MB memory and 1 second processing on a modern CPU to access a key +* light needs 4MB memory and 100 millisecond processing on a modern CPU to access a key + + +The encrypted keystore is implemented by the [`accounts.Manager`][accounts-manager] struct +from the [`accounts`][accounts-pkg] package, which also contains the configuration constants for the +*standard* or *light* security modes described above. Hence client side account management +simply requires importing the `accounts` package into the application code. + +```go +import "github.com/ethereum/go-ethereum/accounts" +import "github.com/ethereum/go-ethereum/accounts/keystore" +import "github.com/ethereum/go-ethereum/common" +``` +Afterwards a new encrypted account manager can be created via: + +```go +ks := keystore.NewKeyStore("/path/to/keystore", keystore.StandardScryptN, keystore.StandardScryptP) +am := accounts.NewManager(&accounts.Config{InsecureUnlockAllowed: false}, ks) +``` + +The path to the keystore folder needs to be a location that is writable by the local user +but non-readable for other system users, such as inside the user's home directory. + +The last two arguments of [`keystore.NewKeyStore`][keystore] are the crypto parameters defining +how resource-intensive the keystore encryption should be. The options are +[`accounts.StandardScryptN, accounts.StandardScryptP`, `accounts.LightScryptN, +accounts.LightScryptP`][pkg-constants] or custom values (requiring understanding of the underlying +cryptography). The *standard* version is recommended. + + +## Account lifecycle + +Once an encrypted keystore for Ethereum accounts exists it, it can be used to manage accounts for the +entire account lifecycle requirements of a Go native application. This includes the basic functionality +of creating new accounts and deleting existing ones as well as updating access credentials, +exporting existing accounts, and importing them on other devices. + +Although the keystore defines the encryption strength it uses to store accounts, there is no global master +password that can grant access to all of them. Rather each account is maintained individually, and stored on +disk in its [encrypted format][wss] individually, ensuring a much cleaner and stricter separation of +credentials. + +This individuality means that any operation requiring access to an account will need to provide the +necessary authentication credentials for that particular account in the form of a passphrase: + + * When creating a new account, the caller must supply a passphrase to encrypt the account + with. This passphrase will be required for any subsequent access, the lack of which + will forever forfeit using the newly created account. + + * When deleting an existing account, the caller must supply a passphrase to verify + ownership of the account. This isn't cryptographically necessary, rather a protective + measure against accidental loss of accounts. + + * When updating an existing account, the caller must supply both current and new + passphrases. After completing the operation, the account will not be accessible via the + old passphrase any more. + + * When exporting an existing account, the caller must supply both the current passphrase + to decrypt the account, as well as an export passphrase to re-encrypt it with before + returning the key-file to the user. This is required to allow moving accounts between + machines and applications without sharing original credentials. + + * When importing a new account, the caller must supply both the encryption passphrase of + the key-file being imported, as well as a new passhprase with which to store the + account. This is required to allow storing account with different credentials than used + for moving them around. + +***Please note, there are no recovery mechanisms for lost passphrases. The +cryptographic properties of the encrypted keystore (using the provided parameters) +guarantee that account credentials cannot be brute forced in any meaningful time.*** + +An Ethereum account is implemented by the [`accounts.Account`][accounts-account] struct from +the Geth [accounts][accounts-pkg] package. Assuming an instance of an +[`accounts.Manager`][accounts-manager] called `am` exists, all of the described lifecycle +operations can be executed with a handful of function calls (error handling omitted). + +```go +// Create a new account with the specified encryption passphrase. +newAcc, _ := ks.NewAccount("Creation password") +fmt.Println(newAcc) + +// Export the newly created account with a different passphrase. The returned +// data from this method invocation is a JSON encoded, encrypted key-file. +jsonAcc, _ := ks.Export(newAcc, "Creation password", "Export password") + +// Update the passphrase on the account created above inside the local keystore. +_ = ks.Update(newAcc, "Creation password", "Update password") + +// Delete the account updated above from the local keystore. +_ = ks.Delete(newAcc, "Update password") + +// Import back the account we've exported (and then deleted) above with yet +// again a fresh passphrase. +impAcc, _ := ks.Import(jsonAcc, "Export password", "Import password") +``` + +*Although instances of [`accounts.Account`][accounts-account] can be used to access various information about +specific Ethereum accounts, they do not contain any sensitive data (such as passphrases or private keys), +rather they act solely as identifiers for client code and the keystore.* + +## Signing authorization + +Account objects do not hold the sensitive private keys of the associated Ethereum accounts. +Account objects are placeholders that identify the cryptographic keys. All operations that +require authorization (e.g. transaction signing) are performed by the account manager after +granting it access to the private keys. + +There are a few different ways to authorize the account manager to execute signing +operations, each having its advantages and drawbacks. Since the different methods have +wildly different security guarantees, it is essential to be clear on how each works: + +* **Single authorization**: The simplest way to sign a transaction via the account + manager is to provide the passphrase of the account every time something needs to be + signed, which will ephemerally decrypt the private key, execute the signing operation + and immediately throw away the decrypted key. The drawbacks are that the passphrase + needs to be queried from the user every time, which can become annoying if done + frequently or the application needs to keep the passphrase in memory, which can have + security consequences if not done properly. Depending on the keystore's configured + strength, constantly decrypting keys can result in non-negligible resource + requirements. + +* **Multiple authorizations**: A more complex way of signing transactions via the account + manager is to unlock the account via its passphrase once, and allow the account manager + to cache the decrypted private key, enabling all subsequent signing requests to + complete without the passphrase. The lifetime of the cached private key may be managed + manually (by explicitly locking the account back up) or automatically (by providing a + timeout during unlock). This mechanism is useful for scenarios where the user may need + to sign many transactions or the application would need to do so without requiring user + input. The crucial aspect to remember is that **anyone with access to the account + manager can sign transactions while a particular account is unlocked** (e.g. + application running untrusted code). + + +Assuming an instance of an [`accounts.Manager`][accounts-manager] called `am` exists, a new +account can be created to sign transactions using [`NewAccount`][new-account]. Creating transactions +is out of scope for this page so instead a random [`common.Hash`][common-hash] will be signed instead. +For information on creating transactions in Go native applications see the [Go API page](/docs/dapp/native). + +```go +// Create a new account to sign transactions with +signer, _ := ks.NewAccount("Signer password") +txHash := common.HexToHash("0x0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef") +``` + +With the boilerplate out of the way, the transaction can be signed using the authorization +mechanisms described above: + +```go +// Sign a transaction with a single authorization +signature, _ := ks.SignHashWithPassphrase(signer, "Signer password", txHash.Bytes()) + +// Sign a transaction with multiple manually cancelled authorizations +_ = ks.Unlock(signer, "Signer password") +signature, _ = ks.SignHash(signer, txHash.Bytes()) +_ = ks.Lock(signer.Address) + +// Sign a transaction with multiple automatically cancelled authorizations +_ = ks.TimedUnlock(signer, "Signer password", time.Second) +signature, _ = ks.SignHash(signer, txHash.Bytes()) +``` + +Note that [`SignWithPassphrase`][sign-w-phrase] takes an [`accounts.Account`][accounts-account] as the +signer, whereas [`Sign`][accounts-sign] takes only a [`common.Address`][common-address]. The reason +for this is that an [`accounts.Account`][accounts-account] object may also contain a custom key-path, allowing +[`SignWithPassphrase`][sign-w-phrase] to sign using accounts outside of the keystore; however +[`Sign`][accounts-sign] relies on accounts already unlocked within the keystore, so it cannot specify custom paths. + + +## Summary + +Account management is a fundamental pillar of Ethereum development. Geth's Go API provides the tools required +to integrate best-practise account security into Go native applications using a simple set of Go functions. + + +[accounts-sign]: (https://godoc.org/github.com/ethereum/go-ethereum/accounts#Manager.Sign) +[common-address]: https://godoc.org/github.com/ethereum/go-ethereum/common#Address +[accounts-sign]: https://godoc.org/github.com/ethereum/go-ethereum/accounts#Manager.Sign +[sign-w-phrase]: https://godoc.org/github.com/ethereum/go-ethereum/accounts#Manager.SignWithPassphrase +[secp256k1]: https://www.secg.org/sec2-v2.pdf +[libsecp256k1]: https://github.com/bitcoin-core/secp256k1 +[wss]:https://github.com/ethereum/wiki/wiki/Web3-Secret-Storage-Definition +[go-accounts]:https://godoc.org/github.com/ethereum/go-ethereum/accounts +[accounts-manager]: https://godoc.org/github.com/ethereum/go-ethereum/accounts#Manager +[accounts-pkg]: https://godoc.org/github.com/ethereum/go-ethereum/accounts +[keystore]: https://godoc.org/github.com/ethereum/go-ethereum/accounts/keystore#NewKeyStore +[pkg-constants]: https://godoc.org/github.com/ethereum/go-ethereum/accounts#pkg-constants +[accounts-account]:https://godoc.org/github.com/ethereum/go-ethereum/accounts#Account +[new-account]: https://godoc.org/github.com/ethereum/go-ethereum/accounts#Manager.NewAccount +[common-hash]: https://godoc.org/github.com/ethereum/go-ethereum/common#Hash +[scrypt-docs]: https://pkg.go.dev/golang.org/x/crypto/scrypt diff --git a/content/docs/developers/dapp-developer/native-bindings.md b/content/docs/developers/dapp-developer/native-bindings.md new file mode 100644 index 0000000000..248bd12e01 --- /dev/null +++ b/content/docs/developers/dapp-developer/native-bindings.md @@ -0,0 +1,617 @@ +--- +title: Go Contract Bindings +sort_key: E +--- + +This page introduces the concept of server-side native dapps. Geth provides the tools required +to generate [Go][go-link] language bindings to any Ethereum contract that is compile-time type safe, +highly performant and can be generated completely automatically from a compiled contract. + +Interacting with a contract on the Ethereum blockchain from Go is already possible via the +RPC interfaces exposed by Ethereum clients. However, writing the boilerplate code that +translates Go language constructs into RPC calls and back is time consuming and brittle - +implementation bugs can only be detected during runtime and it's almost impossible to evolve +a contract as even a tiny change in Solidity is awkward to port over to Go. Therefore, +Geth provides tools for easily converting contract code into Go code that can be used directly +in Go applications. + +This page provides an introduction to generating Go contract bindings and using them in a simple +Go application. + +{:toc} + +- this will be removed by the toc + +## Prerequisites + +This page is fairly beginner-friendly and designed for people starting out with +writing Go native dapps. The core concepts will be introduced gradually as a developer +would encounter them. However, some basic familiarity with [Ethereum](https://ethereum.org), +[Solidity](https://docs.soliditylang.org/en/v0.8.15/) and [Go](https://go.dev/) is +assumed. + + +## What is an ABI? + +Ethereum smart contracts have a schema that defines its functions and return types in the form +of a JSON file. This JSON file is known as an *Application Binary Interface*, or ABI. The ABI +acts as a specification for precisely how to encode data sent to a contract and how to +decode the data the contract sends back. The ABI is the only essential piece of information required to +generate Go bindings. Go developers can then use the bindings to interact with the contract +from their Go application without having to deal directly with data encoding and decoding. +An ABI is generated when a contract is compiled. + +## Abigen: Go binding generator + +Geth includes a source code generator called `abigen` that can convert Ethereum ABI definitions +into easy to use, type-safe Go packages. With a valid Go development environment +set up and the go-ethereum repository checked out correctly, `abigen` can be built as follows: + +``` +$ cd $GOPATH/src/github.com/ethereum/go-ethereum +$ go build ./cmd/abigen +``` + +### Generating the bindings + +To demonstrate the binding generator a contract is required. The contract `Storage.sol` implements two +very simple functions: `store` updates a user-defined `uint256` to the contract's storage, and `retrieve` +displays the value stored in the contract to the user. The Solidity code is as follows: + +```solidity +// SPDX-License-Identifier: GPL-3.0 + +pragma solidity >0.7.0 < 0.9.0; +/** +* @title Storage +* @dev store or retrieve variable value +*/ + +contract Storage { + + uint256 value; + + function store(uint256 number) public{ + value = number; + } + + function retrieve() public view returns (uint256){ + return value; + } +} +``` + +This contract can be pasted into a text file and saved as `Storage.sol`. + +The following code snippet shows how an ABI can be generated for `Storage.sol` +using the Solidity compiler `solc`. + +```shell +solc --abi Storage.sol -o build +``` + +The ABI can also be generated in other ways such as using the `compile` commands in development +frameworks such as [Truffle][truffle-link], [Hardhat][hardhat-link] and [Brownie][brownie-link] +or in the online IDE [Remix][remix-link]. ABIs for existing +verified contracts can be downloaded from [Etherscan](etherscan.io). + + +The ABI for `Storage.sol` (`Storage.abi`) looks as follows: + +```json +[{"inputs":[],"name":"retrieve","outputs":[{"internalType":"uint256","name":"","type":"uint256"}],"stateMutability":"view","type":"function"},{"inputs":[{"internalType":"uint256","name":"number","type":"uint256"}],"name":"store","outputs":[],"stateMutability":"nonpayable","type":"function"}] +``` + +The contract binding can then be generated by passing the ABI to `abigen` as follows: + +``` +$ abigen --abi Storage.abi --pkg main --type Storage --out Storage.go +``` + +Where the flags are: + + * `--abi`: Mandatory path to the contract ABI to bind to + * `--pkg`: Mandatory Go package name to place the Go code into + * `--type`: Optional Go type name to assign to the binding struct + * `--out`: Optional output path for the generated Go source file (not set = stdout) + +This will generate a type-safe Go binding for the Storage contract. The generated code will +look something like the snippet below, the full version of which can be viewed +[here](https://gist.github.com/jmcook1186/a78e59d203bb54b06e1b81f2cda79d93). + +```go +// Code generated - DO NOT EDIT. +// This file is a generated binding and any manual changes will be lost. + +package main + +import ( + "errors" + "math/big" + "strings" + + ethereum "github.com/ethereum/go-ethereum" + "github.com/ethereum/go-ethereum/accounts/abi" + "github.com/ethereum/go-ethereum/accounts/abi/bind" + "github.com/ethereum/go-ethereum/common" + "github.com/ethereum/go-ethereum/core/types" + "github.com/ethereum/go-ethereum/event" +) + +// Reference imports to suppress errors if they are not otherwise used. +var ( + _ = errors.New + _ = big.NewInt + _ = strings.NewReader + _ = ethereum.NotFound + _ = bind.Bind + _ = common.Big1 + _ = types.BloomLookup + _ = event.NewSubscription +) + +// StorageMetaData contains all meta data concerning the Storage contract. +var StorageMetaData = &bind.MetaData{ + ABI: "[{\"inputs\":[],\"name\":\"retrieve\",\"outputs\":[{\"internalType\":\"uint256\",\"name\":\"\",\"type\":\"uint256\"}],\"stateMutability\":\"view\",\"type\":\"function\"},{\"inputs\":[{\"internalType\":\"uint256\",\"name\":\"number\",\"type\":\"uint256\"}],\"name\":\"store\",\"outputs\":[],\"stateMutability\":\"nonpayable\",\"type\":\"function\"}]", +} + +// StorageABI is the input ABI used to generate the binding from. +// Deprecated: Use StorageMetaData.ABI instead. +var StorageABI = StorageMetaData.ABI + +// Storage is an auto generated Go binding around an Ethereum contract. +type Storage struct { + StorageCaller // Read-only binding to the contract + StorageTransactor // Write-only binding to the contract + StorageFilterer // Log filterer for contract events +} +... + +``` + +`Storage.go` contains all the bindings required to interact with `Storage.sol` from a Go application. +However, this isn't very useful unless the contract is actually deployed on Ethereum or one of +Ethereum's testnets. The following sections will demonstrate how to deploy the contract to +an Ethereum testnet and interact with it using the Go bindings. + +### Deploying contracts to Ethereum + +In the previous section, the contract ABI was sufficient for generating the contract bindings from its ABI. +However, deploying the contract requires some additional information in the form of the compiled +bytecode. + +The bytecode is obtained by running the compiler again but this passing the `--bin` flag, e.g. + +```shell +solc --bin Storage.sol -o Storage.bin +``` + +Then `abigen` can be run again, this time passing `Storage.bin`: + + +``` +$ abigen --abi Storage.abi --pkg main --type Storage --out Storage.go --bin Storage.bin +``` + +This will generate something similar to the bindings generated in the previous section. However, +an additional `DeployStorage` function has been injected: + +```go +// DeployStorage deploys a new Ethereum contract, binding an instance of Storage to it. +func DeployStorage(auth *bind.TransactOpts, backend bind.ContractBackend) (common.Address, *types.Transaction, *Storage, error) { + parsed, err := StorageMetaData.GetAbi() + if err != nil { + return common.Address{}, nil, nil, err + } + if parsed == nil { + return common.Address{}, nil, nil, errors.New("GetABI returned nil") + } + + address, tx, contract, err := bind.DeployContract(auth, *parsed, common.FromHex(StorageBin), backend) + if err != nil { + return common.Address{}, nil, nil, err + } + return address, tx, &Storage{StorageCaller: StorageCaller{contract: contract}, StorageTransactor: StorageTransactor{contract: contract}, StorageFilterer: StorageFilterer{contract: contract}}, nil +} +``` +View the full file [here](https://gist.github.com/jmcook1186/91124cfcbc7f22dcd3bb4f148d2868a8). + +The new `DeployStorage()` function can be used to deploy the contract to an Ethereum testnet from a Go application. To do this +requires incorporating the bindings into a Go application that also handles account management, authorization and Ethereum backend +to deploy the contract through. Specifically, this requires: + +1. A running Geth node connected to an Ethereum testnet (recommended Goerli) +2. An account in the keystore prefunded with enough ETH to cover gas costs for deploying and interacting with the contract + +Assuming these prerequisites exist, a new `ethclient` can be instantiated with the local Geth node's ipc file, providing +access to the testnet from the Go application. The key can be instantiated as a variable in the application by copying the +JSON object from the keyfile in the keystore. + +Putting it all together would result in: + +```go +package main + +import ( + "fmt" + "log" + "math/big" + "strings" + "time" + + "github.com/ethereum/go-ethereum/accounts/abi/bind" + "github.com/ethereum/go-ethereum/ethclient" + +) + +const key = `<>` + +func main() { + // Create an IPC based RPC connection to a remote node and an authorized transactor + conn, err := rpc.NewIPCClient("/home/go-ethereum/goerli/geth.ipc") + if err != nil { + log.Fatalf("Failed to connect to the Ethereum client: %v", err) + } + auth, err := bind.NewTransactor(strings.NewReader(key), "<>") + if err != nil { + log.Fatalf("Failed to create authorized transactor: %v", err) + } + // Deploy the contract passing the newly created `auth` and `conn` vars + address, tx, instance, err := DeployStorage(auth, conn), new(big.Int), "Storage contract in Go!", 0, "Go!") + if err != nil { + log.Fatalf("Failed to deploy new storage contract: %v", err) + } + fmt.Printf("Contract pending deploy: 0x%x\n", address) + fmt.Printf("Transaction waiting to be mined: 0x%x\n\n", tx.Hash()) + + time.Sleep(250 * time.Millisecond) // Allow it to be processed by the local node :P + + // function call on `instance`. Retrieves pending name + name, err := instance.Name(&bind.CallOpts{Pending: true}) + if err != nil { + log.Fatalf("Failed to retrieve pending name: %v", err) + } + fmt.Println("Pending name:", name) +} +``` + +Running this code requests the creation of a brand new `Storage` contract on the Goerli blockchain. +The contract functions can be called while the contract is waiting to be mined. + +``` +Contract pending deploy: 0x46506d900559ad005feb4645dcbb2dbbf65e19cc +Transaction waiting to be mined: 0x6a81231874edd2461879b7280ddde1a857162a744e3658ca7ec276984802183b + +Pending name: Storage contract in Go! +``` + +Once mined, the contract exists permanently at its deployment address and can now be interacted with +from other applications without ever needing to be redeployed. + +Note that `DeployStorage` returns four variables: + +- `address`: the deployment address of the contract + +- `tx`: the transaction hash that can be queried using Geth or a service like [Etherscan](etherscan.io) + +- `instance`: an instance of the deployed contract whose functions can be called in the Go application + +- `err`: a variable that handles errors in case of a deployment failure + + +### Accessing an Ethereum contract + +To interact with a contract already deployed on the blockchain, the deployment `address` is required and +a `backend` through which to access Ethereum must be defined. The binding generator provides an RPC +backend out-of-the-box that can be used to attach to an existing Ethereum node via IPC, HTTP or WebSockets. + +As in the previous section, a Geth node running on an Ethereum testnet (recommend Goerli) and an account +with some test ETH to cover gas is required. The `Storage.sol` deployment address is also needed. + +Again, an instance of `ethclient` can be created, passing the path to Geth's ipc file. In the example +below this backend is assigned to the variable `conn`. + +```go +// Create an IPC based RPC connection to a remote node +// NOTE update the path to the ipc file! +conn, err := ethclient.Dial("/home/go-ethereum/goerli/geth.ipc") +if err != nil { + log.Fatalf("Failed to connect to the Ethereum client: %v", err) +} +``` + +The functions available for interacting with the `Storage` contract are defined in `Storage.go`. To create +a new instance of the contract in a Go application, the `NewStorage()` function can be used. The function +is defined in `Storage.go` as follows: + +```go +// NewStorage creates a new instance of Storage, bound to a specific deployed contract. +func NewStorage(address common.Address, backend bind.ContractBackend) (*Storage, error) { + contract, err := bindStorage(address, backend, backend, backend) + if err != nil { + return nil, err + } + return &Storage{StorageCaller: StorageCaller{contract: contract}, StorageTransactor: StorageTransactor{contract: contract}, StorageFilterer: StorageFilterer{contract: contract}}, nil +} +``` + +`NewStorage()` takes two arguments: the deployment address and a backend (`conn`) and returns +an instance of the deployed contract. In the example below, the instance is assigned to `store`. + + +```go +package main + +import ( + "fmt" + "log" + + "github.com/ethereum/go-ethereum/common" + "github.com/ethereum/go-ethereum/ethclient" +) + +func main() { + // Create an IPC based RPC connection to a remote node + // NOTE update the path to the ipc file! + conn, err := ethclient.Dial("/home/go-ethereum/goerli/geth.ipc") + if err != nil { + log.Fatalf("Failed to connect to the Ethereum client: %v", err) + } + // Instantiate the contract and display its name + // NOTE update the deployment address! + store, err := NewStorage(common.HexToAddress("0x21e6fc92f93c8a1bb41e2be64b4e1f88a54d3576"), conn) + if err != nil { + log.Fatalf("Failed to instantiate Storage contract: %v", err) + } + +``` + +The contract instance is then available to interact with in the Go application. To read a value from +the blockchain, for example the `value` stored in the contract, the contract's `Retrieve()` function +can be called. Again, the function is defined in `Storage.go` as follows: + +```go +// Retrieve is a free data retrieval call binding the contract method 0x2e64cec1. +// +// Solidity: function retrieve() view returns(uint256) +func (_Storage *StorageCaller) Retrieve(opts *bind.CallOpts) (*big.Int, error) { + var out []interface{} + err := _Storage.contract.Call(opts, &out, "retrieve") + + if err != nil { + return *new(*big.Int), err + } + + out0 := *abi.ConvertType(out[0], new(*big.Int)).(**big.Int) + + return out0, err + +} +``` + +Note that the `Retrieve()` function requires a parameter to be passed, even though the +original Solidity contract didn't require any at all none. The parameter required is +a `*bind.CallOpts` type, which can be used to fine tune the call. If no adjustments to the +call are required, pass `nil`. Adjustments to the call include: + +* `Pending`: Whether to access pending contract state or the current stable one +* `GasLimit`: Place a limit on the computing resources the call might consume + +So to call the `Retrieve()` function in the Go application: + +```go +value, err := store.Retrieve(nil) +if err != nil { + log.Fatalf("Failed to retrieve value: %v", err) +} +fmt.Println("Value: ", value) +} +``` + +The output will be something like: + +```terminal +Value: 56 +``` + +### Transacting with an Ethereum contract + +Invoking a method that changes contract state (i.e. transacting) is a bit more involved, +as a live transaction needs to be authorized and broadcast into the network. **Go bindings +require local signing of transactions and do not delegate this to a remote node.** This is +to keep accounts private within dapps, and not shared (by default) between them. + +Thus to allow transacting with a contract, your code needs to implement a method that +given an input transaction, signs it and returns an authorized output transaction. Since +most users have their keys in the [Web3 Secret Storage][web3-ss-link] format, the `bind` +package contains a small utility method (`bind.NewTransactor(keyjson, passphrase)`) that can +create an authorized transactor from a key file and associated password, without the user +needing to implement key signing themselves. + +Changing the previous code snippet to update the value stored in the contract: + +```go +package main + +import ( + "fmt" + "log" + "math/big" + "strings" + + "github.com/ethereum/go-ethereum/accounts/abi/bind" + "github.com/ethereum/go-ethereum/common" + "github.com/ethereum/go-ethereum/ethclient" +) + +const key = `json object from keystore` + +func main() { + // Create an IPC based RPC connection to a remote node and instantiate a contract binding + conn, err := ethclient.Dial("/home/go-ethereum/goerli/geth.ipc") + if err != nil { + log.Fatalf("Failed to connect to the Ethereum client: %v", err) + } + store, err := NewStorage(common.HexToAddress("0x21e6fc92f93c8a1bb41e2be64b4e1f88a54d3576"), conn) + if err != nil { + log.Fatalf("Failed to instantiate a Storage contract: %v", err) + } + // Create an authorized transactor and call the store function + auth, err := bind.NewStorageTransactor(strings.NewReader(key), "strong_password") + if err != nil { + log.Fatalf("Failed to create authorized transactor: %v", err) + } + // Call the store() function + tx, err := store.Store(auth, big.NewInt(420)) + if err != nil { + log.Fatalf("Failed to update value: %v", err) + } + fmt.Printf("Update pending: 0x%x\n", tx.Hash()) +} +``` + +And the output: + +```terminal +Update pending: 0x4f4aaeb29ed48e88dd653a81f0b05d4df64a86c99d4e83b5bfeb0f0006b0e55b +``` + +Similar to the method invocations in the previous section which only read contract state, +transacting methods also require a mandatory first parameter, a `*bind.TransactOpts` type, +which authorizes the transaction and potentially fine tunes it: + +* `From`: Address of the account to invoke the method with (mandatory) +* `Signer`: Method to sign a transaction locally before broadcasting it (mandatory) +* `Nonce`: Account nonce to use for the transaction ordering (optional) +* `GasLimit`: Place a limit on the computing resources the call might consume (optional) +* `GasPrice`: Explicitly set the gas price to run the transaction with (optional) +* `Value`: Any funds to transfer along with the method call (optional) + +The two mandatory fields are automatically set by the `bind` package if the auth options are +constructed using `bind.NewTransactor`. The nonce and gas related fields are automatically +derived by the binding if they are not set. Unset values are assumed to be zero. + + +### Pre-configured contract sessions + +Reading and state modifying contract-calls require a mandatory first parameter which can +authorize and fine tune some of the internal parameters. However, most of the time the +same accounts and parameters will be used to issue many transactions, so constructing +the call/transact options individually quickly becomes unwieldy. + +To avoid this, the generator also creates specialized wrappers that can be pre-configured with +tuning and authorization parameters, allowing all the Solidity defined methods to be invoked +without needing an extra parameter. + +These are named similarly to the original contract type name but suffixed with `Sessions`: + +```go +// Wrap the Storage contract instance into a session +session := &StorageSession{ + Contract: store, + CallOpts: bind.CallOpts{ + Pending: true, + }, + TransactOpts: bind.TransactOpts{ + From: auth.From, + Signer: auth.Signer, + GasLimit: big.NewInt(3141592), + }, +} +// Call the previous methods without the option parameters +session.Store(big.NewInt(69)) +``` + +## Bind Solidity directly + +In the past, abigen allowed compilation and binding of a Solidity source file directly to a Go package in a single step. +This feature has been discontinued from [v1.10.18](https://github.com/ethereum/go-ethereum/releases/tag/v1.10.18) +onwards due to maintenance synchronization challenges with the compiler in Geth. + +The compilation and binding steps can be joined together into a pipeline, for example: +``` +solc Storage.sol --combined-json abi,bin | abigen --pkg main --type storage --out Storage.go --combined-json - +``` + +### Project integration (`go generate`) + +The `abigen` command was made in such a way as to integrate easily into existing +Go toolchains: instead of having to remember the exact command needed to bind an Ethereum +contract into a Go project, `go generate` can handle all the fine details. + +Place the binding generation command into a Go source file before the package definition: + +``` +//go:generate abigen --sol Storage.sol --pkg main --out Storage.go +``` + +After which whenever the Solidity contract is modified, instead of needing to remember and +run the above command, we can simply call `go generate` on the package (or even the entire +source tree via `go generate ./...`), and it will correctly generate the new bindings for us. + + +## Blockchain simulator + +Being able to deploy and access deployed Ethereum contracts from native Go code is a powerful +feature. However, using public testnets as a backend does not lend itself well to +*automated unit testing*. Therefore, Geth also implements a *simulated blockchain* +that can be set as a backend to native contracts the same way as a live RPC backend, using the +command `backends.NewSimulatedBackend(genesisAccounts)`. The code snippet below shows how this +can be used as a backend in a Go applicatioon. + +```go +package main + +import ( + "fmt" + "log" + "math/big" + + "github.com/ethereum/go-ethereum/accounts/abi/bind" + "github.com/ethereum/go-ethereum/accounts/abi/bind/backends" + "github.com/ethereum/go-ethereum/core" + "github.com/ethereum/go-ethereum/crypto" +) + +func main() { + // Generate a new random account and a funded simulator + key, _ := crypto.GenerateKey() + auth := bind.NewKeyedTransactor(key) + + sim := backends.NewSimulatedBackend(core.GenesisAccount{Address: auth.From, Balance: big.NewInt(10000000000)}) + + // instantiate contract + store, err := NewStorage(common.HexToAddress("0x21e6fc92f93c8a1bb41e2be64b4e1f88a54d3576"), sim) + if err != nil { + log.Fatalf("Failed to instantiate a Storage contract: %v", err) + } + // Create an authorized transactor and call the store function + auth, err := bind.NewStorageTransactor(strings.NewReader(key), "strong_password") + if err != nil { + log.Fatalf("Failed to create authorized transactor: %v", err) + } + // Call the store() function + tx, err := store.Store(auth, big.NewInt(420)) + if err != nil { + log.Fatalf("Failed to update value: %v", err) + } + fmt.Printf("Update pending: 0x%x\n", tx.Hash()) +} +``` + +Note, that it is not necessary to wait for a local private chain miner, or testnet miner to +integrate the currently pending transactions. To mine the next block, simply `Commit()` the simulator. + + +## Summary + +To make interacting with Ethereum contracts easier for Go developers, Geth provides tools that generate +contract bindings automatically. This makes contract functions available in Go native applications. + + +[go-link]:https://github.com/golang/go/wiki#getting-started-with-go +[truffle-link]:https://trufflesuite.com/docs/truffle/ +[hardhat-link]:https://hardhat.org/ +[brownie-link]:https://eth-brownie.readthedocs.io/en/stable/ +[remix-link]:https://remix.ethereum.org/ +[web3-ss-link]:https://github.com/ethereum/wiki/wiki/Web3-Secret-Storage-Definition \ No newline at end of file diff --git a/content/docs/developers/dapp-developer/native.md b/content/docs/developers/dapp-developer/native.md new file mode 100644 index 0000000000..34d9816d97 --- /dev/null +++ b/content/docs/developers/dapp-developer/native.md @@ -0,0 +1,249 @@ +--- +title: Go API +sort_key: C +--- + +Ethereum was originally conceptualized to be the base layer for [Web3][web3-link], providing +the backbone for a new generation of decentralized, permissionless and censorship resistant +applications called [dapps][dapp-link]. The first step towards this vision was the development +of clients providing an RPC interface into the peer-to-peer protocols. This allowed users to +transact between accounts and interact with smart contracts using command line tools. +Geth was one of the original clients to provide this type of gateway to the Ethereum network. + +Before long, web-browser-like graphical interfaces (e.g. Mist) were created to extend clients, and +client functions were built into websites built using the time-tested HTML/CSS/JS stack. +However, to support the most diverse, complex dapps, developers require programmatic access to client +functions through an API. This opens up client technologies as re-usable, composable units that +can be applied in creative ways by a global community of developers. + +To support this, Geth ships official Go packages that can be embedded into third party +desktop and server applications. There is also a [mobile API](/docs/dapp/mobile) that can be +used to embed Geth into mobile applications. + +This page provides a high-level overview of the Go API. + +*Note, this guide will assume some familiarity with Go development. It does not cover general topics +about Go project layouts, import paths or any other standard methodologies. If you are new to Go, +consider reading [Getting Started with Go][go-guide] first.* + +## Overview + +Geth's reusable Go libraries focus on three main usage areas: + +- Simplified client side account management +- Remote node interfacing via different transports +- Contract interactions through auto-generated bindings + +The libraries are updated synchronously with the Geth Github repository. +The Go libraries can be viewed in full at [Go Packages][go-pkg-link]. + +Péter Szilágyi (@karalabe) gave a high level overview of the Go libraries in +a talk at DevCon2 in Shanghai in 2016. The slides are still a useful resource +([available here][peter-slides]) and the talk itself can be viewed by clicking +the image below (it is also archived on [IPFS][ipfs-link]). + +[![Peter's Devcon2 talk](/static/images/devcon2_labelled.webp)](https://www.youtube.com/watch?v=R0Ia1U9Gxjg) + +## Go packages + +The `go-ethereum` library is distributed as a collection of standard Go packages straight from go-ethereum's +GitHub repository. The packages can be used directly via the official Go toolkit, without needing any +third party tools. + +The canonical import path for Geth is `github.com/ethereum/go-ethereum`, with all packages residing +underneath. Although there are [lots of them][go-ethereum-dir] most developers will only care about +a limited subset. + +All the Geth packages can be downloaded using: + +``` +$ go get -d github.com/ethereum/go-ethereum/... +``` + +More Go API support for dapp developers can be found on the [Go Contract Bindings](/docs/dapp/native-bindings) +and [Go Account Management](/docs/dapp/native-accounts) pages. + + +## Tutorial + +This section includes some basic usage examples for the `ethclient` and `gethclient` packages available as +part of the Go API. The `ethclient` package provides a client that implements the full Ethereum JSON-RPC API, +whereas `gethclient` offers the Geth-specific API. + +### Instantiating a client + +The client is an instance of the `Client` struct which has associated functions that wrap requests to the Ethereum +or Geth RPC API endpoints. + +A client is instantiated by passing a raw url or path to an ipc file to the client's `Dial` function. In the following +code snippet the path to the ipc file for a local Geth node is provided to `ethclient.Dial()`. + +```go +// create instance of ethclient and assign to cl +cl, err := ethclient.Dial("/tmp/geth.ipc") +if err != nil { + panic(err) +} +_ = cl +``` + +### Interacting with the client + +The client can now be used to handle requests to the Geth node using the full JSON-RPC API. For example, the function +`BlockNumer()` wraps a call to the `eth_blockNumber` endpoint. The function `SendTransaction` wraps a call to +`eth_sendTransaction`. The full list of client methods can be found [here][ethclient-pkg]. + +Frequently, the functions take an instance of the `Context` type as their leading argument. This defines context about requests sent from the application such as deadlines, cancellation signals etc. More information on this can +be found in the [Go documentation](https://pkg.go.dev/golang.org/x/net/context). An empty context instance can be +created using `Context.Background()`. + +### Querying client for data + +A simple starting point is to fetch the chain ID from the client. This e.g. is needed when signing a transaction as is to be seen in the next section. + +```go +chainid, err := cl.ChainID(context.Background()) +if err != nil { + return err +} +``` + +Unlike `ChainID`, many functions require arguments other than context. The Go API takes in and returns high-level types which are used in Geth internals as well to simplify programming and remove the need for knowing how data needs to be formatted exactly as per the JSON-RPC API spec. For example to find out the nonce for an account at a given block the address needs to be provided as a `common.Address` type and the block number as a `*big.Int`: + +```go +addr := common.HexToAddress("0xb02A2EdA1b317FBd16760128836B0Ac59B560e9D") +nonce, err := cl.NonceAt(context.Background(), addr, big.NewInt(14000000)) +``` + +### Querying past events + +Contracts emit events during execution which can be queried from the client. The parameters for the event one is interested in have to be filled out in the `ethereum.FilterQuery` object. This includes which event topics are of interested, from which contracts and during which range of blocks. The example below queries `Transfer` events of all ERC-20 tokens for the last 10 blocks: + +```go +blockNum, err := cl.BlockNumber(context.Background()) +if err != nil { + return err +} +q := ethereum.FilterQuery{ + FromBlock: new(big.Int).Sub(blockNum, big.NewInt(10)), + ToBlock: blockNum, + Topics: [][]common.Hash{common.HexToHash("0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef")}, +} +logs, err := cl.FilterLogs(context.Background(), q) +if err != nil { + return err +} +``` + +### Sending a transaction + +Sending a transaction is achieved using the `SendTransaction()` function. `SendTransaction` takes an instance of +`context.Context` as its leading argument and a signed transaction as its second argument. The signed transaction +must be generated in advance. Building the signed transaction is a multi-stage +process that requires first generating a key pair if none exists already, retrieving some chain data and defining sender and recipient +addresses. Then these data can be collected into a transaction object and signed. The resulting signed transaction +can then be passed to `SendTransaction`. + +The example below assumes the following key pair has already been generated: + +```go +// SK and ADDR are the secret key and sender address +SK = "0xaf5ead4413ff4b78bc94191a2926ae9ccbec86ce099d65aaf469e9eb1a0fa87f" +ADDR = "0x6177843db3138ae69679A54b95cf345ED759450d" +``` + +The secret key and address can be used to send a transaction. In the example below 1 ETH is sent from the +address `ADDR` to an arbitrary recipient. + +```go +import ( + "context" + "math/big" + + "github.com/ethereum/go-ethereum/common" + "github.com/ethereum/go-ethereum/core/types" + "github.com/ethereum/go-ethereum/crypto" + "github.com/ethereum/go-ethereum/ethclient" + "github.com/ethereum/go-ethereum/params" +) + +// sendTransaction sends a transaction with 1 ETH to a specified address. +func sendTransaction(cl *ethclient.Client) error { + var ( + sk = crypto.ToECDSAUnsafe(common.FromHex(SK)) + to = common.HexToAddress("0xb02A2EdA1b317FBd16760128836B0Ac59B560e9D") + value = new(big.Int).Mul(big.NewInt(1), big.NewInt(params.Ether)) + sender = common.HexToAddress(ADDR) + gasLimit = uint64(21000) + ) + // Retrieve the chainid (needed for signer) + chainid, err := cl.ChainID(context.Background()) + if err != nil { + return err + } + // Retrieve the pending nonce + nonce, err := cl.PendingNonceAt(context.Background(), sender) + if err != nil { + return err + } + // Get suggested gas price + tipCap, _ := cl.SuggestGasTipCap(context.Background()) + feeCap, _ := cl.SuggestGasPrice(context.Background()) + // Create a new transaction + tx := types.NewTx( + &types.DynamicFeeTx{ + ChainID: chainid, + Nonce: nonce, + GasTipCap: tipCap, + GasFeeCap: feeCap, + Gas: gasLimit, + To: &to, + Value: value, + Data: nil, + }) + // Sign the transaction using our keys + signedTx, _ := types.SignTx(tx, types.NewLondonSigner(chainid), sk) + // Send the transaction to our node + return cl.SendTransaction(context.Background(), signedTx) +} +``` + +### gethclient + +An instance of `gethclient` can be used in exactly the same way as `ethclient`. However, `gethclient` +includes Geth-specific API methods. These additional methods are: + +```shell +CallContract() +CreatAccessList() +GCStats() +GetNodeInfo() +GetProof() +MemStats() +SetHead() +SubscribePendingTransactions() +``` +*Note that both `ethclient` and `gethclient` have a `CallContract()` function - the difference is that +the `gethclient` version includes an `overrides` argument.* + +Details relating to these endpoints can be found at [pkg.go.dev][go-api-docs] or the Geth [Github][ethclient-link]. +The code snippets in this tutorial were adapted from a more more in-depth set of examples available on +[Github][web3go-link]. + + +## Summary + +There are a wide variety of Go APIs available for dapp developers that abstract away the complexity of interacting with Ethereum +using a set of composable, reusable functions provided by Geth. + +[go-guide]: https://github.com/golang/go/wiki#getting-started-with-go +[peter-slides]: https://ethereum.karalabe.com/talks/2016-devcon.html +[go-ethereum-dir]: https://pkg.go.dev/github.com/ethereum/go-ethereum/#section-directories +[ethclient-pkg]:https://pkg.go.dev/github.com/ethereum/go-ethereum/ethclient#Client +[go-pkg-link]: https://pkg.go.dev/github.com/ethereum/go-ethereum#section-directories +[ipfs-link]: https://ipfs.io/ipfs/QmQRuKPKWWJAamrMqAp9rytX6Q4NvcXUKkhvu3kuREKqXR +[dapp-link]: https://ethereum.org/en/glossary/#dapp +[web3-link]: https://ethereum.org/en/web3/ +[ethclient-link]: https://github.com/ethereum/go-ethereum/tree/master/ethclient +[go-api-docs]:https://pkg.go.dev/github.com/ethereum/go-ethereum@v1.10.19/ethclient/gethclient +[web3go-link]:https://github.com/MariusVanDerWijden/web3go diff --git a/content/docs/developers/dapp-developer/tracing.md b/content/docs/developers/dapp-developer/tracing.md new file mode 100644 index 0000000000..74dca40f83 --- /dev/null +++ b/content/docs/developers/dapp-developer/tracing.md @@ -0,0 +1,232 @@ +--- +title: EVM Tracing +sort_key: A +--- + +There are two different types of [transactions][transactions] +in Ethereum: simple value transfers and contract executions. A value transfer just +moves Ether from one account to another. If however the recipient of a transaction is +a contract account with associated [EVM][evm] (Ethereum Virtual Machine) bytecode - beside +transferring any Ether - the code will also be executed as part of the transaction. + +Having code associated with Ethereum accounts permits transactions to do arbitrarily +complex data storage and enables them to act on the previously stored data by further +transacting internally with outside accounts and contracts. This creates an interlinked +ecosystem of contracts, where a single transaction can interact with tens or hundreds of +accounts. + +The downside of contract execution is that it is very hard to say what a transaction +actually did. A transaction receipt does contain a status code to check whether execution +succeeded or not, but there is no way to see what data was modified, nor what external +contracts where invoked. Geth resolves this by re-running transactions locally and collecting +data about precisely what was executed by the EVM. This is known as "tracing" the transaction. + + +* TOC +{:toc} + + +## Tracing prerequisites + +In its simplest form, tracing a transaction entails requesting the Ethereum node to +reexecute the desired transaction with varying degrees of data collection and have it +return the aggregated summary for post processing. Reexecuting a transaction however has a +few prerequisites to be met. + +In order for an Ethereum node to reexecute a transaction, all historical state accessed +by the transaction must be available. This includes: + +* Balance, nonce, bytecode and storage of both the recipient as well as all internally invoked contracts. +* Block metadata referenced during execution of both the outer as well as all internally created transactions. +* Intermediate state generated by all preceding transactions contained in the same block as the one being traced. + +This means there are limits on the transactions that can be traced imposed by the synchronization and +pruning configuration of a node. + +* An **archive** node retains **all historical data** back to genesis. It can therefore +trace arbitrary transactions at any point in the history of the chain. Tracing a single +transaction requires reexecuting all preceding transactions in the same block. + +* A **full synced** node retains the most recent 128 blocks in memory, so transactions in +that range are always accessible. Full nodes also store occasional checkpoints back to genesis +that can be used to rebuild the state at any point on-the-fly. This means older transactions +can be traced but if there is a large distance between the requested transaction and the most +recent checkpoint rebuilding the state can take a long time. Tracing a single +transaction requires reexecuting all preceding transactions in the same block +**and** all preceding blocks until the previous stored snapshot. + +* A **snap synced** node holds the most recent 128 blocks in memory, so transactions in that +range are always accessible. However, snap-sync only starts processing from a relatively recent +block (as opposed to genesis for a full node). Between the initial sync block and the 128 most +recent blocks, the node stores occasional checkpoints that can be used to rebuild the state on-the-fly. +This means transactions can be traced back as far as the block that was used for the initial sync. +Tracing a single transaction requires reexecuting all preceding transactions in the same block, +**and** all preceding blocks until the previous stored snapshot. + +* A **light synced** node retrieving data **on demand** can in theory trace transactions +for which all required historical state is readily available in the network. This is because the data +required to generate the trace is requested from an les-serving full node. In practice, data +availability **cannot** be reasonably assumed. + +*There are exceptions to the above rules when running batch traces of entire blocks or +chain segments. Those will be detailed later.* + +## Basic traces + +The simplest type of transaction trace that Geth can generate are raw EVM opcode +traces. For every VM instruction the transaction executes, a structured log entry is +emitted, containing all contextual metadata deemed useful. This includes the *program +counter*, *opcode name*, *opcode cost*, *remaining gas*, *execution depth* and any +*occurred error*. The structured logs can optionally also contain the content of the +*execution stack*, *execution memory* and *contract storage*. + +The entire output of a raw EVM opcode trace is a JSON object having a few metadata +fields: *consumed gas*, *failure status*, *return value*; and a list of *opcode entries*: + +```json +{ + "gas": 25523, + "failed": false, + "returnValue": "", + "structLogs": [] +} +``` + +An example log for a single opcode entry has the following format: + +```json +{ + "pc": 48, + "op": "DIV", + "gasCost": 5, + "gas": 64532, + "depth": 1, + "error": null, + "stack": [ + "00000000000000000000000000000000000000000000000000000000ffffffff", + "0000000100000000000000000000000000000000000000000000000000000000", + "2df07fbaabbe40e3244445af30759352e348ec8bebd4dd75467a9f29ec55d98d" + ], + "memory": [ + "0000000000000000000000000000000000000000000000000000000000000000", + "0000000000000000000000000000000000000000000000000000000000000000", + "0000000000000000000000000000000000000000000000000000000000000060" + ], + "storage": { + } +} +``` + +### Generating basic traces + +To generate a raw EVM opcode trace, Geth provides a few [RPC API endpoints](/docs/rpc/ns-debug). +The most commonly used is [`debug_traceTransaction`](/docs/rpc/ns-debug#debug_tracetransaction). + +In its simplest form, `traceTransaction` accepts a transaction hash as its only argument. It then +traces the transaction, aggregates all the generated data and returns it as a **large** +JSON object. A sample invocation from the Geth console would be: + +```js +debug.traceTransaction("0xfc9359e49278b7ba99f59edac0e3de49956e46e530a53c15aa71226b7aa92c6f") +``` + +The same call can also be invoked from outside the node too via HTTP RPC (e.g. using Curl). In this +case, the HTTP endpoint must be enabled in Geth using the `--http` command and the `debug` API +namespace must be exposed using `--http.api=debug`. + +``` +$ curl -H "Content-Type: application/json" -d '{"id": 1, "method": "debug_traceTransaction", "params": ["0xfc9359e49278b7ba99f59edac0e3de49956e46e530a53c15aa71226b7aa92c6f"]}' localhost:8545 +``` + +To follow along with this tutorial, transaction hashes can be found from a local Geth node (e.g. by +attaching a [Javascript console](/docs/interface/javascript-console) and running `eth.getBlock('latest')` +then passing a transaction hash from the returned block to `debug.traceTransaction()`) or from a block +explorer (for [Mainnet](https://etherscan.io/) or a [testnet](https://goerli.etherscan.io/)). + +It is also possible to configure the trace by passing Boolean (true/false) values for four parameters +that tweak the verbosity of the trace. By default, the *EVM memory* and *Return data* are not reported +but the *EVM stack* and *EVM storage* are. To report the maximum amount of data: + +```shell +enableMemory: true +disableStack: false +disableStorage: false +enableReturnData: true +``` + +An example call, made in the Geth Javascript console, configured to report the maximum amount of data +looks as follows: + +```js +debug.traceTransaction("0xfc9359e49278b7ba99f59edac0e3de49956e46e530a53c15aa71226b7aa92c6f",{enableMemory: true, disableStack: false, disableStorage: false, enableReturnData: true}) +``` + +Running the above operation on the Rinkeby network (with a node retaining enough history) +will result in this [trace dump](https://gist.github.com/karalabe/c91f95ac57f5e57f8b950ec65ecc697f). + +Alternatively, disabling *EVM Stack*, *EVM Memory*, *Storage* and *Return data* (as demonstrated in the Curl request below) +results in the following, much shorter, [trace dump](https://gist.github.com/karalabe/d74a7cb33a70f2af75e7824fc772c5b4). + +``` +$ curl -H "Content-Type: application/json" -d '{"id": 1, "method": "debug_traceTransaction", "params": ["0xfc9359e49278b7ba99f59edac0e3de49956e46e530a53c15aa71226b7aa92c6f", {"disableStack": true, "disableStorage": true}]}' localhost:8545 +``` + +### Limits of basic traces + +Although the raw opcode traces generated above are useful, having an individual log entry for every single +opcode is too low level for most use cases, and will require developers to create additional tools to +post-process the traces. Additionally, a full opcode trace can easily go into the hundreds of +megabytes, making them very resource intensive to get out of the node and process externally. + +To avoid those issues, Geth supports running custom JavaScript tracers *within* the Ethereum node, +which have full access to the EVM stack, memory and contract storage. This means developers only have to +gather the data they actually need, and do any processing at the source. + +## Pruning + +Geth does in-memory state-pruning by default, discarding state entries that it deems +no longer necessary to maintain. This is configured via the `--gcmode` command. An error +message alerting the user that the necessary state is not available is common in EVM tracing on +anything other than an archive node. + +```sh +Error: required historical state unavailable (reexec=128) + at web3.js:6365:37(47) + at send (web3,js:5099:62(35)) + at :1:23(13) +``` + +The pruning behaviour, and consequently the state availability and tracing capability of +a node depends on its sync and pruning configuration. The 'oldest' block after which +state is immediately available, and before which state is not immediately available, +is known as the "pivot block". There are then several possible cases for a trace request +on a Geth node. + +For tracing a transaction in block `B` where the pivot block is `P` can regenerate the desired +state by replaying blocks from the last : + +1. a fast-sync'd node can regenerate the desired state by replaying blocks from the most recent +checkpoint between `P` and `B` as long as `P` < `B`. If `P` > `B` there is no available checkpoint +and the state cannot be regenerated without replying the chain from genesis. + +2. a fully sync'd node can regenerate the desired state by replaying blocks from the last available +full state before `B`. A fully sync'd node re-executes all blocks from genesis, so checkpoints are available +across the entire history of the chain. However, database pruning discards older data, moving `P` to a more +recent position in the chain. If `P` > `B` there is no available checkpoint and the state cannot be +regenerated without replaying the chain from genesis. + +3. A fully-sync'd node without pruning (i.e. an archive node configured with `--gcmode=archive`) +does not need to replay anything, it can immediately load up any state and serve the request for any `B`. + +The time taken to regenerate a specific state increases with the distance between `P` and `B`. If the distance +between `P` and `B` is large, the regeneration time can be substantial. + +## Summary + +This page covered the concept of EVM tracing and how to generate traces with the default opcode-based tracers using RPC. +More advanced usage is possible, including using other built-in tracers as well as writing [custom tracing](/docs/dapp/custom-tracer) code in Javascript +and Go. The API as well as the JS tracing hooks are defined in [the reference](/docs/rpc/ns-debug#debug_traceTransaction). + + +[transactions]: https://ethereum.org/en/developers/docs/transactions +[evm]: https://ethereum.org/en/developers/docs/evm diff --git a/content/docs/developers/geth-developer/Code-Review-Guidelines.md b/content/docs/developers/geth-developer/Code-Review-Guidelines.md new file mode 100644 index 0000000000..cfe6ae85d3 --- /dev/null +++ b/content/docs/developers/geth-developer/Code-Review-Guidelines.md @@ -0,0 +1,103 @@ +--- +title: Code Review Guidelines +sort_key: B +--- + +The only way to get code into go-ethereum is to send a pull request. Those pull requests +need to be reviewed by someone. This document is a guide that explains our expectations +around PRs for both authors and reviewers. + +## Terminology + +* The **author** of a pull request is the entity who wrote the diff and submitted it to + GitHub. +* The **team** consists of people with commit rights on the go-ethereum repository. +* The **reviewer** is the person assigned to review the diff. The reviewer must be a team + member. +* The **code owner** is the person responsible for the subsystem being modified by the PR. + +## The Process + +The first decision to make for any PR is whether it's worth including at all. This +decision lies primarily with the code owner, but may be negotiated with team members. + +To make the decision we must understand what the PR is about. If there isn't enough +description content or the diff is too large, request an explanation. Anyone can do this +part. + +We expect that reviewers check the style and functionality of the PR, providing comments +to the author using the GitHub review system. Reviewers should follow up with the PR until +it is in good shape, then **approve** the PR. Approved PRs can be merged by any code owner. + +When communicating with authors, be polite and respectful. + +### Code Style + +We expect `gofmt`ed code. For contributions of significant size, we expect authors to +understand and use the guidelines in [Effective Go][effgo]. Authors should avoid common +mistakes explained in the [Go Code Review Comments][revcomment] page. + +### Functional Checks + +For PRs that fix an issue, reviewers should try reproduce the issue and verify that the +pull request actually fixes it. Authors can help with this by including a unit test that +fails without (and passes with) the change. + +For PRs adding new features, reviewers should attempt to use the feature and comment on +how it feels to use it. Example: if a PR adds a new command line flag, use the program +with the flag and comment on whether the flag feels useful. + +We expect appropriate unit test coverage. Reviewers should verify that new code is covered +by unit tests. + +### CI + +Code submitted must pass all unit tests and static analysis ("lint") checks. We use Travis +CI to test code on Linux, macOS and AppVeyor to test code on Microsoft Windows. + +For failing CI builds, the issue may not be related to the PR itself. Such failures are +usually related to flakey tests. These failures can be ignored (authors don't need to fix +unrelated issues), but please file a GH issue so the test gets fixed eventually. + +### Commit Messages + +Commit messages on the master branch should follow the rule below. PR authors are not +required to use any particular style because the message can be modified at merge time. +Enforcing commit message style is the responsibility of the person merging the PR. + +The commit message style we use is similar to the style used by the Go project: + +The first line of the change description is conventionally a one-line summary of the +change, prefixed by the primary affected Go package. It should complete the sentence "This +change modifies go-ethereum to _____." The rest of the description elaborates and should +provide context for the change and explain what it does. + +Template: + +```text +package/path: change XYZ + +Longer explanation of the change in the commit. You can use +multiple sentences here. It's usually best to include content +from the PR description in the final commit message. + +issue notices, e.g. "Fixes #42353". +``` + +### Special Situations And How To Deal With Them + +As a reviewer, you may find yourself in one of the sitations below. Here's how to deal +with those: + +* The author doesn't follow up: ping them after a while (i.e. after a few days). If there + is no further response, close the PR or complete the work yourself. + +* Author insists on including refactoring changes alongside bug fix: We can tolerate small + refactorings alongside any change. If you feel lost in the diff, ask the author to + submit the refactoring as an independent PR, or at least as an independent commit in the + same PR. + +* Author keeps rejecting your feedback: reviewers have authority to reject any change for technical reasons. If you're unsure, ask the team for a second opinion. You may close the PR if no consensus can be reached. + +[effgo]: https://golang.org/doc/effective_go.html +[revcomment]: https://github.com/golang/go/wiki/CodeReviewComments diff --git a/content/docs/developers/geth-developer/Private-Network.md b/content/docs/developers/geth-developer/Private-Network.md new file mode 100644 index 0000000000..e54dabcdac --- /dev/null +++ b/content/docs/developers/geth-developer/Private-Network.md @@ -0,0 +1,481 @@ +--- +title: Private Networks +sort_key: D +--- + +This guide explains how to set up a private network of multiple Geth nodes. An Ethereum network is private if the nodes are not connected to the main network. In this context private only means reserved or isolated, rather than protected or secure. A fully controlled, private Ethereum network is useful as a backend for core developers working on issues relating to networking/blockchain syncing etc. Private networks are also useful for Dapp developers testing multi-block and multi-user scenarios. + +## Prerequisites + +To follow the tutorial on this page it is necessary to have a working Geth installation (instructions [here](/docs/install-and-build/installing-geth)). It is also helpful to understand Geth fundamentals (see [Getting Started](/docs/getting-started)). + + +## Private Networks + +A private network is composed of multiple Ethereum nodes that can only connect to each other. In order to run multiple nodes locally, each one requires a separate data directory (`--datadir`). The nodes must also know about each other and be able to exchange information, share an initial state and a common consensus algorithm. The remainder of this page will explain how to configure Geth so that these basic requirements are met, enabling a private network to be started. + + +### Choosing A Network ID + +Ethereum Mainnet has Network ID = 1. There are also many other networks that Geth can connect to by providing alternative Chain IDs, some are testnets and others are alternative networks built from forks of the Geth source code. Providing a network ID that is not already being used by an existing network or testnet means the nodes using that network ID can only connect to each other, creating a private network. A list of current network IDs is available at [Chainlist.org](https://chainlist.org/). The network ID is controlled using the `networkid` flag, e.g. + +```shell +geth --networkid 12345 +``` + + +### Choosing A Consensus Algorithm + +While the main network uses proof-of-work (PoW) to secure the blockchain, Geth also supports the the 'Clique' proof-of-authority (PoA) consensus algorithm as an alternative for private networks. Clique is strongly recommended for private testnets because PoA is far less resource-intensive than PoW. Clique is currently used as the consensus algorithm in public testnets such as [Rinkeby](https://www.rinkeby.io) and [Görli](https://goerli.net). The key differences between the consensus algorithms available in Geth are: + +#### Ethash + +Geth's PoW algorithm, [Ethhash](https://ethereum.org/en/developers/docs/consensus-mechanisms/pow/mining-algorithms/ethash), is a system that allows open participation by anyone willing to dedicate resources to mining. While this is a critical property for a public network, the overall security of the blockchain strictly depends on the total amount of resources used to secure it. As such, PoW is a poor choice for private networks with few miners. The Ethash mining 'difficulty' is adjusted automatically so that new blocks are created approximately 12 seconds apart. As more mining resources are deployed on the network, creating a new block becomes harder so that the average block time matches the target block time. + +#### Clique + +Clique consensus is a PoA system where new blocks can be created by authorized 'signers' only. The clique consenus protocol is specified in [EIP-225][clique-eip]. The initial set of authorized signers is configured in the genesis block. Signers can be authorized and de-authorized using a voting mechanism, thus allowing the set of signers to change while the blockchain operates. Clique can be configured to target any block time (within reasonable limits) since it isn't tied to the difficulty adjustment. + + +[clique-eip]: https://eips.ethereum.org/EIPS/eip-225 + + +### Creating The Genesis Block + +Every blockchain starts with a genesis block. When Geth is run with default settings for the first time, it commits the Mainnet genesis to the database. For a private network, it is generally preferable to use a different genesis block. The genesis block is configured using a _genesis.json_ file whose path must be provided to Geth on start-up. When creating a genesis block, a few initial parameters for the private blockchain must be defined: + +- Ethereum platform features enabled at launch (`config`). Enabling and disabling features once the blockchain is running requires scheduling a [hard fork](https://ethereum.org/en/glossary/#hard-fork). + +- Initial block gas limit (`gasLimit`). This impacts how much EVM computation can happen within a single block. Mirroring the main Ethereum network is generally a [good choice][gaslimit-chart]. The block gas limit can be adjusted after launch using the `--miner.gastarget` command-line flag. + +- Initial allocation of ether (`alloc`). This determines how much ether is available to the addresses listed in the genesis block. Additional ether can be created through mining as the chain progresses. + + +#### Clique Example + +Below is an example of a `genesis.json` file for a PoA network. The `config` section ensures that all known protocol changes are available and configures the 'clique' engine to be used for consensus. Note that the initial signer set must be configured through the `extradata` field. This field is required for Clique to work. + +The signer account keys can be generated using the [geth account](./managing-your-accounts) command (this command can be run multiple times to create more than one signer key). + +```shell +geth account new --datadir data +``` + +The Ethereum address printed by this command should be recorded. To encode the signer addresses in `extradata`, concatenate 32 zero bytes, all signer addresses and 65 further zero bytes. The result of this concatenation is then used as the value accompanying the `extradata` key in `genesis.json`. In the example below, `extradata` contains a single initial signer address, `0x7df9a875a174b3bc565e6424a0050ebc1b2d1d82`. + +The `period` configuration option sets the target block time of the chain. + +```json +{ + "config": { + "chainId": 12345, + "homesteadBlock": 0, + "eip150Block": 0, + "eip155Block": 0, + "eip158Block": 0, + "byzantiumBlock": 0, + "constantinopleBlock": 0, + "petersburgBlock": 0, + "istanbulBlock": 0, + "berlinBlock": 0, + "clique": { + "period": 5, + "epoch": 30000 + } + }, + "difficulty": "1", + "gasLimit": "8000000", + "extradata": "0x00000000000000000000000000000000000000000000000000000000000000007df9a875a174b3bc565e6424a0050ebc1b2d1d820000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", + "alloc": { + "7df9a875a174b3bc565e6424a0050ebc1b2d1d82": { "balance": "300000" }, + "f41c74c9ae680c1aa78f42e5647a62f353b7bdde": { "balance": "400000" } + } +} +``` + +#### Ethash Example + +Since Ethash is the default consensus algorithm, no additional parameters need to be configured in order to use it. The initial mining difficulty is influenced using the `difficulty` parameter, but note that the difficulty adjustment algorithm will quickly adapt to the amount of mining resources deployed on the chain. + +```json +{ + "config": { + "chainId": 12345, + "homesteadBlock": 0, + "eip150Block": 0, + "eip155Block": 0, + "eip158Block": 0, + "byzantiumBlock": 0, + "constantinopleBlock": 0, + "petersburgBlock": 0, + "istanbulBlock": 0, + "berlinBlock": 0, + "ethash": {} + }, + "difficulty": "1", + "gasLimit": "8000000", + "alloc": { + "7df9a875a174b3bc565e6424a0050ebc1b2d1d82": { "balance": "300000" }, + "f41c74c9ae680c1aa78f42e5647a62f353b7bdde": { "balance": "400000" } + } +} +``` + +### Initializing the Geth Database + +To create a blockchain node that uses this genesis block, first use `geth init` to import and sets the canonical genesis block for the new chain. This requires the path to `genesis.json` to be passed as an argument. + +```shell +geth init --datadir data genesis.json +``` + +When Geth is started using `--datadir data` the genesis block defined in `genesis.json` will be used. For example: + + +```shell +geth --datadir data --networkid 12345 +``` + +### Scheduling Hard Forks + +As Ethereum protocol development progresses, new features become available. To enable these features on an existing private network, a hard fork must be scheduled. To do this, a future block number must be chosen which determines precisely when the hard fork will activate. Continuing the `genesis.json` example above and assuming the current block number is 35421, a hard fork might be scheduled for block 40000. This hard fork might upgrade the network to conform to the 'London' specs. First, all the Geth instances on the private network must be recent enough to support the specific hard fork. If so, `genesis.json` can be updated so that the `londonBlock` key gets the value 40000. The Geth instances are then shut down and `geth init` is run to update their configuration. When the nodes are restarted they will pick up where they left off and run normally until block 40000, at which point they will automatically upgrade. + +The modification to `genesis.json` is as follows: + +```json +{ + "config": { + + "londonBlock": 40000, + + }, + +} +``` + +The upgrade command is: + +```shell +geth init --datadir data genesis.json +``` + + +### Setting Up Networking + +With the node configured and initialized, the next step is to set up a peer-to-peer network. This requires a bootstrap node. The bootstrap node is a normal node that is designated to be the entry point that other nodes use to join the network. Any node can be chosen to be the bootstrap node. + +To configure a bootstrap node, the IP address of the machine the bootstrap node will run on must be known. The bootsrap node needs to know its own IP address so that it can broadcast it to other nodes. On a local machine this can be found using tools such as `ifconfig` and on cloud instances such as Amazon EC2 the IP address of the virtual machine can be found in the management console. Any firewalls must allow UDP and TCP traffic on port 30303. + +The bootstrap node IP is set using the `--nat` flag (the command below contains an example address - replace it with the correct one). + +```shell +geth --datadir data --networkid 15 --nat extip:172.16.254.4 +``` + +The 'node record' of the bootnode can be extracted using the JS console: + +```shell +geth attach data/geth.ipc --exec admin.nodeInfo.enr +``` + +This command should print a base64 string such as the following example. Other nodes will use the information contained in the bootstrap node record to connect to the peer-to-peer network. + +```text +"enr:-Je4QEiMeOxy_h0aweL2DtZmxnUMy-XPQcZllrMt_2V1lzynOwSx7GnjCf1k8BAsZD5dvHOBLuldzLYxpoD5UcqISiwDg2V0aMfGhGlQhqmAgmlkgnY0gmlwhKwQ_gSJc2VjcDI1NmsxoQKX_WLWgDKONsGvxtp9OeSIv2fRoGwu5vMtxfNGdut4cIN0Y3CCdl-DdWRwgnZf" +``` + +If the nodes are intended to connect across the Internet, the bootnode and all other nodes must have public IP addresses assigned, and both TCP and UDP traffic can pass their firewalls. If Internet connectivity is not required or all member nodes connect using well-known IPs, Geth should be set up to restrict peer-to-peer connectivity to an IP subnet. Doing so will further isolate the network and prevents cross-connecting with other blockchain networks in case the nodes are reachable from the Internet. Use the +`--netrestrict` flag to configure a whitelist of IP networks: + +```shell +geth --netrestrict 172.16.254.0/24 +``` + +With the above setting, Geth will only allow connections from the 172.16.254.0/24 subnet, and will not attempt to connect to other nodes outside of the set IP range. + +### Running Member Nodes + +Before running a member node, it must be initialized with the same genesis file as used for the bootstrap node. With the bootnode operational and externally reachable (`telnet ` will confirm that it is indeed reachable), more Geth nodes can be started and connected to them via the bootstrap node using the `--bootnodes` flag. The process is to start Geth on the same machine as the bootnode, with a separate data directory and listening port and the bootnode node record provided as an argument: + +For example, using data directory (example: `data2`) and listening port (example: `30305`): + +```shell +geth --datadir data2 --networkid 12345 --port 30305 --bootnodes +``` + +With the member node running, it is possible to check that it is connected to the bootstrap node or any other node in the network by attaching a console and running `admin.peers`. It may take up to a few seconds for the nodes to get connected. + +```shell +geth attach data2/geth.ipc --exec admin.peers +``` + +### Running A Signer (Clique) + +To set up Geth for signing blocks in Clique, a signer account must be available. The account must already be available as a keyfile in the keystore. To use it for signing blocks, it must be unlocked. The following command, for address `0x7df9a875a174b3bc565e6424a0050ebc1b2d1d82` will prompt for the account password, then start signing blocks: + +```shell +geth --unlock 0x7df9a875a174b3bc565e6424a0050ebc1b2d1d82 --mine +``` + +Mining can be further configured by changing the default gas limit blocks converge to (with `--miner.gastarget`) and the price transactions are accepted at (with `--miner.gasprice`). + +### Running A Miner (Ethash) + +For PoW in a simple private network, a single CPU miner instance is enough to create a stable stream of blocks at regular intervals. To start a Geth instance for mining, it can be run with all the usual flags plus the following to configure mining: + +```shell +geth --mine --miner.threads=1 --miner.etherbase=0xf41c74c9ae680c1aa78f42e5647a62f353b7bdde +``` + +This will start mining bocks and transactions on a single CPU thread, crediting all block rewards to the account specified by `--miner.etherbase`. + + +## End-to-end example {#end-to-end-example} + +This section will run through the commands for setting up a simple private network of two nodes. Both nodes will run on the local machine using the same genesis block and network ID. The data directories for each node will be named `node1` and `node2`. + +```shell +`mkdir node1 node2` +``` + +Each node will have an associated account that will receive some ether at launch. The following command creates an account for Node 1: + +```shell +geth --datadir node1 account new +``` + +This command returns a request for a password. Once a password has been provided the following information is returned to the terminal: + +```terminal +Your new account is locked with a password. Please give a password. Do not foget this password. +Password: +Repeat password: + +Your new key was generated + +Public address of the key: 0xC1B2c0dFD381e6aC08f34816172d6343Decbb12b +Path of the secret key file: node1/keystore/UTC--2022-05-13T14-25-49.229126160Z--c1b2c0dfd381e6ac08f34816172d6343decbb12b + +- You can share your public address with anyone. Others need it to interact with you. +- You must NEVER share the secret key with anyone! The key controls access to your funds! +- You must BACKUP your key file! Without the key, it's impossible to access account funds! +- You must remember your password! Without the password, it's impossible to decrypt the key! +``` + +The keyfile and account password should be backed up securely. These steps can then be repeated for Node 2. These commands create keyfiles that are stored in the `keystore` directory in `node1` and `node2` data directories. In order to unlock the accounts later the passwords for each account should be saved to a text file in each node's data directory. + +In each data directory save a copy of the following `genesis.json` to the top level project directory. The account addresses in the `alloc` field should be replaced with those created for each node in the previous step (without the leading `0x`). + + +```json +{ + "config": { + "chainId": 12345, + "homesteadBlock": 0, + "eip150Block": 0, + "eip155Block": 0, + "eip158Block": 0, + "byzantiumBlock": 0, + "constantinopleBlock": 0, + "petersburgBlock": 0, + "istanbulBlock": 0, + "muirGlacierBlock": 0, + "berlinBlock": 0, + "londonBlock": 0, + "arrowGlacierBlock": 0, + "grayGlacierBlock": 0, + "clique": { + "period": 5, + "epoch": 30000 + } + }, + "difficulty": "1", + "gasLimit": "800000000", + "extradata": "0x00000000000000000000000000000000000000000000000000000000000000007df9a875a174b3bc565e6424a0050ebc1b2d1d820000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", + "alloc": { + "C1B2c0dFD381e6aC08f34816172d6343Decbb12b": { "balance": "500000" }, + "c94d95a5106270775351eecfe43f97e8e75e59e8": { "balance": "500000" } + } +} + +``` + +The nodes can now be set up using `geth init` as follows: + +```shell +geth init --datadir node1 genesis.json +``` + +This should be repeated for both nodes. The following will be returned to the terminal: + +```terminal +INFO [05-13|15:41:47.520] Maximum peer count ETH=50 LES=0 total=50 +INFO [05-13|15:41:47.520] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory" +INFO [05-13|15:41:47.520] Set global gas cap cap=50,000,000 +INFO [05-13|15:41:47.520] Allocated cache and file handles database=/home/go-ethereum/node2/geth/chaindata cache=16.00MiB handles=16 +INFO [05-13|15:41:47.542] Writing custom genesis block +INFO [05-13|15:41:47.542] Persisted trie from memory database nodes=3 size=397.00B time="41.246µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B +INFO [05-13|15:41:47.543] Successfully wrote genesis state database=chaindata hash=c9a158..d415a0 +INFO [05-13|15:41:47.543] Allocated cache and file handles database=/home/go-ethereum/node2/geth/chaindata cache=16.00MiB handles=16 +INFO [05-13|15:41:47.556] Writing custom genesis block +INFO [05-13|15:41:47.557] Persisted trie from memory database nodes=3 size=397.00B time="81.801µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B +INFO [05-13|15:41:47.558] Successfully wrote genesis state database=chaindata hash=c9a158..d415a0 +``` + +The next step is to configure a bootnode. This can be any node, but for this tutorial the developer tool `bootnode` will be used to quickly and easily configure a dedicated bootnode. First the bootnode requires a key, which can be created with the following command, which will save a key to `boot.key`: + +```shell +bootnode -genkey boot.key +``` + +This key can then be used to generate a bootnode as follows: + +``` +bootnode -nodekey boot.key -addr :30305 +``` + +The choice of port passed to `-addr` is arbitrary, but public Ethereum networks use 30303, so this is best avoided. The `bootnode` command returns the following logs to the terminal, confirming that it is running: + +```terminal +enode://f7aba85ba369923bffd3438b4c8fde6b1f02b1c23ea0aac825ed7eac38e6230e5cadcf868e73b0e28710f4c9f685ca71a86a4911461637ae9ab2bd852939b77f@127.0.0.1:0?discport=30305 +Note: you're using cmd/bootnode, a developer tool. +We recommend using a regular node as bootstrap node for production deployments. +INFO [05-13|15:50:03.645] New local node record seq=1,652,453,403,645 id=a2d37f4a7d515b3a ip=nil udp=0 tcp=0 +``` + +The two nodes can now be started. Open separate terminals for each node, leaving the bootnode running in the original terminal. In each terminal, run the following command (replacing `node1` with `node2` where appropriate, and giving each node a different port ID. The account address and password file for node 1 must also be provided: + +```shell +./geth --datadir node1 --port 30306 --bootnodes enode://f7aba85ba369923bffd3438b4c8fde6b1f02b1c23ea0aac825ed7eac38e6230e5cadcf868e73b0e28710f4c9f685ca71a86a4911461637ae9ab2bd852939b77f@127.0.0.1:0?discport=30305 --networkid 123454321 --unlock 0xC1B2c0dFD381e6aC08f34816172d6343Decbb12b --password node1/password.txt +``` + +This will start the node using the bootnode as an entry point. Repeat the same command with the information appropriate to node 2. In each terminal, the following logs indicate success: + +```terminal +INFO [05-13|16:17:40.061] Maximum peer count ETH=50 LES=0 total=50 +INFO [05-13|16:17:40.061] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory" +INFO [05-13|16:17:40.061] Set global gas cap cap=50,000,000 +INFO [05-13|16:17:40.061] Allocated trie memory caches clean=154.00MiB dirty=256.00MiB +INFO [05-13|16:17:40.061] Allocated cache and file handles database=/home/go-ethereum/node1/geth/chaindata cache=512.00MiB handles=524,288 +INFO [05-13|16:17:40.094] Opened ancient database database=/home/go-ethereum/node1/geth/chaindata/ancient readonly=false +INFO [05-13|16:17:40.095] Initialised chain configuration config="{ChainID: 123454321 Homestead: 0 DAO: nil DAOSupport: false EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: nil, Muir Glacier: nil, Berlin: nil, London: nil, Arrow Glacier: nil, MergeFork: nil, Terminal TD: nil, Engine: clique}" +INFO [05-13|16:17:40.096] Initialising Ethereum protocol network=123,454,321 dbversion=8 +INFO [05-13|16:17:40.098] Loaded most recent local header number=0 hash=c9a158..d415a0 td=1 age=53y1mo2w +INFO [05-13|16:17:40.098] Loaded most recent local full block number=0 hash=c9a158..d415a0 td=1 age=53y1mo2w +INFO [05-13|16:17:40.098] Loaded most recent local fast block number=0 hash=c9a158..d415a0 td=1 age=53y1mo2w +INFO [05-13|16:17:40.099] Loaded local transaction journal transactions=0 dropped=0 +INFO [05-13|16:17:40.100] Regenerated local transaction journal transactions=0 accounts=0 +INFO [05-13|16:17:40.100] Gasprice oracle is ignoring threshold set threshold=2 +WARN [05-13|16:17:40.100] Unclean shutdown detected booted=2022-05-13T16:16:46+0100 age=54s +INFO [05-13|16:17:40.100] Starting peer-to-peer node instance=Geth/v1.10.18-unstable-8d84a701-20220503/linux-amd64/go1.18.1 +INFO [05-13|16:17:40.130] New local node record seq=1,652,454,949,228 id=f1364e6d060c4625 ip=127.0.0.1 udp=30306 tcp=30306 +INFO [05-13|16:17:40.130] Started P2P networking self=enode://87606cd0b27c9c47ca33541d4b68cf553ae6765e22800f0df340e9788912b1e3d2759b3d1933b6f739c720701a56ce26f672823084420746d04c25fc7b8c6824@127.0.0.1:30306 +INFO [05-13|16:17:40.133] IPC endpoint opened url=/home/go-ethereum/node1/geth.ipc +INFO [05-13|16:17:40.785] Unlocked account address=0xC1B2c0dFD381e6aC08f34816172d6343Decbb12b +INFO [05-13|16:17:42.636] New local node record seq=1,652,454,949,229 id=f1364e6d060c4625 ip=82.11.59.221 udp=30306 tcp=30306 +INFO [05-13|16:17:43.309] Mapped network port proto=tcp extport=30306 intport=30306 interface="UPNP IGDv1-IP1" +INFO [05-13|16:17:43.822] Mapped network port proto=udp extport=30306 intport=30306 interface="UPNP IGDv1-IP1" +[05-13|16:17:50.150] Looking for peers peercount=0 tried=0 static=0 +INFO [05-13|16:18:00.164] Looking for peers peercount=0 tried=0 static=0 +``` + +In the first terminal that is currently running the logs resembling the following will be displayed, showing the discovery process in action: + +```terminal +INFO [05-13|15:50:03.645] New local node record seq=1,652,453,403,645 id=a2d37f4a7d515b3a ip=nil udp=0 tcp=0 +TRACE[05-13|16:15:49.228] PING/v4 id=f1364e6d060c4625 addr=127.0.0.1:30306 err=nil +TRACE[05-13|16:15:49.229] PONG/v4 id=f1364e6d060c4625 addr=127.0.0.1:30306 err=nil +TRACE[05-13|16:15:49.229] PING/v4 id=f1364e6d060c4625 addr=127.0.0.1:30306 err=nil +TRACE[05-13|16:15:49.230] PONG/v4 id=f1364e6d060c4625 addr=127.0.0.1:30306 err=nil +TRACE[05-13|16:15:49.730] FINDNODE/v4 id=f1364e6d060c4625 addr=127.0.0.1:30306 err=nil +TRACE[05-13|16:15:49.731] NEIGHBORS/v4 id=f1364e6d060c4625 addr=127.0.0.1:30306 err=nil +TRACE[05-13|16:15:50.231] FINDNODE/v4 id=f1364e6d060c4625 addr=127.0.0.1:30306 err=nil +TRACE[05-13|16:15:50.231] NEIGHBORS/v4 id=f1364e6d060c4625 addr=127.0.0.1:30306 err=nil +TRACE[05-13|16:15:50.561] FINDNODE/v4 id=f1364e6d060c4625 addr=127.0.0.1:30306 err=nil +TRACE[05-13|16:15:50.561] NEIGHBORS/v4 id=f1364e6d060c4625 addr=127.0.0.1:30306 err=nil +TRACE[05-13|16:15:50.731] FINDNODE/v4 id=f1364e6d060c4625 addr=127.0.0.1:30306 err=nil +TRACE[05-13|16:15:50.731] NEIGHBORS/v4 id=f1364e6d060c4625 addr=127.0.0.1:30306 err=nil +TRACE[05-13|16:15:51.231] FINDNODE/v4 id=f1364e6d060c4625 addr=127.0.0.1:30306 err=nil +TRACE[05-13|16:15:51.232] NEIGHBORS/v4 id=f1364e6d060c4625 addr=127.0.0.1:30306 err=nil +TRACE[05-13|16:15:52.591] FINDNODE/v4 id=f1364e6d060c4625 addr=127.0.0.1:30306 err=nil +TRACE[05-13|16:15:52.591] NEIGHBORS/v4 id=f1364e6d060c4625 addr=127.0.0.1:30306 err=nil +TRACE[05-13|16:15:57.767] PING/v4 id=f1364e6d060c4625 addr=127.0.0.1:30306 err=nil +``` + +It is now possible to attach a Javascript console to either node to query the network properties: + + +```shell +geth attach node1/geth.ipc +``` + +Once the Javascript console is running, check that the node is connected to one other peer (node 2): + +```shell +net.peerCount +``` + +The details of this peer can also be queried and used to check that the peer really is Node 2: + +``` +admin.peers +``` + +This should return the following: + +```terminal +[{ + caps: ["eth/66", "snap/1"], + enode: "enode://6a4576fb12004aa13949dbf25de978102483a6521e6d5d87c5b7ccb1944bbf8995dc730303ae891732410b1dd2e684277e9292fc0a17372a789bb4e87bdf366b@127.0.0.1:30307", + id: "d300c59ba301abcb5f4a3866aab6f833857c3ddf2f0febb583410b1dc466f175", + name: "Geth/v1.10.18-unstable-8d84a701-20220503/linux-amd64/go1.18.1", + network: { + inbound: false, + localAddress: "127.0.0.1:56620", + remoteAddress: "127.0.0.1:30307", + static: false, + trusted: false + }, + protocols: { + eth: { + difficulty: 1, + head: "0xc9a158a687eff8a46128bd5b9aaf6b2f04f10f0683acbd7f031514db9ad415a0", + version: 66 + }, + snap: { + version: 1 + } + } +}] +``` + +The account associated with Node 1 was supposed to be funded with some ether at the chain genesis. This can be checked easily using `eth.getBalance()`: + +```shell +eth.getBalance(eth.accounts[0]) +``` + +This account can then be unlocked and some ether sent to Node 2, using the following commands: + + +```javascript +// unlock account +personal.unlock(eth.accounts[0]) + +// send some Wei +eth.sendTransaction({to: "0xc94d95a5106270775351eecfe43f97e8e75e59e8", from: eth.accounts[0], value: 25000}) + +//check the transaction was successful by querying Node 2's account balance +eth.getBalance("0xc94d95a5106270775351eecfe43f97e8e75e59e8") +``` + +The same steps can then be repeated to attach a console to Node 2. + + +## Summary + +This page explored the various options for configuring a local private network. A step by step guide showed how to set up and launch a private network, unlock the associated accounts, attach a console to check the network status and make some basic interactions. + + + +[gaslimit-chart]: https://etherscan.io/chart/gaslimit diff --git a/content/docs/developers/geth-developer/dev-mode.md b/content/docs/developers/geth-developer/dev-mode.md new file mode 100644 index 0000000000..472a0b946d --- /dev/null +++ b/content/docs/developers/geth-developer/dev-mode.md @@ -0,0 +1,353 @@ +--- +title: Developer mode +sort_key: B +--- + +It is often convenient for developers to work in an environment where changes to client or application software can be deployed and tested rapidly and without putting real-world users or assets at risk. For this purpose, Geth has a `--dev` flag that spins up Geth in "developer mode". This creates a single-node Ethereum test network with no connections to any external peers. It exists solely on the local machine. Starting Geth in developer mode does the following: + +- Initializes the data directory with a testing genesis block +- Sets max peers to 0 (meaning Geth does not search for peers) +- Turns off discovery by other nodes (meaning the node is invisible to other nodes) +- Sets the gas price to 0 (no cost to send transactions) +- Uses the Clique proof-of-authority consensus engine which allows blocks to be mined as-needed without excessive CPU and memory consumption +- Uses on-demand block generation, producing blocks when transactions are waiting to be mined + +This configuration enables developers to experiment with Geth's source code or develop new applications without having to sync to a pre-existing public network. Blocks are only mined when there are pending transactions. Developers can break things on this network without affecting other users. This page will demonstrate how to spin up a local Geth testnet and a simple smart contract will be deployed to it using the Remix online integrated development environment (IDE). + +## Prerequisites + +It is assumed that the user has a working Geth installation (see [installation guide](/docs/install-and-build/installing-geth)). +It would also be helpful to have basic knowledge of Geth and the Geth console. See [Getting Started](/docs/getting-started). +Some basic knowledge of [Solidity](https://docs.soliditylang.org/) and [smart contract deployment](https://ethereum.org/en/developers/tutorials/deploying-your-first-smart-contract/) would be useful. + +## Start Geth in Dev Mode + +Starting Geth in developer mode is as simple as providing the `--dev` flag. It is also possible to create a realistic block creation frequency by setting `--dev.period 13` instead of creating blocks only when transactions are pending. There are also additional configuration options required to follow this tutorial. + +First, `http` (or `ws`) must be enabled so that the Javascript console can be attached to the Geth node, and some namespaces must be specified so that certain functions can be executed from the Javascript console, specifically `eth`, `web3` and `personal`. Alternatively, Geth can be started with the `console` command. + +Finally, Remix will be used to deploy a smart contract to the node which requires information to be exchanged externally to Geth's own domain. To permit this, the `net` namespace must be enabled and the Remix URL must be provided to `--http.corsdomain`. The full command is as follows: + +```shell + +geth --dev --http --http.api eth,web3,personal,net --http.corsdomain "http://remix.ethereum.org" + +``` + +The terminal will display the following logs, confirming Geth has started successfully in developer mode: + +```terminal + + +INFO [05-09|10:49:02.951] Starting Geth in ephemeral dev mode... +INFO [05-09|10:49:02.952] Maximum peer count ETH=50 LES=0 total=50 +INFO [05-09|10:49:02.952] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory" +INFO [05-09|10:49:02.953] Set global gas cap cap=50,000,000 +INFO [05-09|10:49:03.133] Using developer account address=0x7Aa16266Ba3d309e3cb278B452b1A6307E52Fb62 +INFO [05-09|10:49:03.196] Allocated trie memory caches clean=154.00MiB dirty=256.00MiB +INFO [05-09|10:49:03.285] Writing custom genesis block +INFO [05-09|10:49:03.286] Persisted trie from memory database nodes=13 size=1.90KiB time="180.524µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B +INFO [05-09|10:49:03.287] Initialised chain configuration config="{ ChainID: 1337 Homestead: 0 DAO: nil DAOSupport: false EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 0, Muir Glacier: 0, Berlin: 0, London: 0, Arrow Glacier: nil, MergeFork: nil, Terminal TD: nil, Engine: clique}" +INFO [05-09|10:49:03.288] Initialising Ethereum protocol network=1337 dbversion= nil +INFO [05-09|10:49:03.289] Loaded most recent local header number=0 hash=c9c3de..579bb8 td=1 age=53y1mo1w +INFO [05-09|10:49:03.289] Loaded most recent local full block number=0 hash=c9c3de..579bb8 td=1 age=53y1mo1w +INFO [05-09|10:49:03.289] Loaded most recent local fast block number=0 hash=c9c3de..579bb8 td=1 age=53y1mo1w +WARN [05-09|10:49:03.289] Failed to load snapshot, regenerating err="missing or corrupted snapshot" +INFO [05-09|10:49:03.289] Rebuilding state snapshot +INFO [05-09|10:49:03.290] Resuming state snapshot generation root=ceb850..0662cb accounts=0 slots=0 storage=0.00B elapsed="778.089µs" +INFO [05-09|10:49:03.290] Regenerated local transaction journal transactions=0 accounts=0 +INFO [05-09|10:49:03.292] Gasprice oracle is ignoring threshold set threshold=2 +INFO [05-09|10:49:03.292] Generated state snapshot accounts=10 slots=0 storage=412.00B elapsed=2.418ms +WARN [05-09|10:49:03.292] Error reading unclean shutdown markers error="leveldb: not found" +INFO [05-09|10:49:03.292] Starting peer-to-peer node instance=Geth/v1.10.18-unstable-8d84a701-20220503/linux-amd64/go1.18.1 +WARN [05-09|10:49:03.292] P2P server will be useless, neither dialing nor listening +INFO [05-09|10:49:03.292] Stored checkpoint snapshot to disk number=0 hash=c9c3de..579bb8 +INFO [05-09|10:49:03.312] New local node record seq=1,652,089,743,311 id=bfedca74bea20733 ip=127.0.0.1 udp=0 tcp=0 +INFO [05-09|10:49:03.313] Started P2P networking self=enode://0544de6446dd5831daa5a391de8d0375d93ac602a95d6a182d499de31f22f75b6645c3f562932cac8328d51321b676c683471e2cf7b3c338bb6930faf6ead389@127.0.0.1:0 +INFO [05-09|10:49:03.314] IPC endpoint opened url=/tmp/geth.ipc +INFO [05-09|10:49:03.315] HTTP server started endpoint=127.0.0.1:8545 auth=false prefix= cors=http:remix.ethereum.org vhosts=localhost +INFO [05-09|10:49:03.315] Transaction pool price threshold updated price=0 +INFO [05-09|10:49:03.315] Updated mining threads threads=0 +INFO [05-09|10:49:03.315] Transaction pool price threshold updated price=1 +INFO [05-09|10:49:03.315] Etherbase automatically configured address=0x7Aa16266Ba3d309e3cb278B452b1A6307E52Fb62 +INFO [05-09|10:49:03.316] Commit new sealing work number=1 sealhash=2372a2..7fb8e7 uncles=0 txs=0 gas=0 fees=0 elapsed="202.366µs" +WARN [05-09|10:49:03.316] Block sealing failed err="sealing paused while waiting for transactions" +INFO [05-09|10:49:03.316] Commit new sealing work number=1 sealhash=2372a2..7fb8e7 uncles=0 txs=0 gas=0 fees=0 elapsed="540.054µs" + +``` + +This terminal must be left running throughout the entire tutorial. In a second terminal, attach a Javascript console: + + +```shell + +geth attach http://127.0.0.1:8545 + +``` + +The Javascript terminal will open with the following welcome message: + +```terminal + +Welcome to the Geth Javascript console! + +instance: Geth/v1.10.18-unstable-8d84a701-20220503/linux-amd64/go.1.18.1 +coinbase: 0x540dbaeb2390f2eb005f7a6dbf3436a0959197a9 +at block: 0 (Thu Jan 01 1970 01:00:00 GMT+0100 (BST)) + modules: eth:1.0 personal:1.0 rpc:1.0 web3:1.0 + +To exit, press ctrl-d or type exit +> + +``` + +In the [Getting Started](/docs/getting-started/) tutorial it was explained that using the external signing and account management tool, Clef, was best practise for generating and securing user accounts. However, for simplicity this tutorial will use Geth's built-in account management. First, the existing accounts can be displayed using `eth.accounts`: + +```shell + +eth.accounts + +``` + +An array containing a single address will be displayed in the terminal, despite no accounts having yet been explicitly created. This is the "coinbase" account. The coinbase address is the recipient of the total amount of ether created at the local network genesis. Querying the ether balance of the coinbase account will return a very large number. The coinbase account can be invoked as `eth.accounts[0]` or as `eth.coinbase`: + +```terminal + +> eth.coinbase==eth.accounts[0] +true + +``` + +The following command can be used to query the balance. The return value is in units of Wei, which is divided by 118 to give units of ether. This can be done explicitly or by calling the `web3.FromWei()` function: + +```shell + +eth.getBalance(eth.coinbase)/1e18 + +// or + +web3.fromWei(eth.getBalance(eth.coinbase)) + +``` + +Using `web3.fromWei()` is less error prone because the correct multiplier is built in. These commands both return the following: + +```terminal + +1.157920892373162e+59 + +``` + +A new account can be created and some of the ether from the coinbase transferred across to it. A new account is generated using the `newAccount` function in the `personal` namespace: + +```shell + +personal.newAccount() + +``` + +The terminal will display a request for a password, twice. Once provided, a new account will be created and its address printed to the terminal. The account creation is also logged in the Geth terminal, including the location of the keyfile in the keystore. It is a good idea to back up the password somewhere at this point. If this were an account on a live network, intended to own assets of real-world value, it would be critical to back up the account password and the keystore in a secure manner. + +To reconfirm the account creation, running `eth.accounts` in the Javascript console should display an array containing two account addresses, one being the coinbase and the other being the newly generated address. The following command transfers 50 ETH from the coinbase to the new account: + +```shell + +eth.sendTransaction({from: eth.coinbase, to: eth.accounts[1], value: web3.toWei(50, "ether")}) + +``` + +A transaction hash will be returned to the console. This transaction hash will also be displayed in the logs in the Geth console, followed by logs confirming that a new block was mined (remember in the local development network blocks are mined when transactions are pending). The transaction details can be displayed in the Javascript console by passing the transaction hash to `eth.getTransaction()`: + +```shell + +eth.getTransaction("0x62044d2cab405388891ee6d53747817f34c0f00341cde548c0ce9834e9718f27") + +``` + +The transaction details are displayed as follows: + +```terminal + +{ + accessList: [], + blockHash: "0xdef68762539ebfb247e31d749acc26ab5df3163aabf9d450b6001c200d17da8a", + blockNumber: 1, + chainId: "0x539", + from: "0x540dbaeb2390f2eb005f7a6dbf3436a0959197a9", + gas: 21000, + gasPrice: 875000001, + hash: "0x2326887390dc04483d435a6303dc05bd2648086eab15f24d7dcdf8c26e8af4b8", + input: "0x", + maxFeePerGas: 2000000001, + maxPriorityFeePerGas: 1, + nonce: 0, + r: "0x3f7b54f095b837ec13480eab5ac7de082465fc79f43b813cca051394dd028d5d", + s: "0x167ef271ae8175239dccdd970db85e06a044d5039252d6232d0954d803bb4e3e", + to: "0x43e3a14fb8c68caa7eea95a02693759de4513950", + transactionIndex: 0, + type: "0x2", + v: "0x0", + value: 50000000000000000000 +} + + +``` + +Now that the user account is funded with ether, a contract can be created ready to deploy to the Geth node. + + +## A simple smart contract + +This tutorial will make use of a classic example smart contract, `Storage.sol`. This contract exposes two public functions, one to add a value to the contract storage and one to view the stored value. The contract, written in Solidity, is provided below: + + +```Solidity + +pragma solidity >=0.7.0; + +contract Storage{ + + + uint256 number; + + function store(uint256 num) public{ + + number = num; + } + + function retrieve() public view returns (uint256){ + return number; + + } +} + +``` + +Solidity is a high-level language that makes code executable by the Ethereum virtual machine (EVM) readable to humans. This means that there is an intermediate step between writing code in Solidity and deploying it to Ethereum. This step is called "compilation" and it converts human-readable code into EVM-executable byte-code. This byte-code is then included in a transaction sent from the Geth node during contract deployment. This can all be done directly from the Geth Javascript console; however this tutorial uses an online IDE called Remix to handle the compilation and deployment of the contract to the local Geth node. + + + +## Compile and deploy using Remix + +In a web browser, open . This opens an online smart contract development environment. On the left-hand side of the screen there is a side-bar menu that toggles between several toolboxes that are displayed in a vertical panel. On the right hand side of the screen there is an editor and a terminal. This layout is similar to the default layout of many other IDEs such as [VSCode](https://code.visualstudio.com/). The contract defined in the previous section, `Storage.sol` is already available in the `Contracts` directory in Remix. It can be opened and reviewed in the editor. + +![Remix](/static/images/remix.png) + + +The Solidity logo is present as an icon in the Remix side-bar. Clicking this icon opens the Solidity compiler wizard. This can be used to compile `Storage.sol` ready. With `Solidity.sol` open in the editor window, simply click the `Compile 1_Storage.sol` button. A green tick will appear next to the Solidity icon to confirm that the contract has compiled successfully. This means the contract bytecode is available. + +![Remix-compiler](/static/images/remix-compiler.png) + +Below the Solidity icon is a fourth icon that includes the Ethereum logo. Clicking this opens the Deploy menu. In this menu, Remix can be configured to connect to the local Geth node. In the drop-down menu labelled `ENVIRONMENT`, select `Injected Web3`. This will open an information pop-up with instructions for configuring Geth - these can be ignored as they were completed earlier in this tutorial. However, at the bottom of this pop-up is a box labelled `Web3 Provider Endpoint`. This should be set to Geth's 8545 port on `localhost` (`127.0.0.1:8545`). Click OK. The `ACCOUNT` field should automatically populate with the address of the account created earlier using the Geth Javascript console. + + +![Remix-deploy](/static/images/remix-deploy.png) + + +To deploy `Storage.sol`, click `DEPLOY`. + +The following logs in the Geth terminal confirm that the contract was successfully deployed. + + +```terminal + +INFO [05-09|12:27:09.680] Setting new local account address=0x7Aa16266Ba3d309e3cb278B452b1A6307E52Fb62 +INFO [05-09|12:27:09.680] Submitted contract creation hash=0xbf2d2d1c393a882ffb6c90e6d1713906fd799651ae683237223b897d4781c4f2 from=0x7Aa16266Ba3d309e3cb278B452b1A6307E52Fb62 nonce=1 contract=0x4aA11DdfD817dD70e9FF2A2bf9c0306e8EC450d3 value=0 +INFO [05-09|12:27:09.681] Commit new sealing work number=2 sealhash=845a53..f22818 uncles=0 txs=1 gas=125,677 fees=0.0003141925 elapsed="335.991µs" +INFO [05-09|12:27:09.681] Successfully sealed new block number=2 sealhash=845a53..f22818 hash=e927bc..f2c8ed elapsed="703.415µs" +INFO [05-09|12:27:09.681] 🔨 mined potential block number=2 hash=e927bc..f2c8ed + +``` + +## Interact with contract using Remix + +The contract is now deployed on a local testnet version of the Etheruem blockchain. This means there is a contract address that contains executable bytecode that can be invoked by sending transactions with instructions, also in bytecode, to that address. Again, this can all be achieved by constructing transactions directly in the Geth console or even by making external http requests using tools such as Curl. Here, Remix is used to retrieve the value, then the same action is taken using the Javascript console. + +After deploying the contract in Remix, the `Deployed Contracts` tab in the sidebar automatically populates with the public functions exposed by `Storage.sol`. To send a value to the contract storage, type a number in the field adjacent to the `store` button, then click the button. + +![Remix-func](/static/images/remix-func.png) + +In the Geth terminal, the following logs confirm that the transaction was successful (the actual values will vary from the example below): + +```terminal + +INFO [05-09|13:41:58.644] Submitted transaction hash=0xfa3cd8df6841c5d3706d3bacfb881d2b985d0b55bdba440f1fdafa4ed5b5cc31 from=0x7Aa16266Ba3d309e3cb278B452b1A6307E52Fb62 nonce=2 recipient=0x4aA11DdfD817dD70e9FF2A2bf9c0306e8EC450d3 value=0 +INFO [05-09|13:41:58.644] Commit new sealing work number=3 sealhash=5442e3..f49739 uncles=0 txs=1 gas=43724 fees=0.00010931 elapsed="334.446µs" +INFO [05-09|13:41:58.645] Successfully sealed new block number=3 sealhash=5442e3..f49739 hash=c076c8..eeee77 elapsed="581.374µs" +INFO [05-09|13:41:58.645] 🔨 mined potential block number=3 hash=c076c8..eeee77 + +``` + +The transaction hash can be used to retrieve the transaction details using the Geth Javascript console, which will return the following information: + +```terminal + +{ + accessList: [], + blockHash: "0xc076c88200618f4cbbfb4fe7c3eb8d93566724755acc6c4e9a355cc090eeee77", + blockNumber: 3, + chainId: "0x539", + from: "0x7aa16266ba3d309e3cb278b452b1a6307e52fb62", + gas: 43724, + gasPrice: 3172359839, + hash: "0xfa3cd8df6841c5d3706d3bacfb881d2b985d0b55bdba440f1fdafa4ed5b5cc31", + input: "0x6057361d0000000000000000000000000000000000000000000000000000000000000038", + maxFeePerGas: 4032048134, + maxPriorityFeePerGas: 2500000000, + nonce: 2, + r: "0x859b88062715c5d66b9a188886ad51b68a1e4938d5932ce2dac874c104d2b26", + s: "0x61ef6bc454d5e6a76c414f133aeb6321197a61e263a3e270a16bd4a65d94da55", + to: "0x4aa11ddfd817dd70e9ff2a2bf9c0306e8ec450d3", + transactionIndex: 0, + type: "0x2", + v: "0x1", + value: 0 +} + +``` + +The `from` address is the account that sent the transaction, the `to` address is the deployment address of the contract. The value entered into Remix is now in storage at that contract address. This can be retrieved using Remix by calling the `retrieve` function - to do this simply click the `retrieve` button. Alternatively, it can be retrieved using `web3.getStorageAt` using the Geth Javascript console. The following command returns the value in the contract storage (replace the given address with the correct one displayed in the Geth logs). + +```shell + +web3.eth.getStorageAt("0x407d73d8a49eeb85d32cf465507dd71d507100c1", 0) + +``` + +This returns a value that looks like the following: + +```terminal + +"0x000000000000000000000000000000000000000000000000000000000000000038" + +``` + + +The returned value is a left-padded hexadecimal value. For example, the return value `0x000000000000000000000000000000000000000000000000000000000000000038` corresponds to a value of `56` entered as a uint256 to Remix. After converting from hexadecimal string to decimal number the returned value should be equal to that provided to Remix in the previous step. + +## Reusing --datadir + +This tutorial used an ephemeral blockchain that is completely destroyed and started afresh during each dev-mode session. However, it is also possible to create persistent blockchain and account data that can be reused across multiple sessions. This is done by providing the `--datadir` flag and a directory name when starting Geth in dev-mode. + +```shell + +geth --datadir dev-chain --dev --http --http.api personal,web3,eth,net --http.corsdomain "remix.ethereum.org" + +``` + +## Re-using accounts + +Geth will fail to start in dev-mode if keys have been manually created or imported into the keystore in the `--datadir` directory. This is because the account cannot be automatically unlocked. To resolve this issue, the password defined when the account was created can be saved to a text file and its path passed to the `--password` flag on starting Geth, for example if `password.txt` is saved in the top-level `go-ethereum` directory: + +```shell + +geth --datadir dev-chain --dev --http --http.api personal,web3,eth,net --http.corsdomain "remix.ethereum.org" --password password.txt + +``` +**Note** that this is an edge-case that applies when both the `--datadir` and `--dev` flags are used and a key has been manually created or imported into the keystore. + + + +## Summary + +This tutorial has demonstrated how to spin up a local developer network using Geth. Having started this development network, a simple contract was deployed to the developer network. Then, Remix was connected to the local Geth node and used to deploy and interact with a contract. Remix was used to add a value to the contract storage and then the value was retrieved using Remix and also using the lower level commands in the Javascript console. diff --git a/content/docs/developers/geth-developer/devguide.md b/content/docs/developers/geth-developer/devguide.md new file mode 100644 index 0000000000..171016a486 --- /dev/null +++ b/content/docs/developers/geth-developer/devguide.md @@ -0,0 +1,122 @@ +--- +title: Developer Guide +sort_key: A +--- + +**NOTE: These instructions are for people who want to contribute Go source code changes. +If you just want to run ethereum, use the regular [Installation Instructions][install-guide].** + +This document is the entry point for developers of the Go implementation of Ethereum. +Developers here refer to the hands-on: who are interested in build, develop, debug, submit +a bug report or pull request or contribute code to go-ethereum. + +## Contributing + +Thank you for considering to help out with the source code! We welcome contributions from +anyone on the internet, and are grateful for even the smallest of fixes! + +GitHub is used to track issues and contribute code, suggestions, feature requests or +documentation. + +If you'd like to contribute to go-ethereum, please fork, fix, commit and send a pull +request (PR) for the maintainers to review and merge into the main code base. If you wish +to submit more complex changes though, please check up with the core devs in the +go-ethereum [Discord Server][discord]. to ensure those changes are in line with the +general philosophy of the project and/or get some early feedback. This can reduce your +effort as well as speeding up our review and merge procedures. + +PRs need to be based on and opened against the `master` branch (unless by explicit +agreement, you contribute to a complex feature branch). + +Your PR will be reviewed according to the [Code Review guidelines][code-review]. + +We encourage a PR early approach, meaning you create the PR the earliest even without the +fix/feature. This will let core devs and other volunteers know you picked up an issue. +These early PRs should indicate 'in progress' status. + +## Building and Testing + +We assume that you have Go installed. Please use Go version 1.13 or later. We use the go +toolchain for development, which you can get from the [Go downloads page][go-install]. + +go-ethereum is a Go module, and uses the [Go modules system][go-modules] to manage +dependencies. Using `GOPATH` is not required to build go-ethereum. + +### Building Executables + +Switch to the go-ethereum repository root directory. + +You can build all code using the go tool, placing the resulting binary in `$GOPATH/bin`. + +```text +go install -v ./... +``` + +go-ethereum exectuables can be built individually. To build just geth, use: + +```text +go install -v ./cmd/geth +``` + +If you want to compile geth for an architecture that differs from your host, please +consult our [cross compilation guide][cross-compile]. + +### Testing + +Testing a package: + +``` +go test -v ./eth +``` + +Running an individual test: + +``` +go test -v ./eth -run TestMethod +``` + +**Note**: here all tests with prefix _TestMethod_ will be run, so if you got TestMethod, +TestMethod1, then both tests will run. + +Running benchmarks, eg.: + +``` +go test -v -bench . -run BenchmarkJoin +``` + +For more information, see the [go test flags][testflag] documentation. + +### Getting Stack Traces + +If `geth` is started with the `--pprof` option, a debugging HTTP server is made available +on port 6060. You can bring up to see the heap, +running routines etc. By clicking "full goroutine stack dump" you can generate a trace +that is useful for debugging. + +Note that if you run multiple instances of `geth`, this port will only work for the first +instance that was launched. If you want to generate stacktraces for these other instances, +you need to start them up choosing an alternative pprof port. Make sure you are +redirecting stderr to a logfile. + +``` +geth -port=30300 -verbosity 5 --pprof --pprof.port 6060 2>> /tmp/00.glog +geth -port=30301 -verbosity 5 --pprof --pprof.port 6061 2>> /tmp/01.glog +geth -port=30302 -verbosity 5 --pprof --pprof.port 6062 2>> /tmp/02.glog +``` + +Alternatively if you want to kill the clients (in case they hang or stalled syncing, etc) +and have the stacktrace too, you can use the `-QUIT` signal with `kill`: + +``` +killall -QUIT geth +``` + +This will dump stack traces for each instance to their respective log file. + +[install-guide]: ../install-and-build/installing-geth +[code-review]: ../developers/code-review-guidelines +[cross-compile]: ../install-and-build/cross-compile +[go-modules]: https://github.com/golang/go/wiki/Modules +[discord]: https://discord.gg/invite/nthXNEv +[go-install]: https://golang.org/doc/install +[testflag]: https://golang.org/cmd/go/#hdr-Testing_flags diff --git a/content/docs/developers/geth-developer/dns-discovery-setup.md b/content/docs/developers/geth-developer/dns-discovery-setup.md new file mode 100644 index 0000000000..715838eea8 --- /dev/null +++ b/content/docs/developers/geth-developer/dns-discovery-setup.md @@ -0,0 +1,125 @@ +--- +title: DNS Discovery Setup Guide +sort_key: C +--- + +This document explains how to set up an [EIP 1459][dns-eip] node list using the devp2p +developer tool. The focus of this guide is creating a public list for the Ethereum mainnet +and public testnets, but you may also find this helpful if you want to set up DNS-based +discovery for a private network. + +DNS-based node lists can serve as a fallback option when connectivity to the discovery DHT +is unavailable. In this guide, we'll create node lists by crawling the discovery DHT, then +publishing the resulting node sets under chosen DNS names. + +### Installing the devp2p command + +cmd/devp2p is a developer utility and is not included in the Geth distribution. You can +install this command using `go get`: + +```shell +go get github.com/ethereum/go-ethereum/cmd/devp2p +``` + +To create a signing key, you might also need the `ethkey` utility. + +```shell +go get github.com/ethereum/go-ethereum/cmd/ethkey +``` + +### Crawling the v4 DHT + +Our first step is to compile a list of all reachable nodes. The DHT crawler in cmd/devp2p +is a batch process which runs for a set amount of time. You should should schedule this command +to run at a regular interval. To create a node list, run + +```shell +devp2p discv4 crawl -timeout 30m all-nodes.json +``` + +This walks the DHT and stores the set of all found nodes in the `all-nodes.json` file. +Subsequent runs of the same command will revalidate previously discovered node records, +add newly-found nodes to the set, and remove nodes which are no longer alive. The quality +of the node set improves with each run because the number of revalidations is tracked +alongside each node in the set. + +### Creating sub-lists through filtering + +Once `all-nodes.json` has been created and the set contains a sizeable number of nodes, +useful sub-sets of nodes can be extracted using the `devp2p nodeset filter` command. This +command takes a node set file as argument and applies filters given as command-line flags. + +To create a filtered node set, first create a new directory to hold the output set. You +can use any directory name, though it's good practice to use the DNS domain name as the +name of this directory. + +```shell +mkdir mainnet.nodes.example.org +``` + +Then, to create the output set containing Ethereum mainnet nodes only, run + +```shell +devp2p nodeset filter all-nodes.json -eth-network mainnet > mainnet.nodes.example.org/nodes.json +``` + +The following filter flags are available: + +* `-eth-network ( mainnet | ropsten | rinkeby | goerli )` selects an Ethereum network. +* `-les-server` selects LES server nodes. +* `-ip ` restricts nodes to the given IP range. +* `-min-age ` restricts the result to nodes which have been live for the + given duration. + +### Creating DNS trees + +To turn a node list into a DNS node tree, the list needs to be signed. To do this, you +need a key pair. To create the key file in the correct format, you can use the cmd/ethkey +utility. Please choose a good password to encrypt the key on disk. + +```shell +ethkey generate dnskey.json +``` + +Now use `devp2p dns sign` to update the signature of the node list. If your list's +directory name differs from the name you want to publish it at, please specify the DNS +name the using the `-domain` flag. This command will prompt for the key file password and +update the tree signature. + +```shell +devp2p dns sign mainnet.nodes.example.org dnskey.json +``` + +The resulting DNS tree metadata is stored in the +`mainnet.nodes.example.org/enrtree-info.json` file. + +### Publishing DNS trees + +Now that the tree is signed, it can be published to a DNS provider. cmd/devp2p currently +supports publishing to CloudFlare DNS and Amazon Route53. You can also export TXT records +as a JSON file and publish them yourself. + +To publish to CloudFlare, first create an API token in the management console. cmd/devp2p +expects the API token in the `CLOUDFLARE_API_TOKEN` environment variable. Now use the +following command to upload DNS TXT records via the CloudFlare API: + +```shell +devp2p dns to-cloudflare mainnet.nodes.example.org +``` + +Note that this command uses the domain name specified during signing. Any existing records +below this name will be erased by cmd/devp2p. + +### Using DNS trees with Geth + +Once your tree is available through a DNS name, you can tell geth to use it with the +`--discovery.dns` command line flag. Node trees are referenced using the `enrtree://` URL +scheme. You can find the URL of your tree in the `enrtree-info.json` file created by +`devp2p dns sign`. Just pass the URL as an argument to the flag in order to make use of +the published tree. + +```shell +geth --discovery.dns "enrtree://AMBMWDM3J6UY3M32TMMROUNLX6Y3YTLVC3DC6HN2AVG5NHNSAXDW6@mainnet.nodes.example.org" +``` + +[dns-eip]: https://eips.ethereum.org/EIPS/eip-1459 diff --git a/content/docs/developers/geth-developer/issue-handling-workflow.md b/content/docs/developers/geth-developer/issue-handling-workflow.md new file mode 100644 index 0000000000..9fb2bdc180 --- /dev/null +++ b/content/docs/developers/geth-developer/issue-handling-workflow.md @@ -0,0 +1,57 @@ +--- +title: Issue Handling Workflow +sort_key: B +--- + +### (Draft proposal) + +Keep the number of open issues under 820 + +Keep the ratio of open issues per all issues under 13% + +Have 50 issues labelled [help wanted](https://github.com/ethereum/go-ethereum/labels/help%20wanted) and 50 [good first issue](https://github.com/ethereum/go-ethereum/labels/good%20first%20issue). + +Use structured labels of the form `: