Merge pull request #4821 from Munkybooty/backports-0.20

backport: v0.20 pr1

Author: UdjinM6

.travis.yml

@@ -202,28 +202,6 @@ after_success:
       script:
         - set -o errexit; source ./ci/extended_lint/06_script.sh
 
-    - stage: extended-lint
-      name: 'lint macOS 10.12 (compat)'
-      os: osx
-      # Use the earliest macOS that can build our lint dependencies:
-      # Xcode 8.3.3, macOS 10.12, JDK 1.8.0_112-b16
-      # https://docs.travis-ci.com/user/reference/osx/#OS-X-Version
-      osx_image: xcode8.3
-      # TODO: if you're updating osx_image, try using "rvm:" to supply the
-      # version of ruby required by homebrew. Despite this "rvm:" declaration,
-      # brew update installs ruby 2.3.7 as its first action.
-      language: ruby
-      rvm:
-        - 2.3.7
-      env:
-      cache: false
-      install:
-        - set -o errexit; source ./ci/lint/04_install.sh
-      before_script:
-        - set -o errexit; source ./ci/lint/05_before_script.sh
-      script:
-        - set -o errexit; source ./ci/lint/06_script.sh
-
     - stage: test
       name: 'ARM  [GOAL: install]  [unit tests, no functional tests]'
       env: >-

ci/lint/04_install.sh

@@ -6,24 +6,11 @@
 
 export LC_ALL=C
 
-if [ "$TRAVIS_OS_NAME" == "osx" ]; then
-  # update first to install required ruby dependency
-  travis_retry brew update
-  travis_retry brew reinstall git -- --with-pcre2 # for  --perl-regexp
-  travis_retry brew install grep # gnu grep for --perl-regexp support
-  PATH="$(brew --prefix grep)/libexec/gnubin:$PATH"
-  travis_retry brew install shellcheck
-  travis_retry brew upgrade python
-  PATH="$(brew --prefix python)/bin:$PATH"
-  export PATH
-else
-  SHELLCHECK_VERSION=v0.7.1
-  curl -sL "https://github.com/koalaman/shellcheck/releases/download/${SHELLCHECK_VERSION}/shellcheck-${SHELLCHECK_VERSION}.linux.x86_64.tar.xz" | tar --xz -xf - --directory /tmp/
-  PATH="/tmp/shellcheck-${SHELLCHECK_VERSION}:${PATH}"
-  export PATH
-fi
-
 travis_retry pip3 install codespell==1.17.1
 travis_retry pip3 install flake8==3.8.3
 travis_retry pip3 install vulture==2.3
 travis_retry pip3 install yq
+
+SHELLCHECK_VERSION=v0.6.0
+curl -s "https://storage.googleapis.com/shellcheck/shellcheck-${SHELLCHECK_VERSION}.linux.x86_64.tar.xz" | tar --xz -xf - --directory /tmp/
+export PATH="/tmp/shellcheck-${SHELLCHECK_VERSION}:${PATH}"

doc/benchmarking.md

@@ -11,7 +11,7 @@ Running
 For benchmarks purposes you only need to compile `dash_bench`. Beware of configuring without `--enable-debug` as this would impact
 benchmarking by unlatching log printers and lock analysis.
 
-    make -C src bench_dash
+    make -C src dash_bench
 
 After compiling Dash Core, the benchmarks can be run with:
 

test/README.md

@@ -100,7 +100,7 @@ By default, up to 4 tests will be run in parallel by test_runner. To specify
 how many jobs to run, append `--jobs=n`
 
 The individual tests and the test_runner harness have many command-line
-options. Run `test_runner.py -h` to see them all.
+options. Run `test/functional/test_runner.py -h` to see them all.
 
 #### Troubleshooting and debugging test failures
 
@@ -113,7 +113,7 @@ killed all its dashd nodes), then there may be a port conflict which will
 cause the test to fail. It is recommended that you run the tests on a system
 where no other dashd processes are running.
 
-On linux, the test_framework will warn if there is another
+On linux, the test framework will warn if there is another
 dashd process running when the tests are started.
 
 If there are zombie dashd processes after test failure, you can kill them
@@ -142,7 +142,7 @@ tests will fail. If this happens, remove the cache directory (and make
 sure dashd processes are stopped as above):
 
 ```bash
-rm -rf cache
+rm -rf test/cache
 killall dashd
 ```
 
@@ -161,6 +161,15 @@ levels using the logger included in the test_framework, e.g.
   fails, the `test_framework.log` and dashd `debug.log`s will all be dumped
   to the console to help troubleshooting.
 
+These log files can be located under the test data directory (which is always
+printed in the first line of test output):
+  - `<test data directory>/test_framework.log`
+  - `<test data directory>/node<node number>/regtest/debug.log`.
+
+The node number identifies the relevant test node, starting from `node0`, which
+corresponds to its position in the nodes list of the specific test,
+e.g. `self.nodes[0]`.
+
 To change the level of logs output to the console, use the `-l` command line
 argument.
 
@@ -169,7 +178,7 @@ aggregate log by running the `combine_logs.py` script. The output can be plain
 text, colorized text or html. For example:
 
 ```
-combine_logs.py -c <test data directory> | less -r
+test/functional/combine_logs.py -c <test data directory> | less -r
 ```
 
 will pipe the colorized logs from the test into less.

test/functional/README.md

@@ -4,13 +4,13 @@
 
 #### Example test
 
-The [example_test.py](example_test.py) is a heavily commented example of a test case that uses both
-the RPC and P2P interfaces. If you are writing your first test, copy that file
-and modify to fit your needs.
+The file [test/functional/example_test.py](example_test.py) is a heavily commented example
+of a test case that uses both the RPC and P2P interfaces. If you are writing your first test, copy
+that file and modify to fit your needs.
 
 #### Coverage
 
-Running `test_runner.py` with the `--coverage` argument tracks which RPCs are
+Running `test/functional/test_runner.py` with the `--coverage` argument tracks which RPCs are
 called by the tests and prints a report of uncovered RPCs in the summary. This
 can be used (along with the `--extended` argument) to find out which RPCs we
 don't have test cases for.
@@ -82,7 +82,7 @@ P2P messages. These can be found in the following source files:
 
 #### Using the P2P interface
 
-- `messages.py` contains all the definitions for objects that pass
+- [messages.py](test_framework/messages.py) contains all the definitions for objects that pass
 over the network (`CBlock`, `CTransaction`, etc, along with the network-level
 wrappers for them, `msg_block`, `msg_tx`, etc).
 
@@ -96,29 +96,32 @@ the Bitcoin Core node application logic. For custom behaviour, subclass the
 P2PInterface object and override the callback methods.
 
 - Can be used to write tests where specific P2P protocol behavior is tested.
-Examples tests are `p2p_unrequested_blocks.py`, `p2p_compactblocks.py`.
+Examples tests are [p2p_unrequested_blocks.py](p2p_unrequested_blocks.py),
+[p2p_compactblocks.py](p2p_compactblocks.py).
 
-### test-framework modules
+### Test framework modules
+The following are useful modules for test developers. They are located in
+[test/functional/test_framework/](test_framework).
 
-#### [test_framework/authproxy.py](test_framework/authproxy.py)
+#### [authproxy.py](test_framework/authproxy.py)
 Taken from the [python-bitcoinrpc repository](https://github.com/jgarzik/python-bitcoinrpc).
 
-#### [test_framework/test_framework.py](test_framework/test_framework.py)
+#### [test_framework.py](test_framework/test_framework.py)
 Base class for functional tests.
 
-#### [test_framework/util.py](test_framework/util.py)
+#### [util.py](test_framework/util.py)
 Generally useful functions.
 
-#### [test_framework/mininode.py](test_framework/mininode.py)
+#### [mininode.py](test_framework/mininode.py)
 Basic code to support P2P connectivity to a dashd.
 
-#### [test_framework/script.py](test_framework/script.py)
+#### [script.py](test_framework/script.py)
 Utilities for manipulating transaction scripts (originally from python-bitcoinlib)
 
-#### [test_framework/key.py](test_framework/key.py)
+#### [key.py](test_framework/key.py)
 Test-only secp256k1 elliptic curve implementation
 
-#### [test_framework/blocktools.py](test_framework/blocktools.py)
+#### [blocktools.py](test_framework/blocktools.py)
 Helper functions for creating blocks and transactions.
 
 ### Benchmarking with perf

test/functional/feature_bip68_sequence.py

@@ -6,9 +6,15 @@
 
 from test_framework.blocktools import create_block, create_coinbase
 from test_framework.messages import COIN, COutPoint, CTransaction, CTxIn, CTxOut, FromHex, ToHex
-from test_framework.script import CScript
 from test_framework.test_framework import BitcoinTestFramework
-from test_framework.util import assert_equal, assert_greater_than, assert_raises_rpc_error, get_bip9_status, satoshi_round
+from test_framework.util import (
+    assert_equal,
+    assert_greater_than,
+    assert_raises_rpc_error,
+    get_bip9_status,
+    satoshi_round
+)
+from test_framework.script_util import DUMMY_P2SH_SCRIPT
 
 SEQUENCE_LOCKTIME_DISABLE_FLAG = (1<<31)
 SEQUENCE_LOCKTIME_TYPE_FLAG = (1<<22) # this means use time (0 means height)
@@ -76,7 +82,7 @@ def test_disable_flag(self):
         # input to mature.
         sequence_value = SEQUENCE_LOCKTIME_DISABLE_FLAG | 1
         tx1.vin = [CTxIn(COutPoint(int(utxo["txid"], 16), utxo["vout"]), nSequence=sequence_value)]
-        tx1.vout = [CTxOut(value, CScript([b'a']))]
+        tx1.vout = [CTxOut(value, DUMMY_P2SH_SCRIPT)]
 
         tx1_signed = self.nodes[0].signrawtransactionwithwallet(ToHex(tx1))["hex"]
         tx1_id = self.nodes[0].sendrawtransaction(tx1_signed)
@@ -88,7 +94,7 @@ def test_disable_flag(self):
         tx2.nVersion = 2
         sequence_value = sequence_value & 0x7fffffff
         tx2.vin = [CTxIn(COutPoint(tx1_id, 0), nSequence=sequence_value)]
-        tx2.vout = [CTxOut(int(value - self.relayfee * COIN), CScript([b'a' * 35]))]
+        tx2.vout = [CTxOut(int(value - self.relayfee * COIN), DUMMY_P2SH_SCRIPT)]
         tx2.rehash()
 
         assert_raises_rpc_error(-26, NOT_FINAL_ERROR, self.nodes[0].sendrawtransaction, ToHex(tx2))
@@ -183,7 +189,7 @@ def test_sequence_lock_confirmed_inputs(self):
                 value += utxos[j]["amount"]*COIN
             # Overestimate the size of the tx - signatures should be less than 120 bytes, and leave 50 for the output
             tx_size = len(ToHex(tx))//2 + 120*num_inputs + 50
-            tx.vout.append(CTxOut(int(value-self.relayfee*tx_size*COIN/1000), CScript([b'a'])))
+            tx.vout.append(CTxOut(int(value-self.relayfee*tx_size*COIN/1000), DUMMY_P2SH_SCRIPT))
             rawtx = self.nodes[0].signrawtransactionwithwallet(ToHex(tx))["hex"]
 
             if (using_sequence_locks and not should_pass):
@@ -212,7 +218,7 @@ def test_sequence_lock_unconfirmed_inputs(self):
         tx2 = CTransaction()
         tx2.nVersion = 2
         tx2.vin = [CTxIn(COutPoint(tx1.sha256, 0), nSequence=0)]
-        tx2.vout = [CTxOut(int(tx1.vout[0].nValue - self.relayfee*COIN), CScript([b'a']))]
+        tx2.vout = [CTxOut(int(tx1.vout[0].nValue - self.relayfee*COIN), DUMMY_P2SH_SCRIPT)]
         tx2_raw = self.nodes[0].signrawtransactionwithwallet(ToHex(tx2))["hex"]
         tx2 = FromHex(tx2, tx2_raw)
         tx2.rehash()
@@ -230,7 +236,7 @@ def test_nonzero_locks(orig_tx, node, relayfee, use_height_lock):
             tx = CTransaction()
             tx.nVersion = 2
             tx.vin = [CTxIn(COutPoint(orig_tx.sha256, 0), nSequence=sequence_value)]
-            tx.vout = [CTxOut(int(orig_tx.vout[0].nValue - relayfee * COIN), CScript([b'a' * 35]))]
+            tx.vout = [CTxOut(int(orig_tx.vout[0].nValue - relayfee * COIN), DUMMY_P2SH_SCRIPT)]
             tx.rehash()
 
             if (orig_tx.hash in node.getrawmempool()):
@@ -343,7 +349,7 @@ def test_bip68_not_consensus(self):
         tx2 = CTransaction()
         tx2.nVersion = 1
         tx2.vin = [CTxIn(COutPoint(tx1.sha256, 0), nSequence=0)]
-        tx2.vout = [CTxOut(int(tx1.vout[0].nValue - self.relayfee*COIN), CScript([b'a']))]
+        tx2.vout = [CTxOut(int(tx1.vout[0].nValue - self.relayfee*COIN), DUMMY_P2SH_SCRIPT)]
 
         # sign tx2
         tx2_raw = self.nodes[0].signrawtransactionwithwallet(ToHex(tx2))["hex"]
@@ -358,7 +364,7 @@ def test_bip68_not_consensus(self):
         tx3 = CTransaction()
         tx3.nVersion = 2
         tx3.vin = [CTxIn(COutPoint(tx2.sha256, 0), nSequence=sequence_value)]
-        tx3.vout = [CTxOut(int(tx2.vout[0].nValue - self.relayfee * COIN), CScript([b'a' * 35]))]
+        tx3.vout = [CTxOut(int(tx2.vout[0].nValue - self.relayfee * COIN), DUMMY_P2SH_SCRIPT)]
         tx3.rehash()
 
         assert_raises_rpc_error(-26, NOT_FINAL_ERROR, self.nodes[0].sendrawtransaction, ToHex(tx3))

test/functional/feature_loadblock.py

@@ -0,0 +1,84 @@
+#!/usr/bin/env python3
+# Copyright (c) 2017-2019 The Bitcoin Core developers
+# Distributed under the MIT software license, see the accompanying
+# file COPYING or http://www.opensource.org/licenses/mit-license.php.
+"""Test loadblock option
+
+Test the option to start a node with the option loadblock which loads
+a serialized blockchain from a file (usually called bootstrap.dat).
+To generate that file this test uses the helper scripts available
+in contrib/linearize.
+"""
+
+import os
+import subprocess
+import sys
+import tempfile
+import urllib
+
+from test_framework.test_framework import (
+    BitcoinTestFramework,
+)
+from test_framework.util import assert_equal, wait_until
+
+
+class LoadblockTest(BitcoinTestFramework):
+    def set_test_params(self):
+        self.setup_clean_chain = True
+        self.num_nodes = 2
+
+    def run_test(self):
+        self.nodes[1].setnetworkactive(state=False)
+        self.nodes[0].generate(100)
+
+        # Parsing the url of our node to get settings for config file
+        data_dir = self.nodes[0].datadir
+        node_url = urllib.parse.urlparse(self.nodes[0].url)
+        cfg_file = os.path.join(data_dir, "linearize.cfg")
+        bootstrap_file = os.path.join(self.options.tmpdir, "bootstrap.dat")
+        genesis_block = self.nodes[0].getblockhash(0)
+        blocks_dir = os.path.join(data_dir, "regtest", "blocks")
+        hash_list = tempfile.NamedTemporaryFile(dir=data_dir,
+                                                mode='w',
+                                                delete=False,
+                                                encoding="utf-8")
+
+        self.log.info("Create linearization config file")
+        with open(cfg_file, "a", encoding="utf-8") as cfg:
+            cfg.write("datadir={}\n".format(data_dir))
+            cfg.write("rpcuser={}\n".format(node_url.username))
+            cfg.write("rpcpassword={}\n".format(node_url.password))
+            cfg.write("port={}\n".format(node_url.port))
+            cfg.write("host={}\n".format(node_url.hostname))
+            cfg.write("output_file={}\n".format(bootstrap_file))
+            cfg.write("max_height=100\n")
+            cfg.write("netmagic=fcc1b7dc\n")
+            cfg.write("input={}\n".format(blocks_dir))
+            cfg.write("genesis={}\n".format(genesis_block))
+            cfg.write("hashlist={}\n".format(hash_list.name))
+
+        base_dir = self.config["environment"]["SRCDIR"]
+        linearize_dir = os.path.join(base_dir, "contrib", "linearize")
+
+        self.log.info("Run linearization of block hashes")
+        linearize_hashes_file = os.path.join(linearize_dir, "linearize-hashes.py")
+        subprocess.run([sys.executable, linearize_hashes_file, cfg_file],
+                       stdout=hash_list,
+                       check=True)
+
+        self.log.info("Run linearization of block data")
+        linearize_data_file = os.path.join(linearize_dir, "linearize-data.py")
+        subprocess.run([sys.executable, linearize_data_file, cfg_file],
+                       check=True)
+
+        self.log.info("Restart second, unsynced node with bootstrap file")
+        self.stop_node(1)
+        self.start_node(1, ["-loadblock=" + bootstrap_file])
+        wait_until(lambda: self.nodes[1].getblockcount() == 100)
+
+        assert_equal(self.nodes[1].getblockchaininfo()['blocks'], 100)
+        assert_equal(self.nodes[0].getbestblockhash(), self.nodes[1].getbestblockhash())
+
+
+if __name__ == '__main__':
+    LoadblockTest().main()

test/functional/test_framework/script_util.py

@@ -0,0 +1,25 @@
+#!/usr/bin/env python3
+# Copyright (c) 2019 The Bitcoin Core developers
+# Distributed under the MIT software license, see the accompanying
+# file COPYING or http://www.opensource.org/licenses/mit-license.php.
+"""Useful Script constants and utils."""
+from test_framework.script import CScript
+
+# To prevent a "tx-size-small" policy rule error, a transaction has to have a
+# size of at least 83 bytes (MIN_STANDARD_TX_SIZE in
+# src/policy/policy.h). Considering a Tx with the smallest possible single
+# input (blank, empty scriptSig), and with an output omitting the scriptPubKey,
+# we get to a minimum size of 60 bytes:
+#
+# Tx Skeleton: 4 [Version] + 1 [InCount] + 1 [OutCount] + 4 [LockTime] = 10 bytes
+# Blank Input: 32 [PrevTxHash] + 4 [Index] + 1 [scriptSigLen] + 4 [SeqNo] = 41 bytes
+# Output:      8 [Amount] + 1 [scriptPubKeyLen] = 9 bytes
+#
+# Hence, the scriptPubKey of the single output has to have a size of at
+# least 23 bytes, which corresponds to the size of a P2SH scriptPubKey.
+# The following script constant consists of a single push of 22 bytes of 'a':
+#   <PUSH_22> <22-bytes of 'a'>
+# resulting in a 23-byte size. It should be used whenever (small) fake
+# scriptPubKeys are needed, to guarantee that the minimum transaction size is
+# met.
+DUMMY_P2SH_SCRIPT = CScript([b'a' * 22])

test/functional/test_runner.py

@@ -231,6 +231,7 @@
     'feature_dip0020_activation.py',
     'feature_uacomment.py',
     'feature_filelock.py',
+    'feature_loadblock.py',
     'p2p_blockfilters.py',
     'feature_asmap.py',
     'feature_includeconf.py',

test/lint/README.md

@@ -7,6 +7,8 @@ Check for missing documentation of command line options.
 commit-script-check.sh
 ======================
 Verification of [scripted diffs](/doc/developer-notes.md#scripted-diffs).
+Scripted diffs are only assumed to run on the latest LTS release of Ubuntu. Running them on other operating systems
+might require installing GNU tools, such as GNU sed.
 
 git-subtree-check.sh
 ====================