b4dd2ef800 [test] Test the add_outbound_p2p_connection functionality (Amiti Uttarwar)
602e69e427 [test] P2PBlocksOnly - Test block-relay-only connections. (Amiti Uttarwar)
8bb6beacb1 [test/refactor] P2PBlocksOnly - Extract transaction violation test into helper. (Amiti Uttarwar)
99791e7560 [test/refactor] P2PBlocksOnly - simplify transaction creation using blocktool helper. (Amiti Uttarwar)
3997ab9154 [test] Add test framework support to create outbound connections. (Amiti Uttarwar)
5bc04e8837 [rpc/net] Introduce addconnection to test outbounds & blockrelay (Amiti Uttarwar)
Pull request description:
The existing functional test framework uses the `addnode` RPC to spin up manual connections between bitcoind nodes. This limits our ability to add integration tests for our networking code, which often executes different code paths for different connection types.
**This PR enables creating `outbound` & `block-relay-only` P2P connections in the functional tests.** This allows us to increase our p2p test coverage, since we can now verify expectations around these connection types.
This builds out the [prototype](https://github.com/bitcoin/bitcoin/issues/14210#issuecomment-527421978) proposed by ajtowns in #14210. 🙌🏽
An overview of this branch:
- introduces a new test-only RPC function `addconnection` which initiates opening an `outbound` or `block-relay-only` connection. (conceptually similar to `addnode` but for different connection types & restricted to regtest)
- adds `test_framework` support so a mininode can open an `outbound`/`block-relay-only` connection to a `P2PInterface`/`P2PConnection`.
- updates `p2p_blocksonly` tests to create a `block-relay-only` connection & verify expectations around transaction relay.
- introduces `p2p_add_connections` test that checks the behaviors of the newly introduced `add_outbound_p2p_connection` test framework function.
With these changes, there are many more behaviors that we can add integration tests for. The blocksonly updates is just one example.
Huge props to ajtowns for conceiving the approach & providing me feedback as I've built out this branch. Also thank you to jnewbery for lots of thoughtful input along the way.
ACKs for top commit:
troygiorshev:
reACK b4dd2ef800
jnewbery:
utACK b4dd2ef800
MarcoFalke:
Approach ACK b4dd2ef800🍢
Tree-SHA512: d1cba768c19c9c80e6a38b1c340cc86a90701b14772c4a0791c458f9097f6a4574b4a4acc7d13d6790c7b1f1f197e2c3d87996270f177402145f084ef8519a6b
39b43298d9 test: add test for banning of non-IP addresses (Vasil Dimov)
94d335da7f net: allow CSubNet of non-IP networks (Vasil Dimov)
Pull request description:
Allow creation of valid `CSubNet` objects of non-IP networks and only
match the single address they were created from (like /32 for IPv4 or
/128 for IPv6).
This fixes a deficiency in `CConnman::DisconnectNode(const CNetAddr& addr)`
and in `BanMan` which assume that creating a subnet from any address
using the `CSubNet(CNetAddr)` constructor would later match that address
only. Before this change a non-IP subnet would be invalid and would not
match any address.
ACKs for top commit:
jonatack:
Code review re-ACK 39b43298d9 per `git diff 5e95ce6 39b4329`; only change since last review is improvements to the functional test; verified the test fails on master @ 616eace0 where expected (`assert(self.is_banned(node, tor_addr))` fails and unban unfails)
laanwj:
code review ACK 39b43298d9
Tree-SHA512: 3239b26d0f2fa2d1388b4fdbc1d05ce4ac1980be699c6ec46049409baefcb2006b1e72b889871e2210e897f6725c48e873f68457eea7e6e4958ab4f959d20297
faabc26a61 test: Replace getmempoolentry with testmempoolaccept in MiniWallet (MarcoFalke)
Pull request description:
This is a refactor to not use the return value of `sendrawtransaction` and `getmempoolentry` with the goal that submitting the tx to the mempool will become optional.
ACKs for top commit:
mjdietzx:
ACK faabc26a61
Tree-SHA512: 4260a59e65fed1c807530dad23f1996ba6e881bcce91995f5b498a0be6001f67b3e0251507c8801750fe105410147c68bb2f393ebe678c6a3a4d045e5d72fc19
faa8f68943 Replace boost::variant with std::variant (MarcoFalke)
Pull request description:
Now that we can use std::variant from the vanilla standard library, drop the third-party boost variant dependency
ACKs for top commit:
fjahr:
Code review ACK faa8f68943
fanquake:
ACK faa8f68943
Tree-SHA512: 6e3aecd33b00c2e31a763f999247944d5b2ce5e3018f1965c516c1000cd08ff6703a8d50fb0be64883153da2925ae72986b8a6b96586db74057bd05d6f4986e6
a7599c80eb test: run mempool_compatibility.py even with wallet disabled (Michael Dietz)
Pull request description:
Another functional test rewritten as proposed in https://github.com/bitcoin/bitcoin/issues/20078
ACKs for top commit:
MarcoFalke:
review ACK a7599c80eb didn't test
Tree-SHA512: cc7a617e5489ed27bbdbdee85a82fa08525375061f7f4524577a6b8ecb340396adac88419b51f513be22ca53edd0a3bd5d572d9f43ffc2c18550b0ef9069d238
a191e23b8e doc: Add release notes (Hennadii Stepanov)
ae749d12dd doc: Add libnatpmp stuff (Hennadii Stepanov)
e28f9be87a ci: Add libnatpmp-dev package to some builds (Hennadii Stepanov)
5a0185b6c9 gui: Add NAT-PMP network option (Hennadii Stepanov)
a39f7336a3 net: Add -natpmp command line option (Hennadii Stepanov)
28acffd9d5 net: Add NAT-PMP to port mapping loop (Hennadii Stepanov)
a8d9f275d0 net: Add libnatpmp support (Hennadii Stepanov)
58e8364dcd gui: Apply port mapping changes on dialog exit (Hennadii Stepanov)
cf151cc68c scripted-diff: Rename UPnP stuff (Hennadii Stepanov)
4e91b1e24d net: Add flags for port mapping protocols (Hennadii Stepanov)
8b50d1b5bb net: Keep trying to use UPnP when -upnp=1 (Hennadii Stepanov)
28e2961fd6 refactor: Replace magic number with named constant (Hennadii Stepanov)
02ccf69dd6 refactor: Move port mapping code to its own module (Hennadii Stepanov)
Pull request description:
Close#11902
This PR is an alternative to:
- #12288
- #15717
To compile with NAT-PMP support on Ubuntu [`libnatpmp-dev`](https://packages.ubuntu.com/source/bionic/libnatpmp) should be available.
Log excerpt:
```
2020-02-05T20:12:28Z [mapport] NAT-PMP: public address = 95.164.65.194
2020-02-05T20:12:28Z [mapport] AddLocal(95.164.65.194:18333,3)
2020-02-05T20:12:28Z [mapport] NAT-PMP: port mapping successful.
```
See: [`libnatpmp`](https://miniupnp.tuxfamily.org/libnatpmp.html)
---
Some follow-ups are out of this PR's scope:
- mention NAT-PMP library in the version message
- ~integrate NAT-PMP into the GUI~ (already [added](https://github.com/bitcoin/bitcoin/pull/18077#issuecomment-589405068))
ACKs for top commit:
laanwj:
Tested and code review ACK a191e23b8e
Tree-SHA512: 10e19267c21bf30f20ff1abfc882d526049f0e790b95e12f109dc2bed7c0aef45de03eaf967f4e667e7509be04f1873a5c508087393d947205f3aab2ad6d7cf1
Open max number of full-relay and block-relay-only connections from a
functional test with different sorts of behaviors to ensure it behaves as
expected.
In the interest of increasing our P2P test coverage, add support to create
full-relay or block-relay-only connections. To support this, a P2P connection
spins up a listening thread & uses a callback to trigger the node initiating
the connection.
Co-authored-by: Anthony Towns <aj@erisian.com.au>
9815332d51 test: Change MuHash Python implementation to match cpp version again (Fabian Jahr)
01297fb3ca fuzz: Add MuHash consistency fuzz test (Fabian Jahr)
b111410914 test: Add MuHash3072 fuzz test (Fabian Jahr)
c122527385 bench: Add Muhash benchmarks (Fabian Jahr)
7b1242229d test: Add MuHash3072 unit tests (Fabian Jahr)
adc708c98d crypto: Add MuHash3072 implementation (Fabian Jahr)
0b4d290bf5 crypto: Add Num3072 implementation (Fabian Jahr)
589f958662 build: Check for 128 bit integer support (Fabian Jahr)
Pull request description:
This is the first split of #18000 which implements the Muhash algorithm and uses it to calculate the UTXO set hash in `gettxoutsetinfo`.
ACKs for top commit:
laanwj:
Code review ACK 9815332d51
Tree-SHA512: 4bc090738f0e3d80b74bdd8122e24a8ce80121120fd37c7e4335a73e7ba4fcd7643f2a2d559e2eebf54b8e3a3bd5f12cfb27ba61ded135fda210a07a233eae45
fa6c114ae6 test: Add sanitizer suppressions for AMD EPYC CPUs (MarcoFalke)
Pull request description:
Currently the ci system only runs on intel cpus (and some arm devices), but it won't run on CPUs `Using the 'shani(1way,2way)' SHA256 implementation` (excerpt from debug log).
For reference, google cloud CPUs (which is what Cirrus CI uses) print `Using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation`
The traceback I got:
```
crypto/sha256_shani.cpp:87:18: runtime error: unsigned integer overflow: 0 - 1 cannot be represented in type 'size_t' (aka 'unsigned long')
#0 0x55c0000e95ec in sha256_shani::Transform(unsigned int*, unsigned char const*, unsigned long) /root/bitcoin/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/crypto/sha256_shani.cpp:87:18
#1 0x55bfffb926f8 in (anonymous namespace)::SelfTest() /root/bitcoin/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/crypto/sha256.cpp:517:9
#2 0x55bfffb906ed in SHA256AutoDetect[abi:cxx11]() /root/bitcoin/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/crypto/sha256.cpp:626:5
#3 0x55bfff87ab97 in BasicTestingSetup::BasicTestingSetup(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<char const*, std::allocator<char const*> > const&) /root/bitcoin/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/test/util/setup_common.cpp:104:5
#4 0x55bffe885877 in main /root/bitcoin/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/qt/test/test_main.cpp:52:27
#5 0x7f20c3bf60b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2)
#6 0x55bffe7a5f6d in _start (/root/bitcoin/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/src/qt/test/test_bitcoin-qt+0x1d00f6d)
SUMMARY: UndefinedBehaviorSanitizer: unsigned-integer-overflow crypto/sha256_shani.cpp:87:18 in
ACKs for top commit:
laanwj:
Anyhow ACK fa6c114ae6
Tree-SHA512: 968a1d28eedec58c337b1323862f583cb1bcd78c5f03396940b9ab53ded12f8c6652877909aba05ee5586532137418fd817ff979bd7bef6e07856094f9d7f9b1
1112035d32 doc: fix various typos (Ikko Ashimine)
e8640849c7 doc: Use https URLs where possible (Sawyer Billings)
Pull request description:
Consolidates / fixes the changes from #20762, #20836, #20810. There is no output when `test/lint/lint-all.sh` is run.
Closes#20807.
ACKs for top commit:
MarcoFalke:
ACK 1112035d32
Tree-SHA512: 22ca824688758281a74e5ebc6a84a358142351434e34c88c6b36045d2d241ab95fd0958565fd2060f98317e62e683323b5320cc7ec13592bf340e6922294ed78
fa749fbea3 rpc: Replace boost::variant with std::variant for RPCArg.m_fallback (MarcoFalke)
Pull request description:
Now that we can use std::variant from the vanilla standard library, drop the third-party boost variant dependency.
Patch is split out from #20480. A step-by-step replacement is possible because we don't have our own `Variant` wrapper and the source code specifies `boost::variant` explicitly.
I think a step-by-step replacement should be preferred, because it simplifies review.
ACKs for top commit:
fjahr:
re-ACK fa749fbea3
Sjors:
re-ACK fa749fbea3
Tree-SHA512: 5e3c12b7d535f73065b4afa8df0a488f78fb25d2234f5ecbf740e624db03a34c35fea100eb7d37e84741721310e6450b7fb4296a2207a7ed1fa24485b3650981
454a4088a8 [doc] Add release notes for removed getpeerinfo fields. (Amiti Uttarwar)
b1a936d4ae [rpc] Remove deprecated "whitelisted" field from getpeerinfo (Amiti Uttarwar)
094c3beaa4 [rpc] Remove deprecated "banscore" field from getpeerinfo (Amiti Uttarwar)
537053336f [rpc] Remove deprecated "addnode" field from getpeerinfo (Amiti Uttarwar)
Pull request description:
This PR removes support for 3 fields on the `getpeerinfo` RPC that were deprecated in v0.21- `addnode`, `banscore` & `whitelisted`.
ACKs for top commit:
sipa:
utACK 454a4088a8
jnewbery:
ACK 454a4088a8.
Tree-SHA512: ccc0e90c0763eeb8529cf0c46162dbaca3f7773981b3b52d9925166ea7421aed086795d56b320e16c9340f68862388785f52a9b78314865070917b33180d7cd6
40fdb2a212 test: Fix Comment Typo in BitcoinTestFramework (Joel Klabo)
Pull request description:
Missing "override" in comment describing use of set_test_params
ACKs for top commit:
michaelfolkson:
ACK 40fdb2a212
Tree-SHA512: bf893a0d5f8dc86a3ec2eaf48cd7c0f0f832f3b3d254b3d99953336db7e294571b1d2c8686030bf8a27cbe67b1a85a54e53ebefb2e57d6d8d6ac864a15dce4e7
fa511042b0 doc: [test] Remove outdated comment in fuzz runner (MarcoFalke)
Pull request description:
All folders are soft-created with `os.makedirs`
ACKs for top commit:
RiccardoMasutti:
ACK fa51104
Tree-SHA512: 4051688946a205a981bbb005300fe3263495ead26591042b38ae44f4297c7689a613b560052fb5405a62054734d2599cfb0554a37c7b7369fb3a3636743d04a8
66d012ad7f test: RPC: getblock fee calculations (Elliott Jin)
bf7d6e31b1 RPC: getblock: tx fee calculation for verbosity 2 via Undo data (Elliott Jin)
Pull request description:
This change is progress towards #18771 . It adapts the fee calculation part of #16083 and addresses some feedback. The additional "verbosity level 3" features are planned for a future PR.
**Original PR description:**
> Using block undo data (like in #14802) we can now show fee information for each transaction in a block without the need for additional -txindex and/or a ton of costly lookups. For a start we'll add transaction fee information to getblock verbosity level 2. This comes at a negligible speed penalty (<1%).
ACKs for top commit:
luke-jr:
tACK 66d012ad7f
fjahr:
tACK 66d012ad7f
MarcoFalke:
review ACK 66d012ad7f 🗜
Tree-SHA512: be1fe4b866946a8dc36427f7dc72a20e10860e320a28fa49bc85bd2a93a0d699768179be29fa52e18b2ed8505d3ec272e586753ef2239b4230e0aefd233acaa2
812baaa1f8 Switch to BIP341's suggested scheme for outputs without script (Pieter Wuille)
Pull request description:
BIP341 suggests using Hash<sub>TapTweak</sub>(pubkey) to derive the tweak in case of key-only outputs. The functional test framework currently uses Hash<sub>TapTweak</sub>(pubkey || 0x00...00) instead. Change this.
There is no technical reason to prefer one over the other, but in case someone looks at it for inspiration, it's better to be consistent with the BIP.
ACKs for top commit:
laanwj:
ACK 812baaa1f8
instagibbs:
ACK 812baaa1f8
Tree-SHA512: 02576c38776ec786255f49d7edecdb1ed8a9dcf0f547d58c23099588b4c3296edf279b103a6eb80e0f07d3c5ee9743f67d152f5244fd63adc6613b004f6969ed
fa957f8dc9 test: Add race:SendZmqMessage tsan suppression (MarcoFalke)
Pull request description:
Add suppression for `race:SendZmqMessage`, which isn't covered by the existing `zmq::*` suppression
Fixes#20618
ACKs for top commit:
hebasto:
re-ACK fa957f8dc9, as my previous comment is not directly related to this pull changes.
Tree-SHA512: 8642a8b79bbfa4bee89042b66e528f27fd78c5e84a33023df440662e9114e31445fd7b04940f44b11fa4ab7438d346385a21816289c818cce9958a9b16730452
fab46b34f4 test: Fix restart node race (MarcoFalke)
Pull request description:
It is not allowed to start a node before it has been fully stopped. Otherwise it could lead to intermittent issues due to access issues (e.g. cookie file https://cirrus-ci.com/task/6409665024098304?command=ci#L4793)
Fix that by waiting for the node to fully stop.
ACKs for top commit:
laanwj:
code review ACK fab46b34f4
Tree-SHA512: 7605cac0573a7b04f05ff110d0131e8940d87f7baf6d698505ed16b363d4d15b1e552c5ffd1a187c8fe5639f7e265c3122734c85283275746e46bd789614fd21
11a32722f0 test: run mempool_resurrect.py even with wallet disabled (Michael Dietz)
Pull request description:
Another functional test rewritten as proposed in #20078
**Request for help:**
`node.gettransaction(txid)` fails for transactions sent with `wallet.send_self_transfer`. Even though the `txid`s look correct, are added to the mempool correctly, and removed from the mempool when a block is mined - all as expected.
However, `node.gettransaction(txid)` throws the error:
```sh
Traceback (most recent call last):
File "/Users/michaeldietz/Documents/bitcoin/test/functional/test_framework/test_framework.py", line 126, in main
self.run_test()
File "/Users/michaeldietz/Documents/bitcoin/test/functional/mempool_resurrect.py", line 43, in run_test
assert_equal(len(list(filter(lambda txid: node.gettransaction(txid)["confirmations"] > 0, spends_ids))), len(spends_ids))
File "/Users/michaeldietz/Documents/bitcoin/test/functional/mempool_resurrect.py", line 43, in <lambda>
assert_equal(len(list(filter(lambda txid: node.gettransaction(txid)["confirmations"] > 0, spends_ids))), len(spends_ids))
File "/Users/michaeldietz/Documents/bitcoin/test/functional/test_framework/coverage.py", line 47, in __call__
return_val = self.auth_service_proxy_instance.__call__(*args, **kwargs)
File "/Users/michaeldietz/Documents/bitcoin/test/functional/test_framework/authproxy.py", line 146, in __call__
raise JSONRPCException(response['error'], status)
test_framework.authproxy.JSONRPCException: Invalid or non-wallet transaction id (-5)
```
Anyone know what's going wrong / can point me in the right direction if I'm making a mistake, or `MiniWallet` needs to be improved for this to work correctly?
ACKs for top commit:
MarcoFalke:
ACK 11a32722f0
Tree-SHA512: 13d83a13ec23920db716e99b68670e61329d1cc73b12063d85bc1679ee6425a9951da4d2e392ca1f27760be7be049ccdc6f504e192ed5cd24ed0ba003b66fab3
Fabian Jahr