This commit effectively splits the "load block index itself" logic from
"derive Chainstate variables from loaded block index" logic.
This means that BlockManager::LoadBlockIndex{,DB} will only load what's
relevant to the BlockManager.
I strongly recommend reviewing with the following git-diff flags:
--color-moved=dimmed_zebra --color-moved-ws=allow-indentation-change
0189df1d31 build, mac: Include arch in codesignature tarball (Andrew Chow)
6e9308c6d4 guix: use latest signapple (Andrew Chow)
Pull request description:
Since we have two architectures for Mac binaries, having the architecture in the code signature tarball generated by `detached-sig-create.sh` allows us to avoid accidentally overwriting an existing code signature tarball during the code signing process.
ACKs for top commit:
fanquake:
ACK 0189df1d31
Tree-SHA512: 7e0d282e4ced1094f36f1d26ff6e18d53449561ab3a1a95ac69eb5ff3d7b33ee4bd8fad004884806064a29541c47f9e5879c2a1fd0f54453413245bdcf53c4c7
5b1aae12ca qt: Avoid crash on startup if int specified in settings.json (Ryan Ofsky)
84b0973e35 test: Add tests for GetArg methods / settings.json type coercion (Ryan Ofsky)
Pull request description:
Should probably add this change to 23.x as suggested by Luke https://github.com/bitcoin/bitcoin/issues/24457#issuecomment-1059825678. If settings like `prune` are added to `settings.json` in the future, it would be preferable for 23.x releases to respect the setting instead of crash.
---
Fix GUI startup crash reported by Rspigler in https://github.com/bitcoin/bitcoin/issues/24457 that happens if `settings.json` contains an integer value for any of the configuration options which GUI settings can currently clash with (-dbcache, -par, -spendzeroconfchange, -signer, -upnp, -natpmp, -listen, -server, -proxy, -proxy, -onion, -onion, -lang, and -prune).
The fix is a one-line change in `ArgsManager::GetArg`. The rest of the PR just adds a regression test for the GUI and unit tests for ArgsManager::GetArg methods.
ACKs for top commit:
laanwj:
Code review ACK 5b1aae12ca
achow101:
ACK 5b1aae12ca
jonatack:
Code review ACK 5b1aae12ca
Tree-SHA512: 958991b4bead9b82a3879fdca0f8d6405e2a212b7c46cf356f078843a4f156e27fd75fc46e2013aa5159582ead06d343c1ed248d678b3e5bbd312f247e37894c
691d45fdc8 Add coinstatsindex_unclean_shutdown test (Ryan Ofsky)
eb6cc05da3 index: Commit DB_MUHASH and DB_BEST_BLOCK to disk together (Martin Zumsande)
Pull request description:
Fixes#24076
Coinstatsindex currently writes the MuHash (`DB_MUHASH`) to disk in `CoinStatsIndex::WriteBlock()` and `CoinStatsIndex::ReverseBlock()`, but the best synced block is written in `BaseIndex::Commit()`. These are called at different points in time, both during the ThreadSync phase, and also after the initial sync is finished and validation callbacks (`BlockConnected()` vs `ChainStateFlushed()`) perform the syncing.
As a result, the index DB is temporarily in an inconsistent state, and if bitcoind is terminated uncleanly (so that there is no time to call `Commit()` by receiving an interrupt or by flushing the chainstate) this leads to problems:
On the next startup, `Init()` will read the best block and a MuHash that corresponds to a different (higher) block. Indexing will be picked up at the the best block processing some blocks again, but since MuHash is a rolling hash, it will process some utxos twice and the muhashes for all future blocks will be wrong, as was observed in #24076.
Fix this by always committing `DB_MUHASH` together with `DB_BEST_BLOCK`.
Note that the block data for the index is still written at different times, but this does not corrupt the index - at worst, these entries will be processed another time and overwritten after an unclean shutdown and restart.
ACKs for top commit:
ryanofsky:
Code review ACK 691d45fdc8. Only change since last review is adding test
fjahr:
ACK 691d45fdc8
Tree-SHA512: e1c3b5f06fa4baacd1b070abb0f8111fe2ea4a001ca8b8bf892e96597cf8b5d5ea10fa8fb837cfbf46648f052c742d912add4ce26d4406294fc5fc20809a0e1b
db27ac9354 tests: Ensure sorted/multi_a descriptors always generate different addrs (Andrew Chow)
Pull request description:
Sometimes the multi_a and sortedmulti_a descriptors will produce some of the same addresses in the tests. This causes the wallets to start generating addresses at a different index as they detect that one of the addresses is used. This subsequently causes a test failure.
To avoid this problem, use descriptors that will produce unique addresses by putting one of the multi_a in a different branch.
ACKs for top commit:
ajtowns:
ACK db27ac9354
theStack:
Tested ACK db27ac9354
Tree-SHA512: 0f57822bf4c7c79da304f092d7d43d6118e78a087cbeb0766fbbf634dc27911ae723d5d41350884d3b63a24d3b3817944f7e5fa534afb849161dd008a1e4a62f
7a68fe4831 bitcoin-chainstate: Lock cs_main to UnloadBlockIndex (Carl Dong)
Pull request description:
This was introduced because of a silent merge conflict.
ACKs for top commit:
promag:
ACK 7a68fe4831
jonatack:
ACK 7a68fe4831
Tree-SHA512: 4c135efd68604452485a129e731675ff5917c157a70c77dd702211d9902c21b3b29380a881723f43ecba4762bc864b036881bb502b3b792e581565dcaa7a7ed4
7abd8b21ba doc: include wtxid in TransactionDescriptionString (brunoerg)
2d596bce6f doc: add wtxid info in release-notes (brunoerg)
a5b66738f1 test: add wtxid in expected_fields for wallet_basic (brunoerg)
e8c659a297 wallet: add wtxid in WalletTxToJSON (brunoerg)
7482b6f895 wallet: add GetWitnessHash() (brunoerg)
Pull request description:
This PR add `wtxid` in `WalletTxToJSON` which allows to return this field in `listsinceblock`, `listtransactions` and `gettransaction` (RPCs).
ACKs for top commit:
achow101:
re-ACK 7abd8b21ba
w0xlt:
crACK 7abd8b2
luke-jr:
re-utACK 7abd8b21ba
Tree-SHA512: f86f2dbb5e38e7b19932006121802f47b759d31bdbffe3263d1db464f6a3a30fddd68416f886a44f6d3a9fd570f7bd4f8d999737ad95c189e7ae5e8ec1ffbdaa
fa097d074b addrman: Log too low compat value (MarcoFalke)
Pull request description:
Before this patch, when writing a negative `lowest_compatible` value, it would be read as a positive value. For example `-32` will be read as `224`. There is generally nothing wrong with that. Though, similarly there shouldn't be anything wrong with refusing to read a negative value. I find the code after this patch more logical than before. Also, this allows dropping a file-wide sanitizer suppression.
In practice none of this should ever happen. Bitcoin Core would never write a negative `lowest_compatible` in normal operation, unless the file storage is later corrupted by external influence.
ACKs for top commit:
mzumsande:
re-ACK fa097d074b
Tree-SHA512: 9aae7b8fe666f52f667f149667025e0160cef1a793cc4d392e36608f65c2bee8096da429235118f40a3368f327aabe30f3732ae78c5874648ea6f423f2687b65
31846b006d test: refactor: use `random.sample` for choosing random keys in wallet_taproot.py (Sebastian Falbesoner)
Pull request description:
The Python3 standard library method `random.sample` has the exact same functionality as the helper method `rand_keys(...)` (that is, random sampling without replacement) on a generic set or sequence, i.e. we can simply replace it. See https://docs.python.org/3/library/random.html#random.sample
Note that this is also safer: in case that the sample size `k` is larger than the population count, `random.sample` throws an error:
```
$ python3
Python 3.8.12 (default, Sep 26 2021, 13:12:50)
[Clang 11.1.0 ] on openbsd7
Type "help", "copyright", "credits" or "license" for more information.
>>> import random
>>> random.sample([23, 42], 3)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.8/random.py", line 363, in sample
raise ValueError("Sample larger than population or is negative")
ValueError: Sample larger than population or is negative
```
while the custom method would get stuck in an endless loop.
ACKs for top commit:
shaavan:
Code Review ACK 31846b006d
Tree-SHA512: d9bd7f8128e43401a5b0388e48ba838155b21db5b4b6ba95c91285880788bc3917cb656b74bbe2d97faf7b44862d20b0899dc3c56aa48b9d2b33b50e37d089f6
Fix GUI startup crash reported by Rspigler in
https://github.com/bitcoin/bitcoin/issues/24457 that happens if
settings.json contains an integer value for any of the configuration
options which GUI settings can currently clash with (-dbcache, -par,
-spendzeroconfchange, -signer, -upnp, -natpmp, -listen, -server, -proxy,
-proxy, -onion, -onion, -lang, and -prune).
Fix is a one-line change in ArgsManager::GetArg.
Just add tests. No changes to application behavior. Tests will be
updated in the next commit changing & improving current behavior.
Include a Qt test for GUI startup crash reported by Rspigler in
https://github.com/bitcoin/bitcoin/issues/24457 caused by GetArg
behavior that happens if settings.json contains an integer value for any
of the configuration options which GUI settings can currently clash with
(-dbcache, -par, -spendzeroconfchange, -signer, -upnp, -natpmp, -listen,
-server, -proxy, -proxy, -onion, -onion, -lang, and -prune).
The macOS and Windows builds do not require a GCC 7 toolchain, and this
is actually causing build issues, i.e #24211. So switch to using a GCC
10 native toolchain for both.
6833aceac9 build: Move guix time machine to prelude (laanwj)
Pull request description:
This deduplicates some code, and enforces consistency of the time machine configuration between scripts.
ACKs for top commit:
achow101:
ACK 6833aceac9
dongcarl:
ACK 6833aceac9
Tree-SHA512: c02ded154cdb982293101986ef863d46554fc428eb5617bee0288dbef0543f994de5044123ac9958e455d0d24276a1c4512149a10dd1efaca8677c8f6b74b0a9
6c23c41561 refactor: Rewrite AddToBlockIndex with try_emplace (Carl Dong)
c05cf7aa1e style: Modernize range-based loops over m_block_index (Carl Dong)
c2a1655799 style-only: Use using instead of typedef for BlockMap (Carl Dong)
dd79dad175 refactor: Rewrite InsertBlockIndex with try_emplace (Carl Dong)
531dce0347 tests: Remove now-unnecessary manual Unload's (Carl Dong)
bec86ae326 blockstorage: Make m_block_index own CBlockIndex's (Carl Dong)
Pull request description:
Part of: #24303
Split off from: #22564
```
Instead of having CBlockIndex's live on the heap, which requires manual
memory management, have them be owned by m_block_index. This means that
they will live and die with BlockManager.
```
The second commit demonstrates how this makes calls to `Unload()` to satisfy the address sanitizer unnecessary.
ACKs for top commit:
ajtowns:
ACK 6c23c41561
MarcoFalke:
re-ACK 6c23c41561🎨
Tree-SHA512: 81b2b5119be27cc0f8a9457b11da60cc60930315d2a5be36be89fe253d32073ffe622348ff153114b9b3212197bddbc791810913a43811b33cc58e7162bd105b
1b20109b04 Squashed 'src/leveldb/' changes from f8ae182c1e..330dd6235f (MarcoFalke)
Pull request description:
A minor change to:
* Consistently use the same symbol names in the whole project.
* Fix compiling with C++20.
ACKs for top commit:
fanquake:
ACK fa0c32eb74
Tree-SHA512: b5d4540dd621cf4aa8caac811bae03bb74e502a31dbdda9354182e4caa39905550e62ad3cf8ea7d7f9bfc3e5120d119d34ab0f1e633716ec8089876037cbf192
ae9ceed3e2 validation, refactoring: remove ChainstateManager::Reset() (Jon Atack)
daad0093e3 validation: replace lock with annotation in UnloadBlockIndex() (Jon Atack)
Pull request description:
Thread safety refactoring seen in #24177:
- replace re-acquiring lock cs_main with a thread safety annotation in UnloadBlockIndex()
- remove ChainstateManager::Reset(), as it is currently unused (can be reintroduced in the test utilities if needed for unit testing)
ACKs for top commit:
laanwj:
Code review ACK ae9ceed3e2
vasild:
ACK ae9ceed3e2
klementtan:
crACK ae9ceed3e2
Tree-SHA512: cebb782572997cc2dda01590d6bb6c5e479e8202324d8b6ff459b814ce09e818b996c881736bfebd1b8bf4b6d7a0f79faf3ffea176a4699dd7d7429de2db2d13
Sometimes the multi_a and sortedmulti_a descriptors will produce some of
the same addresses in the tests. This causes the wallets to start
generating addresses at a different index as they detect that one of
the addresses is used. This subsequently causes a test failure.
To avoid this problem, use descriptors that will produce unique
addresses by putting one of the multi_a in a different branch.
68c4a9ed38 ci: Bump vcpkg to the latest version (Hennadii Stepanov)
Pull request description:
It seems reasonable to run a CI task against the most recent dependencies.
Dependency changes:
- boost 1.75.0 -> 1.78.0
- double-conversion 3.1.5 -> 3.2.0
- sqlite3 3.35.4 -> 3.37.2
ACKs for top commit:
fanquake:
ACK 68c4a9ed38
Tree-SHA512: 8d8ea42cb37b5eb2e6b82db4fd14b2984a1dee87a5d79e2378feff2e2576403207f5a27d3da7c4b351c1cc570ec8d971ae963c179789ef0ee55e004fbd399fe1
60aa179d8f Use GetPathArg where possible (Pavol Rusnak)
5b946edd73 util, refactor: Use GetPathArg to read "-settings" value (Ryan Ofsky)
687e655ae2 util: Add GetPathArg default path argument (Ryan Ofsky)
Pull request description:
Improve `ArgsManager::GetPathArg` method added in recent PR #24265, so it is usable more places. This PR starts to use it for the `-settings` option. This can also be helpful for #24274 which is parsing more path options.
- Add `GetPathArg` default argument so it is less awkward to use to parse options that have default values.
- Fix `GetPathArg` negated argument handling. Return path{} not path{"0"} when path argument is negated.
- Add unit tests for default and negated cases
- Move `GetPathArg` method declaration next to `GetArg` declaration. The two methods are close substitutes for each, so this should help keep them consistent and make them more discoverable.
ACKs for top commit:
w0xlt:
Tested ACK 60aa179 on Ubuntu 21.10
hebasto:
re-ACK 60aa179d8f
Tree-SHA512: 3d24b885d8bbeef39ea5d0556e2f09b9e5f4a21179cef11cbbbc1b84da29c8fb66ba698889054ce28d80bc25926687654c8532ed46054bf5b2dd1837866bd1cd
faa329fd46 refactor: Release cs_main before MaybeSendFeefilter (MarcoFalke)
Pull request description:
There is no need for any lock to be held, because net processing is single threaded. So holding the validation lock cs_main for sending a feefilter is confusing and might even degrade blockchain-related RPC performance minimally.
ACKs for top commit:
ajtowns:
ACK faa329fd46 ; code review only
vasild:
ACK faa329fd46
Tree-SHA512: 3e7f9faff1631cc64c86fc1a354ada67617ad1e7a046625cc741f4711854eb41ca8aad5a51ef0d94ff65947b68dba8345c9f786b20ee0a8b7a2e8741cfced21f
29862bdd40 guix: use same commit for codesigning time-machine (fanquake)
Pull request description:
The time machines should be updated in lockstep.
ACKs for top commit:
gruve-p:
ACK 29862bdd40
achow101:
ACK 29862bdd40
hebasto:
ACK 29862bdd40, I have reviewed the code and it looks OK, I agree it can be merged.
Tree-SHA512: 1cd6e449bba62041f6fde1463c490488b8d60dd86714818f27e92725008af850d3ce3fbde0c302406044cc73094cccc9e9bf4179256e9783627b18c5e66f8b49
6f2593dc23 gui, refactor: use std::chrono for formatDurationStr() helper (Jon Atack)
Pull request description:
Updates `formatDurationStr()` to use the `chrono` standard lib. No change in behavior.
ACKs for top commit:
RandyMcMillan:
tACK 6f2593dc23
shaavan:
ACK 6f2593dc23
w0xlt:
tACK 6f2593d on Ubuntu 21.10 Qt 5.15.2
promag:
Code review ACK 6f2593dc23.
Tree-SHA512: 61e9afdb1db779150df338e6af08727c34f69639add465c2f7003ff775d97dce3e78e78d325bc6dea5bc13f0fce9ef1c3506d13f1661a5e083e52bba8a32ba44
fa7dada1fc build: update ax_cxx_compile_stdcxx to serial 14 (MarcoFalke)
Pull request description:
No strong reason for the bump, but this makes it easier to experiment with cpp20, see https://github.com/bitcoin/bitcoin/pull/24169#issuecomment-1048702236
ACKs for top commit:
Empact:
Code Review ACK fa7dada1fc
fanquake:
ACK fa7dada1fc
Tree-SHA512: bd3e884bae5319d434520a2947608913c3884de89aa563aa46ef17ba4e5b41ba209bd4780c8eaf81648297b2dc9534803be88d1b214ad05a93ee5992bee887c0
4828d53ecc Add (sorted)multi_a descriptors to doc/descriptors.md (Pieter Wuille)
b5f33ac1f8 Simplify wallet_taproot.py functional test (Pieter Wuille)
eb0667ea96 Add tests for (sorted)multi_a derivation/signing (Pieter Wuille)
c17c6aa08d Add signing support for (sorted)multi_a scripts (Pieter Wuille)
3eed6fca57 Add multi_a descriptor inference (Pieter Wuille)
79728c4a3d Add (sorted)multi_a descriptor and script derivation (Pieter Wuille)
25e95f9ff8 Merge/generalize IsValidMultisigKeyCount/GetMultisigKeyCount (Pieter Wuille)
Pull request description:
This adds a new `multi_a(k,key_1,key_2,...,key_n)` (and corresponding `sortedmulti_a`) descriptor for k-of-n policies inside `tr()`. Semantically it is very similar to the existing `multi()` descriptor, but with the following changes:
* The corresponding script is `<key1> OP_CHECKSIG <key2> OP_CHECKSIGADD <key3> OP_CHECKSIGADD ... <key_n> OP_CHECKSIGADD <k> OP_NUMEQUAL`, rather than the traditional `OP_CHECKMULTISIG`-based script, making it usable inside the `tr()` descriptor.
* The keys can optionally be specified in x-only notation.
* Both the number of keys and the threshold can be as high as 999; this is the limit due to the consensus stacksize=1000 limit
I expect that this functionality will later be replaced with a miniscript-based implementation, but I don't think it's necessary to wait for that.
Limitations:
* The wallet code will for not estimate witness size incorrectly for script path spends, which may result in a (dramatic) fee underpayment with large multi_a scripts.
* The multi_a script construction is (slightly) suboptimal for n-of-n (where a `<key1> OP_CHECKSIGVERIFY ... <key_n-1> OP_CHECKSIGVERIFY <key_n> OP_CHECKSIG` would be better). Such a construction is not included here.
ACKs for top commit:
achow101:
ACK 4828d53ecc
gruve-p:
ACK 4828d53ecc
sanket1729:
code review ACK 4828d53ecc
darosior:
Code review ACK 4828d53ecc
Tree-SHA512: 5dcd434b79585f0ff830f7d501d27df5e346f5749f47a3109ec309ebf2cbbad0e1da541eec654026d911ab67fd7cf7793fab0f765628d68d81b96ef2a4d234ce
bbbbeaf9c8 fuzz: Limit script_format to 100kB (MarcoFalke)
Pull request description:
The target is still one of the slowest ones, but doesn't seem incredibly important. Especially for sizes larger than the standard tx size.
Fix that by limiting the script size.
ACKs for top commit:
fanquake:
ACK bbbbeaf9c8
Tree-SHA512: b6cf7248753909ef2f21d8824f187e7c05732dd3b99619c0067f862f3c2b0f9a87779d4ddbbd3a7a4bae5c794280e2f0a223bf835d6bc6ccaba01817d69479a2
2c03cec2ff ci: Build bitcoin-chainstate (Carl Dong)
095aa6ca37 build: Add example bitcoin-chainstate executable (Carl Dong)
Pull request description:
Part of: #24303
This PR introduces an example/demo `bitcoin-chainstate` executable using said library which can print out information about a datadir and take in new blocks on stdin.
Please read the commit messages for more details.
-----
#### You may ask: WTF?! Why is `index/*.cpp`, etc. being linked in?
This PR is meant only to capture the state of dependencies in our consensus engine as of right now. There are many things to decouple from consensus, which will be done in subsequent PRs. Listing the files out right now in `bitcoin_chainstate_SOURCES` is purely to give us a clear picture of the task at hand, it is **not** to say that these dependencies _belongs_ there in any way.
### TODO
1. Clean up `bitcoin-chainstate.cpp`
It is quite ugly, with a lot of comments I've left for myself, I should clean it up to the best of my abilities (the ugliness of our init/shutdown might be the upper bound on cleanliness here...)
ACKs for top commit:
ajtowns:
ACK 2c03cec2ff
ryanofsky:
Code review ACK 2c03cec2ff. Just rebase, comments, formatting change since last review
MarcoFalke:
re-ACK 2c03cec2ff 🏔
Tree-SHA512: 86e7fb5718caa577df8abc8288c754f4a590650d974df9d2f6476c87ed25c70f923c4db651c6963f33498fc7a3a31f6692b9a75cbc996bf4888c5dac2f34a13b
23.x was forked off, release notes on master should be empty.
Tree-SHA512: 0b48006073302b7b1c7602b4843d3a3048e88f357fb7049e478ec946f12eb16ca813272e719e47de5fb9713984ccf59551372a7ccd7ced7afaac6b5f5687d78b
On the master branch, bump to 23.99 (pre-24.0).
Tree-SHA512: 1e3b0cee8a2b5080170b59a4c445a3c1b69b99152e8eec7eba7080ab447cc6f9c6bd8f69df2b18ee9416de44a6ed88009a200ad26e89275f6230339330d12314
and harmonize them as follows
- s/outgoing/automatic outbound/
- s/Incoming/Inbound and manual/ (are not affected by this option.)
- s/only through network/only to network/
- s/this option. This option/this option. It/
- s/network types/networks/
and also pick up a few nits in doc/p2p-bad-ports.md
7d64ea4a01 net: only assume all local addresses if listening on any (Vasil Dimov)
0cfc0cd322 net: fix GetListenPort() to derive the proper port (Vasil Dimov)
f98cdcb357 net: pass Span by value to CaptureMessage() (Vasil Dimov)
3cb9d9c861 net: make CaptureMessage() mockable (Vasil Dimov)
43868ba416 timedata: rename variables to match the coding style (Vasil Dimov)
60da1eaa11 timedata: make it possible to reset the state (Vasil Dimov)
Pull request description:
`GetListenPort()` uses a simple logic: "if `-port=P` is given, then we
must be listening on `P`, otherwise we must be listening on `8333`".
This is however not true if `-bind=` has been provided with `:port` part
or if `-whitebind=` has been provided. Thus, extend `GetListenPort()` to
return the port from `-bind=` or `-whitebind=`, if any.
Also, if `-bind=` is provided then we would bind only to a particular address
and should not add all the other addresses of the machine to the list of
local addresses.
Fixes https://github.com/bitcoin/bitcoin/issues/20184
ACKs for top commit:
sipa:
utACK 7d64ea4a01. I didn't review the tests in detail.
jonatack:
re-ACK 7d64ea4a01 per `git range-diff 08bcfa27 35ec977 7d64ea4`, changes are rebase-only, light re-review, re-ran the new tests locally
Tree-SHA512: 45135ab9c0ec3cc8c83e3b3e58a1c1f77eaeaba00618d54f1010db1d23d6db7d9c0dc7807e72ebc34e8b2d0e91f1e0d0e9239d13b90f1bdce8be84459e7837f0
If `-bind=` is provided then we would bind only to a particular address
and should not add all the other addresses of the machine to the list of
local addresses.
Fixes https://github.com/bitcoin/bitcoin/issues/20184 (case 4.)
`GetListenPort()` uses a simple logic: "if `-port=P` is given, then we
must be listening on `P`, otherwise we must be listening on `8333`".
This is however not true if `-bind=` has been provided with `:port` part
or if `-whitebind=` has been provided. Thus, extend `GetListenPort()` to
return the port from `-bind=` or `-whitebind=`, if any.
Fixes https://github.com/bitcoin/bitcoin/issues/20184 (cases 1. 2. 3. 5.)
Rename `CaptureMessage()` to `CaptureMessageToFile()` and introduce a
`std::function` variable called `CaptureMessage` whose value can be
changed by unit tests, should they need to inspect message contents.
Carl Dong