Running Bitcoin Core on unsupported OSes may expose users to security
issues.
macOS Monterey 12 received its final security update (12.7.6) on July
2024. Apple classifies the hardware that can run macOS 12 at most as
"obsolete worldwide".
fa2b7d8d6b Remove redundant unterminated-logprintf tidy check (MarcoFalke)
bbbb2e43ee log: Enforce trailing newline, Remove redundant m_started_new_line (MarcoFalke)
Pull request description:
There are many problems around missing a trailing newline while logging:
* All log lines are currently terminated by a trailing newline. This means any runtime code trying to handle a "missing" newline is currently dead code.
* Leaving a line unterminated is racy and can cause content corruption by mixing log lines from different sources.
* It requires extra code like `m_started_new_line` to keep track of, which is annoying and pointless to maintain, because it is currently dead code, see https://github.com/bitcoin/bitcoin/pull/30386#discussion_r1684380835.
* It requires a standalone `unterminated-logprintf` clang-tidy plugin, which is unmaintained (no one updated it for the new log function names), probably harder to maintain than normal C++ code (because it requires clang AST matcher knowledge), brittle (it can fail to detect issues at any time, if it goes out-of-sync, or be explicitly disabled via `NOLINT`), and annoying for devs (it is slow and intricate to run locally and thus only effectively run on CI or via the CI scripts).
Fix all issues by enforcing the trailing newline in logs directly in the code. Then remove all the other stuff.
This refactor does not change behavior.
ACKs for top commit:
stickies-v:
re-ACK fa2b7d8d6b
achow101:
ACK fa2b7d8d6b
ryanofsky:
Code review ACK fa2b7d8d6b. Just comment and test cleanup since last review
Tree-SHA512: 10ed420f6c2fdb0f491d6c880be8dd2e8beef628f510adebadf4c3849d9f5e28906519d5cbaeb295f4c7c1b07c4c88a9905b3cfe30fee3a2c91ac9fd24ae6755
fd38711217 ci: make CI job fail when check-deps.sh script fails (Ryan Ofsky)
d51edecddc common: move pcp.cpp and netif.cpp files from util to common library since they depend on netaddress.cpp (Ryan Ofsky)
Pull request description:
Move util/pcp.cpp and util/netif.cpp to common/ because they depend on netaddress.cpp which is part of the common library. This was causing check-deps.sh script to fail as reported by _fanquake_ in https://github.com/bitcoin/bitcoin/pull/30415#issuecomment-2385475097.
Also make CI fail when the `check-deps.sh` script fails. Previously it would output errors but not cause the job to fail (which was not intended).
ACKs for top commit:
Sjors:
utACK fd38711217
laanwj:
Untested ACK fd38711217
achow101:
ACK fd38711217
tdb3:
ACK fd38711217
Tree-SHA512: 06316e68617ded7d96d540c9934b08cf9fbba5ff5e7f54d7a0c0a9087a26bf8adc97e9e8c39a2bfd3da34e27f3652b1531ec6136a2c69393ae0b26585abadb6b
Previously the check-deps.sh would write information about unexpected
dependencies to stderr, but return exit code 0, so the error would be ignored
by CI. Now it will return code 1 and cause CI to fail if unexpected
dependencies are detected.
a9964c0444 doc: Updating docs from autotools to cmake (kevkevinpal)
Pull request description:
A bit of a followup from https://github.com/bitcoin/bitcoin/pull/30840
- In this change the documentation where we refer to the `./configure` script which is now gone and have converted the configure params to use the `cmake` equivalent.
ACKs for top commit:
maflcko:
ACK a9964c0444
jonatack:
utACK a9964c0444
jarolrod:
ACK a9964c0444
tdb3:
ACK a9964c0444
pablomartin4btc:
re-ACK a9964c0444
Tree-SHA512: f7ed20b8ad61f028c0d242b9cc70650d8da63057d4a8f7da88f0117a8d3241c5fe8fcf19d56ec82088160b9fee9b175fe9f64e5a845260d3696dc7e94bfdd0bd
replaced --enable-debug with -DCMAKE_BUILD_TYPE=Debug in developer-notes
replaced --enable-multiprocess with -DWITH_MULTIPROCESS=ON
replaced --disable-zmq with -DWITH_ZMQ=OFF
3ae35b427f ci: run check-deps.sh as part of clang-tidy job (Ryan Ofsky)
0aaa1298a0 contrib: fix check-deps.sh when libraries do not import symbols (Ryan Ofsky)
3c99f5a38a contrib: fix check-deps.sh to check for weak symbols (Ryan Ofsky)
86c80e9cf2 contrib: make check-deps.sh script work with cmake (Ryan Ofsky)
Pull request description:
Fix check-deps.sh to check for weak symbols so it can detect when an exported template function like is used from another library.
Also update the script to work with cmake and configure it to run as part of CI.
Problem was reported by hebasto in https://github.com/bitcoin/bitcoin/pull/29015#issuecomment-2209258843
ACKs for top commit:
TheCharlatan:
Re-ACK 3ae35b427f
hebasto:
ACK 3ae35b427f, I have reviewed the code and it looks OK. Also I've tested it locally.
Tree-SHA512: c3b58175450b675e6e848549b81bcfe42930ea9bcd693063867ce3f0ac3999c98cd2c3e961f163ff06641e8288f3a4e81530936a296a83d45d33364f27489521
fafdb7df34 lint: Speed up flake8 checks (MarcoFalke)
faf17df7fb lint: Document missing py_lint dependency (MarcoFalke)
faebeb828f lint: Remove python whitespace and shadowing lint rules (MarcoFalke)
7777047835 lint: Remove python lint rules that are SyntaxError (MarcoFalke)
faaf3e53f0 test: [refactor] Fix F841 flake8 (MarcoFalke)
444421db69 test: [refactor] Fix E714 pycodestyle (MarcoFalke)
Pull request description:
The checks have many issues:
* Some checks that could in theory hide bugs are not applied -> Fix them and apply them going forward
* Some checks are redundant Python 2 checks, or of low value -> Remove them
* The checks are slow -> Speed them up from ~10 seconds to about ~20 milliseconds
ACKs for top commit:
davidgumberg:
review and tested reACK fafdb7df34
kevkevinpal:
ACK [fafdb7d](fafdb7df34)
achow101:
ACK fafdb7df34
Tree-SHA512: a0488b722cfaf7071bd6848cd3be002e0b6c38af80d8b5cbb08613c0b174ef63277289f960db8ac31adb09fe563a4973203b8fb10b83cbcfdc6f0ef39bd04410
Script was failing when called on libraries that do not import symbols, because
bash pipefail option was specified, and grep was used in some pipelines to
filter symbols, and grep returns status 1 when it doesn't match any lines. This
could cause the script to fail on some systems and configurations, such as the
clang-tidy CI configuration
https://cirrus-ci.com/task/4801670352207872?logs=ci#L6191 where the
libbitcoin_crypto_x86_shani.a library does not import symbols.
Fix check-deps.sh to check for weak symbols so it can detect when an exported
template function is used from another library.
In a previous version of this commit, this change caused an invalid dependency
in the consensus library on the TryParseHex template function from the util
library to be detected, and a suppression was added here. But #30377 removed
the invalid dependency so the suppression is no longer needed.
The invalid dependency and problem detecting weak symbol usage was originally
reported by Hennadii Stepanov in
https://github.com/bitcoin/bitcoin/pull/29015#issuecomment-2209258843
94b0adcc37 rpc, refactor: Prevent potential race conditions in dumptxoutset (Fabian Jahr)
e868a6e070 doc: Improve assumeutxo guide and add more docs/comments (Fabian Jahr)
b29c21fc92 assumeutxo: Remove devtools/utxo_snapshot.sh (Fabian Jahr)
20a1c77aa7 contrib: Remove test_utxo_snapshots.sh (Fabian Jahr)
8426850352 test: Test for dumptxoutset at specific height (Fabian Jahr)
993cafe7e4 RPC: Add type parameter to dumptxoutset (Fabian Jahr)
fccf4f91d2 RPC: Extract ReconsiderBlock helper (Fabian Jahr)
446ce51c21 RPC: Extract InvalidateBlock helper (Fabian Jahr)
Pull request description:
This adds a height parameter to the `dumptxoutset` RPC. This internalizes the workflow that was previously done by scripts: roll back the chain to the height we actually want the snapshot from, create the snapshot, roll forward to the real tip again.
The nice thing about internalizing this functionality is that we can write tests for the code and it gives us more options to make the functionality robust. The shell scripts we have so far will be more cumbersome to maintain in the long run, especially since we will only notice later when we have broken them. I think it's safe to remove these `test_utxo_snapshots.sh` as well when we have this option in `dumptxoutset` because we have also added some good additional functional test coverage for this functionality.
ACKs for top commit:
Sjors:
re-utACK 94b0adcc37
achow101:
ACK 94b0adcc37
mzumsande:
ACK 94b0adcc37
pablomartin4btc:
re-ACK 94b0adcc37
Tree-SHA512: a4c9af5f687d1ca7bfb579a36f363882823386b5fa80c05de531b05a2782b5da6ff5baf3ada4bca8f32f63975d86f1948175abed9affe51fc958472b5f838dab
5b4f34006d devtools, utxo-snapshot: Fix block height out of range (pablomartin4btc)
Pull request description:
<details>
<summary>Fixing a <a href="https://github.com/bitcoin/bitcoin/pull/28553#pullrequestreview-2251032570">bug</a> in <code>utxo_snapshot.sh</code>.</summary>
```
/contrib/devtools/utxo_snapshot.sh 840000 snapshot2.dat ./src/bitcoin-cli -datadir=${AU_DATADIR}
Do you want to disable network activity (setnetworkactive false) before running invalidateblock? (Y/n):
Disabling network activity
false
error code: -8
error message:
Block height out of range
```
And the user will see the following in the node and it would stay there if not reset:
```
2024-08-21T14:44:13Z UpdateTip: new best=00000000000000afa0cd000a16e244f56032735d41acd32ac00337aceb2a5240 height=235382 version=0x00000002 log2_work=69.987697 tx=17492185 date='2013-05-09T23:54:32Z' progress=0.016219 cache=71.0MiB(571085txo)
2024-08-21T14:44:13Z UpdateTip: new best=0000000000000087c5e0b820afff496b95ba44ad64640c73b234d3261d3f99d2 height=235383 version=0x00000002 log2_work=69.987750 tx=17492341 date='2013-05-09T23:54:47Z' progress=0.016219 cache=71.0MiB(571291txo)
2024-08-21T14:44:13Z UpdateTip: new best=000000000000014a4b5fddf3c8abb6209247255ca9e8df786b271dd1b2ac82a6 height=235384 version=0x00000002 log2_work=69.987804 tx=17492344 date='2013-05-10T00:20:18Z' progress=0.016219 cache=71.0MiB(571297txo)
2024-08-21T14:44:13Z SetNetworkActive: false
```
</details>
This is a "temporary" fix until #29553 gets merged, which will remove the script entirely.
Handle the "Block height out of range" error gracefully by checking if the node has synchronized to or beyond the required block height, otherwise without this validation the node would keep the network disabled if the user selected that option.
<details>
<summary>Provide a user-friendly message if the block height is out of range and exit the script cleanly.</summary>
```
/contrib/devtools/utxo_snapshot.sh 840000 snapshot2.dat ./src/bitcoin-cli -datadir=${AU_DATADIR}
Error: The node has not yet synchronized to block height 840001.
Please wait until the node has synchronized past this block height and try again.
```
</details>
ACKs for top commit:
achow101:
ACK 5b4f34006d
fjahr:
tACK 5b4f34006d
Tree-SHA512: 2b71286b627872d7cfdb367e29361afa3806a7ef9d65075b93892b735ff2ab729069e2f7259d30262909e73cef17fb7dca231615cc1863968cd042f4a2a4f901
Handle the Block height out of range error gracefully by checking if
the node has synchronized to or beyond the required block height,
otherwise without this validation the node would keep the network
disabled if the user selected that option.
Provide a user-friendly message if the block height is out of range
and exit the script cleanly.
chainparams.cpp - workaround for MSVC bug triggering C7595 - Calling consteval constructors in initializer lists fails, but works on GCC (13.2.0) & Clang (17.0.6).
If the binaries don't exist, the Guix build has failed for some other
reason.
There's no need to check for unknown architectures, or executable
formats, as the only ones that could be built are those that we've
configured toolchains for in Guix.
We've also been doing this inconsistently across the two scripts.
9010b1343b contrib: c++ify test stubs after switching to c++ compilers (Cory Fields)
261f770333 contrib: rename cc to cxx in binary checking scripts (Cory Fields)
a38c960005 contrib: use c++ rather than c for binary tests (Cory Fields)
Pull request description:
From hebasto's CMake repo. See discussion here: https://github.com/hebasto/bitcoin/pull/252#discussion_r1664657488
Use CXX/CXXFLAGS rather than CC/CFLAGS to test our actual compiler for binary checks rather than the one we only forward to secp256k1.
ACKs for top commit:
hebasto:
ACK 9010b1343b.
fanquake:
ACK 9010b1343b
Tree-SHA512: 7b8788d7d3760103062eff10056c995e1ad14c0c487d9414683ad54d816c255d0ca751f4d0e2d2ad7f9e8a7101d8c7f1e9333fa5b137558ed68fa593c4b4ce6d
34c9cee380 clang-tidy: add check for non-trivial thread_local vars (Cory Fields)
Pull request description:
Forbid thread_local vars with non-trivial destructors.
This is a follow-up from: https://github.com/bitcoin/bitcoin/pull/30095#discussion_r1608423170
ACKs for top commit:
maflcko:
ACK 34c9cee380
TheCharlatan:
Re-ACK 34c9cee380
Tree-SHA512: 3a798607fb189a5bbc714ed6e86dea462fe29d366b790e96d10a7b4ffcf1f194da9b8f4cd0b82154408709b8e3c58d3f613d6311903bd65a76d8b556ab230d21
b5fc6d46a3 guix: use glibc 2.31 (fanquake)
Pull request description:
Set minimum required glibc to 2.31.
The glibc 2.31 branch is still maintained: https://sourceware.org/git/?p=glibc.git;a=shortlog;h=refs/heads/release/2.31/master.
Remove the stack-protector check from test-security-check, as the test
no-longer fails, and given the control we have of the end, the actual
security-check test seems sufficient (this might also be applied to some
of the other checks).
Drops runtime support for Ubuntu Bionic 18.04 and RHEL-8 from the release binaries.
ACKs for top commit:
TheCharlatan:
ACK b5fc6d46a3
Tree-SHA512: ba7e727240fa0ebebfb8b749024c71cbfdec37c33b39627866d78f9318ccdc687fd5103a63ee0e98cf809d9954dde56b1b305691c33d1de275ed0519f716c921
c7376babd1 doc: Clarify distinction between util and common libraries in libraries.md (Ryan Ofsky)
4f74c59334 util: Move util/string.h functions to util namespace (Ryan Ofsky)
4d05d3f3b4 util: add TransactionError includes and namespace declarations (Ryan Ofsky)
680eafdc74 util: move fees.h and error.h to common/messages.h (Ryan Ofsky)
02e62c6c9a common: Add PSBTError enum (Ryan Ofsky)
0d44c44ae3 util: move error.h TransactionError enum to node/types.h (Ryan Ofsky)
9bcce2608d util: move spanparsing.h to script/parsing.h (Ryan Ofsky)
6dd2ad4792 util: move spanparsing.h Split functions to string.h (Ryan Ofsky)
23cc8ddff4 util: move HexStr and HexDigit from util to crypto (TheCharlatan)
6861f954f8 util: move util/message to common/signmessage (Ryan Ofsky)
cc5f29fbea build: move memory_cleanse from util to crypto (Ryan Ofsky)
5b9309420c build: move chainparamsbase from util to common (Ryan Ofsky)
ffa27af24d test: Add check-deps.sh script to check for unexpected library dependencies (Ryan Ofsky)
Pull request description:
Remove `fees.h`, `errors.h`, and `spanparsing.h` from the util library. Specifically:
- Move `Split` functions from `util/spanparsing.h` to `util/string.h`, using `util` namespace for clarity.
- Move remaining spanparsing functions to `script/parsing.h` since they are used for descriptor and miniscript parsing.
- Combine `util/fees.h` and `util/errors.h` into `common/messages.h` so there is a place for simple functions that generate user messages to live, and these functions are not part of the util library.
Motivation for this change is that the util library is a dependency of the kernel, and we should remove functionality from util that shouldn't be called by kernel code or kernel applications. These changes should also improve code organization and make functions easier to discover. Some of these same moves are (or were) part of #28690, but did not help with code organization, or made it worse, so it is better to move them and clean them up in the same PR so code only has to change one time.
ACKs for top commit:
achow101:
ACK c7376babd1
TheCharlatan:
Re-ACK c7376babd1
hebasto:
re-ACK c7376babd1.
Tree-SHA512: 5bcef16c1255463b1b69270548711e7ff78ca0dd34e300b95e3ca1ce52ceb34f83d9ddb2839e83800ba36b200de30396e504bbb04fa02c6d0c24a16d06ae523d
Set minimum required glibc to 2.31.
The glibc 2.31 branch is still maintained:
https://sourceware.org/git/?p=glibc.git;a=shortlog;h=refs/heads/release/2.31/master.
Remove the stack-protector check from test-security-check, as the test
no-longer fails, and given the control we have of the end, the actual
security-check test seems sufficient (this might also be applied to some
of the other checks).
Drops runtime support for Ubuntu Bionic 18.04 and RHEL-8 from the release binaries.
Adjust the security check for:
ld64.lld: warning: Option `-allow_stack_execute' is not yet implemented.
ld64.lld: error: -fixup_chains is incompatible with -no_pie
and to account for the embedding of LLVMs version number.
Hennadii Stepanov