32e2ffc393 Remove the syscall sandbox (fanquake)
Pull request description:
After initially being merged in #20487, it's no-longer clear that an internal syscall sandboxing mechanism is something that Bitcoin Core should have/maintain, especially when compared to better maintained/supported alterantives, i.e [firejail](https://github.com/netblue30/firejail).
There is more related discussion in #24771.
Note that given where it's used, the sandbox also gets dragged into the kernel.
If it's removed, this should not require any sort of deprecation, as this was only ever an opt-in, experimental feature.
Closes#24771.
ACKs for top commit:
davidgumberg:
crACK 32e2ffc393
achow101:
ACK 32e2ffc393
dergoegge:
ACK 32e2ffc393
Tree-SHA512: 8cf71c5623bb642cb515531d4a2545d806e503b9d57bfc15a996597632b06103d60d985fd7f843a3c1da6528bc38d0298d6b8bcf0be6f851795a8040d71faf16
This replaces (but does not collide with) the previous bind_on_load. There
is technically no need to opt-in to this functionality as long as >= MacOS 11.0
is being targetted, but it will be helpful to see in the logs.
After initially being merged in #20487, it's no-longer clear that an
internal syscall sandboxing mechanism is something that Bitcoin Core
should have/maintain, especially when compared to better
maintained/supported alterantives, i.e firejail.
Note that given where it's used, the sandbox also gets dragged into the
kernel.
There is some related discussion in #24771.
This should not require any sort of deprecation, as this was only ever
an opt-in, experimental feature.
Closes#24771.
Disable boost multi index safe mode by default when configuring with
--enable-debug.
This option can cause transactions to take a long time to be accepted
into the mempool under certain conditions; iterator destruction takes
O(n) time vs O(1) as they are stored in a singly linked list. See
27586 for more information.
Re-enable it on the CI builds which previously had it enabled.
Re-enable it on the msan fuzz target so that we have fuzz tasks testing
with it enabeld and disabled in this repo.
5228223e1f ci: remove MSAN getrandom syscall workaround (fanquake)
d5e06919db random: switch to using getrandom() directly (fanquake)
c2ba3f5b0c random: add [[maybe_unused]] to GetDevURandom (fanquake)
c13c97dbf8 random: getentropy on macOS does not need unistd.h (fanquake)
Pull request description:
This requires a linux kernel of `3.17`+, which seems entirely
reasonable. `3.17` went EOL in 2015, and the last supported `3.x` kernel
(`3.16`) went EOL > 4 years ago, in 2020. For reference, the current
oldest maintained kernel is `4.14` (released 2017, going EOL Jan 2024).
Support for `getrandom()` (and `getentropy()`) was added to
glibc `2.25` https://sourceware.org/legacy-ml/libc-alpha/2017-02/msg00079.html:
> * The getentropy and getrandom functions, and the <sys/random.h> header
file have been added.
and we already require `2.27` or later.
All that being said, I don't think you would encounter a current day (+~6 months from now)
system, running with kernel headers older than 3.17 (released 2014) but also having a
glibc of 2.27+ (released 2018)?
Removing this (our only) use of `syscall()` also means we can drop a workaround in our MSAN jobs.
If this is merged, I'll drop the [same workaround in oss-fuzz](25946a5448/projects/bitcoin-core/build.sh (L49-L56)).
ACKs for top commit:
josibake:
ACK 5228223e1f
hebasto:
ACK 5228223e1f, I've tested build system changes on Ubuntu 22.04 and macOS Monterey 12.6.6 (x86_64).
Tree-SHA512: cc978e08510c461b875ca8c08ae176b4519fa1108f0efd74dcb7474518945357e0184e54423282c9a496de195e4ddc3e221ee78623bd63e24c50cc86acdf32e2
fa5831bd6f build: Do not define `ENABLE_ZMQ` when ZMQ is not available (Hennadii Stepanov)
Pull request description:
A new behavior is consistent with the other optional dependencies.
The source code contains `#if ENABLE_ZMQ` lines only:
```
$ git grep ENABLE_ZMQ -- src/*.cpp
src/init.cpp:#if ENABLE_ZMQ
src/init.cpp:#if ENABLE_ZMQ
src/init.cpp:#if ENABLE_ZMQ
src/init.cpp:#if ENABLE_ZMQ
src/init.cpp:#if ENABLE_ZMQ
```
Change in description line -- "Define to 1..." --> "Define this symbol.." -- is motivated by the fact that the actual value of the defined `ENABLE_ZMQ` macro does not matter at all.
Related to:
- https://github.com/bitcoin/bitcoin/issues/16419
- https://github.com/bitcoin/bitcoin/pull/25302
ACKs for top commit:
TheCharlatan:
ACK fa5831bd6f
jarolrod:
ACK fa5831bd6f
Tree-SHA512: 5e72ff0d34c4b33205338daea0aae8d7aa0e48fd633e21af01af32b7ddb0532ef68dd3dd74deb2c1d2599691929617e8c09676bcbaaf7d669b88816f866f1db2
This requires a linux kernel of 3.17.0+, which seems entirely
reasonable. 3.17 went EOL in 2015, and the last supported 3.x kernel
(3.16) went EOL > 4 years ago, in 2020. For reference, the current
oldest maintained kernel is 4.14 (released 2017, EOL Jan 2024).
Support for `getrandom()` (and `getentropy()`) was added to
glibc 2.25, https://sourceware.org/legacy-ml/libc-alpha/2017-02/msg00079.html,
and we already require 2.27+.
All that being said, I don't think you would encounter a current day
system, running with kernel headers older than 3.17 (released 2014) but
also having a glibc of 2.27+ (released 2018).
Remove it. Make this change, so in a future commit, we can
combine #ifdefs, and avoid duplicate <sys/random.h> includes once we
switch to using getrandom directly.
Also remove the comment about macOS 10.12. We already require macOS >
10.15, so it is redundant.
b53cab0083 build: Detect USDT the same way how it is used in the code (Hennadii Stepanov)
Pull request description:
In the code we do not use string literals.
Also a check for `DTRACE_PROBE7` macro has been added as not all systems define`DTRACE_PROBE{6,7,8,9,10,11,12}` macros (e.g., FreeBSD).
ACKs for top commit:
0xB10C:
ACK b53cab0083
Tree-SHA512: 74f49424d57bf1929f2b09edba1449cef5a1a2448161952da35302343f3003d5bedeab1417e166b656c5f629303e2de888550b1219e886a1b991b12b9c880794
libtool gets a false-positive from the warning produced by lld -single_module
because it is already the default and unneeded.
Skip the check unconditionally for Darwin linkers.
glibc 2.33 introduced a new fortification level, _FORTIFY_SOURCE=3.
Which improves the coverage of cases where _FORTIFY_SOURCE can use _chk
functions. For example, using GCC 13 and glibc 2.36 (Fedora Rawhide),
compiling master:
```bash
nm -C src/bitcoind | grep _chk
U __fprintf_chk@GLIBC_2.17
U __memcpy_chk@GLIBC_2.17
U __snprintf_chk@GLIBC_2.17
U __sprintf_chk@GLIBC_2.17
U __stack_chk_fail@GLIBC_2.17
U __stack_chk_guard@GLIBC_2.17
U __vsnprintf_chk@GLIBC_2.17
objdump -d src/bitcoind | grep "_chk@plt" | wc -l
33
```
vs this branch:
```bash
nm -C src/bitcoind | grep _chk
U __fprintf_chk@GLIBC_2.17
U __memcpy_chk@GLIBC_2.17
U __memset_chk@GLIBC_2.17
U __snprintf_chk@GLIBC_2.17
U __sprintf_chk@GLIBC_2.17
U __stack_chk_fail@GLIBC_2.17
U __stack_chk_guard@GLIBC_2.17
U __vsnprintf_chk@GLIBC_2.17
objdump -d src/bitcoind | grep "_chk@plt" | wc -l
61
```
Usage of level 3 requires LLVM/Clang 9+, or GCC 12+. Older
compilers/glibc will still use _FORTIFY_SOURCE=2. For example, in the
glibc we currently use for Linux release builds (2.24), FORTIFY_LEVEL is
determined using the following:
```c
```
so any value > 1 will turn on _FORTIFY_SOURCE=2.
https://sourceware.org/pipermail/libc-alpha/2021-February/122207.htmlhttps://developers.redhat.com/blog/2021/04/16/broadening-compiler-checks-for-buffer-overflows-in-_fortify_source
Even though all other targets are disabled, we still need Boost CPPFLAGS
(use_boost) to compile. This currently works everywhere, except on arm
macOS (where the include path is pretty non-standard), because
generally, the Boost include path is generic, i.e `/usr/include`.
d4c59da8d6 build: Avoid `BOOST_NO_CXX98_FUNCTION_BASE` macro redefinition (Hennadii Stepanov)
Pull request description:
With GCC 12 and Boost 1.81 (from depends) having multiple warnings:
```
In file included from /home/hebasto/bitcoin/depends/x86_64-pc-linux-gnu/include/boost/config.hpp:48:
/home/hebasto/bitcoin/depends/x86_64-pc-linux-gnu/include/boost/config/stdlib/libstdcpp3.hpp:397:9: warning: 'BOOST_NO_CXX98_FUNCTION_BASE' macro redefined [-Wmacro-redefined]
#define BOOST_NO_CXX98_FUNCTION_BASE
^
<command line>:8:9: note: previous definition is here
#define BOOST_NO_CXX98_FUNCTION_BASE 1
^
1 warning generated.
```
This PR fixes those warnings.
Defining of the `BOOST_NO_CXX98_FUNCTION_BASE` macro was introduced in https://github.com/bitcoin/bitcoin/pull/25436, but since https://github.com/boostorg/config/pull/430, it is required to check it before adding.
ACKs for top commit:
fanquake:
ACK d4c59da8d6 - it works now.
Tree-SHA512: 53b9ddcf8dad729638ed41251e30c80f2d7d1ae3ffe47466865834f1f10184fe0881abeb339b3e46c270c3eb11fb63d19ab12cc9461bf5c2be12b4763c1b1c34
d51f0fa4b7 doc: add release notes for 26896 (fanquake)
2b248798d9 build: remove --enable-upnp-default from configure (fanquake)
02f5a5e7b5 build: remove --enable-natpmp-default from configure (fanquake)
25a0e8ba0b Remove configure-time setting of DEFAULT_UPNP (fanquake)
06562e5fa7 Remove configure-time setting of DEFAULT_NATPMP (fanquake)
Pull request description:
This PR removes the `--enable-upnp-default` and `--enable-natpmp-default` options from configure.
It's odd to me that we maintain configure-time options for setting the default port-forwarding runtime state (but no other similar options), and I'm not sure what use-case it satisfies, that can't be achieved by multiple other means. I also doubt that we'll ever restart using these in release builds, or turning on any of this by default.
I think the only scenario these options would be used is when you want to compile your own binaries (we don't use them in Guix), with port-forwarding on by default, but otherwise can't or don't want to use a `.conf` file, can't or don't want to pass command line options at runtime, and also don't want to modify the source code?
ACKs for top commit:
hebasto:
ACK d51f0fa4b7, rebased and comments have been addressed since my recent [review](https://github.com/bitcoin/bitcoin/pull/26896#pullrequestreview-1273910740).
TheCharlatan:
ACK d51f0fa4b7
Tree-SHA512: 481decd8bddd8b03b7319591e3acf189f7b6b96c9a9a8c5bc1a3f8ec00d0b8f9b52d2f5c28a298a2ec947cfe9611cfd184e393ccb2e4e21bfce86ca7d4de60d3
0f883df7a5 build: fix configuring with only bitcoin-util (fanquake)
Pull request description:
Fixes the issue presented in #25037 in a single (easily backportable) diff, with no additional refactoring/changes.
Can be tested with:
```bash
./configure \
--disable-tests \
--disable-bench \
--without-libs \
--without-daemon \
--without-gui \
--disable-fuzz-binary \
--without-utils \
--enable-util-util
```
ACKs for top commit:
TheCharlatan:
tACK 0f883df7a5
hebasto:
ACK 0f883df7a5, tested on Ubuntu 22.04.
Tree-SHA512: 3682712405c360852c4edd90c171e21302154bf8789252c64083974a5c873cf04d97e8721c7916d5b2dafa6acd2b8dc32deecf550e90e03bcbbabbbbf75ce959
2022917223 Add secp256k1_selftest call (Pieter Wuille)
3bfca788b0 Remove explicit enabling of default modules (Pieter Wuille)
4462cb0498 Adapt to libsecp256k1 API changes (Pieter Wuille)
9d47e7b71b Squashed 'src/secp256k1/' changes from 44c2452fd3..21ffe4b22a (Pieter Wuille)
Pull request description:
Now that libsecp256k1 has a release (https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-December/021271.html), update the subtree to match it.
The changes themselves are not very impactful for Bitcoin Core, but include:
* It's no longer needed to specify whether contexts are for signing or verification or both (all contexts support everything), so make use of that in this PR.
* Verification operations can use the static context now, removing the need for some infrastructure in pubkey.cpp to make sure a context exists.
* Most modules are now enabled by default, so we can drop explicit enabling for them.
* CI improvements (in particular, MSVC and more recent MacOS)
* Introduction of an internal int128 type, which has no effect for GCC/Clang builds, but enables 128-bit multiplication in MSVC, giving a ~20% speedup there (but still slower than GCC/Clang).
* Release process changes (process documentation, changelog, ...).
ACKs for top commit:
Sjors:
ACK 2022917223, but 4462cb0498 could use more eyes on it.
achow101:
ACK 2022917223
jonasnick:
utACK 2022917223
Tree-SHA512: 8a9fe28852abe74abd6f96fef16a94d5a427b1d99bff4caab1699014d24698aab9b966a5364a46ed1001c07a7c1d825154ed4e6557c7decce952b77330a8616b
Fixes the issue presented in #25037 in a single (easily backportable)
diff, with no additional refactoring/changes.
Can be tested with:
```bash
./configure \
--disable-tests \
--disable-bench \
--without-libs \
--without-daemon \
--without-gui \
--disable-fuzz-binary \
--without-utils \
--enable-util-util
```
These headers are already included in a default set which are checked
early during configure.
We already use at least sys/types.h and unistd.h unconditionally in
configure.
98868633d1 Bugfix: configure: bitcoin-{cli,tx,util} don't need UPnP, NAT-PMP, or ZMQ (Luke Dashjr)
Pull request description:
As with #23345, these other tools likewise don't use various deps.
ACKs for top commit:
achow101:
ACK 98868633d1
Tree-SHA512: 4be056b8e0c9f69834229aa257187457de1bc34214d320b770834e21ecc1f0ca7aa7b9689fba525928947bfabbb461528795f709014fb9618b82f088fe64f271
d216d714aa Revert "build: Use Homebrew's sqlite package if it is available" (fanquake)
Pull request description:
This reverts ee7b84e63c from #20527.
That change was made without any rationale, maybe other than, a brew
installed version might be newer, and that's "better". However when
building from source on macOS, it just results in drastically worse
performance, and issues / confusion like #25724.
The difference in performance can be observed using the example from https://github.com/bitcoin/bitcoin/issues/25724#issuecomment-1213554922,
but minified i.e:
```bash
time src/bitcoin-cli createwallet speedy true
time src/bitcoin-cli importdescriptors '[
{"desc":"raw(00145846369f3d6ba366d6f5a903fb5cf4dca3763c0e)#k9wh6v62","timestamp":"now"},
{"desc":"raw(001420800aabf13f3a4c4ce3ce4c66cecf1d17f21a6e)#6m0hlfh4","timestamp":"now"},
{"desc":"raw(0014c6bf9715e06d73ebf9b3b02d5cc48d24d8bbabc1)#wyavh36r","timestamp":"now"},
{"desc":"raw(00141ba7807b3f46af113beaea5c698428ce7138cd8a)#jctdsups","timestamp":"now"},
{"desc":"raw(00140c1bd27f10fff01b36ddf3c1febaa1acff19b080)#9s6nc3pk","timestamp":"now"},
{"desc":"raw(00141226e31987e4bc2e63c0ee12908f675e40464b20)#9pp7qm39","timestamp":"now"},
{"desc":"raw(0014f73f149f7503960a5e849c6ee7a8a8c336f631cb)#qtkxv9fc","timestamp":"now"},
{"desc":"raw(0014c8ccb4d81ffc769fc5fdd8d7eed69b0e0cae5749)#hn39qayv","timestamp":"now"},
{"desc":"raw(001498565aead2d67a22a6021d55210f2a917fc22169)#6ar3vwsx","timestamp":"now"},
{"desc":"raw(001403013248ac0cd9eabe176cad162cda2a19f771e1)#4m47mukd","timestamp":"now"}
]'
```
Running master, when building from souce and using brew installed
sqlite, this takes ~3.4s. With this PR, the same operation takes ~0.3s.
Resolves the "build from source" portion of #25724. Building from
depends is still not ideal, however I have some other changes that might
help improve things in that case.
Related performance issue reports:
* https://github.com/bitcoindevkit/bdk/issues/749
* https://bitcoin.stackexchange.com/questions/113898/bitcoin-v23-is-10-times-slower-than-v22-on-macos-for-basic-regtest-tests
* https://github.com/bitcoin/bitcoin/issues/25724
* https://github.com/bitcoin/bitcoin/pull/25985#issuecomment-1245942400
ACKs for top commit:
achow101:
ACK d216d714aa
jarolrod:
ACK d216d714aa
hebasto:
ACK d216d714aa, I have reviewed the code and it looks OK, I agree it can be merged. No conflicts with our build [docs](d216d714aa/doc/build-osx.md (descriptor-wallet-support)).
Tree-SHA512: 1bb4b44385b11fa9fe66edd7449278f9e47a6cc679b7111f9adf17db94c34e29c9cceafc917454e134420db40b24b56da29226af6f43e6dbeff822b79b77ed60
We currently perform the same check twice, to put the same set of flags
in two different variables. Split the checks so we test for crc and crypto
extensions independently.
If we don't want to split, we should just delete the second AX_CHECK_COMPILE_FLAG
check, and set ARM_CRC_CXXFLAGS & ARM_CRC_CXXFLAGS at the same time.
We already use a mix of <cstdlib> and stdlib.h unconditionally throughout
the codebase.
Us checking this header also duplicates work already done by autotools.
Currently stdlib.h is checked for 3 times during a ./configure run, after
this change, at least it's only twice.
We already use a mix of <cstdio> and stdio.h unconditionally throughout
the codebase.
Us checking this header also duplicates work already done by autotools.
Currently stdio.h is checked for 3 times during a ./configure run, after
this change, at least it's only twice.
We don't include strings.h anywhere.
This is also already checked for by autoconf, so us checking for it just
means a 3rd existence check during ./configure.
Andrew Chow