42b589d18f scripts: test for MACHO control flow instrumentation (fanquake)
469a5bc4fa build: build Boost with -fcf-protection when targeting Darwin (fanquake)
Pull request description:
Addresses the macOS portion of #21888.
Build Boost with `-fcf-protection` when targeting Darwin. This should be ok, because our cross-compiler (Clang 10) supports the option, and I'd expect all versions of Apple Clang being used to compile Core would also support it. Building Boost with this option is required so that the `main` provided to `test_bitcoin` has instrumentation.
Note that the presence of instrumentation does not mean it will be used, as that is determined at runtime by the CPU.
From the Intel control flow enforcement documentation:
> The ENDBR32 and ENDBR64 instructions will have the same effect as the NOP instruction on Intel 64 processors that do not support CET. On processors supporting CET, these instructions do not change register or flag state. This allows CET instrumented programs to execute on processors that do not support CET. Even when CET is supported and enabled, these NOP–like instructions do not affect the execution state of the program, do not cause any additional register pressure, and are minimally intrusive from power and performance perspectives.
Follow up from #21135.
Guix builds:
```bash
663df8471400f06d4da739e39a886aa17f56a36d66e0ff7cc290686294ef39c9 guix-build-42b589d18fed/output/dist-archive/bitcoin-42b589d18fed.tar.gz
45e841661e1659a634468b6f8c9fb0a7956c31ba296f1fd0c02cd880736d6127 guix-build-42b589d18fed/output/x86_64-apple-darwin18/bitcoin-42b589d18fed-osx-unsigned.dmg
0ea85c99fef35429a5048fa14850bce6b900eaa887aeea419b019852f8d2be78 guix-build-42b589d18fed/output/x86_64-apple-darwin18/bitcoin-42b589d18fed-osx-unsigned.tar.gz
85857a5a4a5d4d3a172d6c361c12c4a94f6505fc12b527ea63b75bfe54ee1001 guix-build-42b589d18fed/output/x86_64-apple-darwin18/bitcoin-42b589d18fed-osx64.tar.gz
```
Gitian builds:
```bash
# macOS:
bdfd677a6b88273a741b433e1e7f554af50cc76b3342d44ab0c441e2b40efc96 bitcoin-42b589d18fed-osx-unsigned.dmg
f3b2d09f3bea7a5cc489b02e8e53dd76a9922338500fae79cad0506655af56f9 bitcoin-42b589d18fed-osx-unsigned.tar.gz
29d5ad5e46bc9fb0056922a8b47c026e5e9f71e6cf447203b74644587d6fb6f7 bitcoin-42b589d18fed-osx64.tar.gz
663df8471400f06d4da739e39a886aa17f56a36d66e0ff7cc290686294ef39c9 src/bitcoin-42b589d18fed.tar.gz
366f8d7a2fc1f3e22cb1018043099126a71ce65380cc27b1c3280cce42d06c98 bitcoin-core-osx-22-res.yml
```
ACKs for top commit:
laanwj:
Code review ACK 42b589d18f
Tree-SHA512: 12cb8d462d64d845b9fe48c5c6978892adff8bf5b5572bb29f35df1f6176e47b32a68bcb6e4883c7d9454e76e8868851005a7325916852a2d0d32659ac7dae3f
b95f7f8ac0 build, qt, refactor: Drop sed commands for win32-g++/qmake.conf (Hennadii Stepanov)
Pull request description:
Such possibility is [available](https://codereview.qt-project.org/c/qt/qtbase/+/165348) since Qt 5.8.0.
ACKs for top commit:
fanquake:
ACK b95f7f8ac0
Tree-SHA512: e56a3d208a6bd5d42c722f8b344010fe7d1b6f7a28486613dfcb03f0403a47cee8476e2366eeaac401a19836cd09f782e8741a1e781ab4d78f72c500a30e4929
The LLVM Clang we use for cross-compilation supports this option, and it's expected
that any builders on macOS will also be using an Apple Clang that supports it.
fa872c9af3 depends: Fix id_string invocations (Carl Dong)
Pull request description:
Closes: #21242
```
Reproduced from depends/Makefile comment:
When invoking a shell, GNU Make special-cases exit code 127 (command not
found) by not capturing the output but instead passing it through. This
is not done for any other exit code.
Therefore, we require a "|| true" to avoid this behaviour when in an
environment where the build_* or host_* may not exist yet.
```
ACKs for top commit:
laanwj:
Concept and light code review ACK fa872c9af3
Tree-SHA512: 9ce88381aec579d956572cf70c4f69dc5a3873f0d2af14a71cf24814192a89452b8280258bed8cca804e4bd2644db056d213ab733df46a10560a47079524d8ac
a5491882a0 build: fix configuring when building depends with NO_BDB=1 (fanquake)
Pull request description:
Currently, if you build depends using `NO_BDB=1` (only sqlite wallets), `./configure` will fail as it still tries to find bdb. i.e:
```bash
make -C depends/ NO_QT=1 NO_BDB=1 NO_UPNP=1 NO_ZMQ=1 NO_NATPMP=1 -j8
...
copying packages: native_b2 boost libevent sqlite
./autogen.sh
./configure --prefix=/home/ubuntu/bitcoin/depends/x86_64-pc-linux-gnu
...
checking for Berkeley DB C++ headers... default
configure: error: Found Berkeley DB other than 4.8, required for portable BDB wallets (--with-incompatible-bdb to ignore or --without-bdb to disable BDB wallet support)
```
This PR fixes the build such that you can build depends, opting out of bdb, without opting out of wallets entirely, and still configure successfully. I think I've tested across most potential configurations. i.e:
```bash
./configure (bdb and sqlite on system)
bdb & sqlite are both are available
./configure --without-bdb (bdb and sqlite on system)
only sqlite
./configure --without-sqlite (bdb and sqlite on system)
only bdb
./configure --disable-wallet (bdb and sqlite on system)
neither bdb or sqlite
depends NO_WALLET=1
./configure --prefix=/bitcoin/depends/x86_64-apple-darwin19.6.0
neither bdb or sqlite
depends NO_BDB=1
./configure --prefix=/bitcoin/depends/x86_64-apple-darwin19.6.0
only sqlite
depends NO_SQLITE=1
./configure --prefix=/bitcoin/depends/x86_64-apple-darwin19.6.0
only bdb
depends
./configure --prefix=/bitcoin/depends/x86_64-apple-darwin19.6.0
bdb and sqlite
```
ACKs for top commit:
laanwj:
Code review ACK a5491882a0
jarolrod:
ACK a5491882a0
Tree-SHA512: baf7d2543a401db0d846095415ff449c04ecfb4a74c734dc51e79453702f9051210daeef686970f11fcffd32cdfadbc58acd54f0706aceecfb3edb0ff17310d7
cf971c9ea0 build: use -isysroot over --sysroot on macOS (fanquake)
Pull request description:
Not only does this seem to be the more correct behaviour when targeting Darwin, but if you use `-isysroot`, Clangs Darwin driver will [infer the deployment target](3e037f8f0e/clang/lib/Driver/ToolChains/Darwin.cpp (L1652)) from the SDK and use other SDK info when parsing arguments to the linker. In the case of [`-platform_version`](3e037f8f0e/clang/lib/Driver/ToolChains/Darwin.cpp (L2656)), which is added if the linker is [new enough](3e037f8f0e/clang/lib/Driver/ToolChains/Darwin.cpp (L342)), the version tuple is constructed from the SDKInfo, and SDKInfo, as far as I can tell, only exists when `-isysroot` has been passed, see [parseSDKSettings](3e037f8f0e/clang/lib/Driver/ToolChains/Darwin.cpp (L1785))
As a result, the SDK version field in the `LC_BUILD_VERSION` command is filled out. i.e when building master:
```bash
cmd LC_BUILD_VERSION
cmdsize 32
platform 1
minos 10.14
sdk n/a
ntools 1
tool 3
version 650.9
```
vs this PR:
```bash
cmd LC_BUILD_VERSION
cmdsize 32
platform 1
minos 10.14
sdk 11.3
ntools 1
tool 3
version 650.9
```
This, from what I understand, will fix the issue we are having with Qt deciding wether or not to enable features like "Dark mode" on macOS, see #21771, however I have not tested that. Thus this is an alternative to #21782.
Our usage of `--sysroot` was added in #17118.
```bash
-isysroot <dir> Set the system root directory (usually /)
```
ACKs for top commit:
Sjors:
tACK cf971c9ea0
hebasto:
re-ACK cf971c9ea0, only rebased and addressed comments since my [previous](https://github.com/bitcoin/bitcoin/pull/21793#pullrequestreview-647321518) review.
Tree-SHA512: f01138179fb85083b5505bbaa48810451098ffa4da5d3c9b673785448790aa76f2e64b2aab6e698f6ee378a21f70626445a3fabee7c61dbfc44e96f3e3964656
c799a19b4b build, qt: No longer need to set QT_RCC_TEST=1 for determinism (Hennadii Stepanov)
Pull request description:
The Qt Resource Compiler (rcc) output order relies on [`QHash`](https://doc.qt.io/qt-5/qhash.html):
> This randomization of `QHash` is enabled by default. Even though programs should never depend on a particular `QHash` ordering, there may be situations where you temporarily need deterministic behavior, for example for debugging or regression testing. To disable the randomization, define the environment variable `QT_HASH_SEED` to have the value 0.
Since #3620 we use `QT_RCC_TEST=1` to achieve a deterministic output.
Since Qt 5.3.1 hash seeding is disabled for all of the bootstrapped tools, including rcc. Therefore, `QT_RCC_TEST=1` is no longer needed.
See commit [5283a6c87beac5a43f612786fefd6e43f2c70bf6](5283a6c87b).
ACKs for top commit:
fanquake:
ACK c799a19b4b
Tree-SHA512: 9d116ac1e8c605ee3e8ed7f618586f0de85d8b06bbbb70fe8c298939ce203d2a7e97264a9afac037179993ab54c5f69a65ebb9ab27ca7f45acb963011bd45743
We now use a script named gen_id to generate the base build_id/host_id.
This solves 2 problems:
1. GNU Make special-casing exit code 127 (command not found) meant that
warnings about missing tools would propagate to the user's terminal
and broke our opportunistic build_id construction.
2. This change ensures that we don't have arbitrary characters in our
make variables that would be misinterpreted by Make.
See comments in depends/Makefile and depends/gen_id for more
information.
Currently, if you build depends using `NO_BDB=1` (only sqlite wallets),
./configure will fail as it still tries to find bdb. i.e:
```bash
checking for Berkeley DB C++ headers... default
configure: error: Found Berkeley DB other than 4.8, required for portable BDB wallets (--with-incompatible-bdb to ignore or --without-bdb to disable BDB wallet support)
```
This PR fixes the build such that you can build depends, opting out of
bdb without opting out of wallets entirely, and still configure
successfully.
7476b46f18 guix: Build dmg as a static binary (Carl Dong)
06d6cf6784 depends: libdmg-hfsplus: Skip CMake RPATH patching (Carl Dong)
65176ab573 guix: Remove codesign_allocate+pagestuff from unsigned tarball (Carl Dong)
ca85679eb4 guix: Use clang-toolchain instead of clang (Carl Dong)
1aec0eda8f guix: Fallback to local build for substitute-enabled Guix users (Carl Dong)
1742f8e12d guix: Add early health check for guix-daemon (Carl Dong)
c1ae726a13 guix: More thoroughly control native toolchain (Carl Dong)
39741128d3 guix: Supply --link-profile (Carl Dong)
d55a1056ee guix: Add troubleshooting documentation entries (Carl Dong)
7f401c953f guix: Adapt guix-build to prelude, restructure hier (Carl Dong)
4eccf063b2 guix: Remove guix-build.sh filename extension (Carl Dong)
7753357a7b guix: Add source-able bash prelude and utils (Carl Dong)
e5b49a01f5 guix: Create windeploy inside distsrc-* (Carl Dong)
3e9982ab38 contrib: Silence git-describe when looking for tag (Carl Dong)
d5a71e9785 guix: Use --cores instead of --max-jobs (Carl Dong)
Pull request description:
This PR addresses a few hiccups encountered by the brave souls who've been experimenting with the Guix scripts:
- Resolves confusion between `--cores=` and `--max-jobs=`
- `guix`'s `--cores=` actually corresponds to make's `--jobs=`, so let's just control `--cores=` with our overridable env var
- `git-describe` will scream `fatal: no tag exactly matches '<hash>'` when looking for a tag, but we don't care, so silence that
- `windeploy/unsigned` should be inside `distsrc-*` and created idempotently (sorry I know this one annoyed people)
- Add troubleshooting documentation to `README.md`
- Add early health check for `guix-daemon` in case user forgot to start a `guix-daemon`
- Depending on configuration, a `--fallback` flag may be needed to tell Guix to not fail if substitutes fail but fallback to building locally
- `codesign_allocate` and `pagestuff` are now unnecessary for codesigning as we're now using `signapple`
A few robustness changes are also included:
- We supply the `--link-profile` flag, as some Guix packages may expect the profile to be available under `$HOME/.guix-profile`
- We now clear and manually set all toolchain-related env vars (e.g. `C*_INCLUDE_PATH`) ourselves, after patching a Qt::moc bug
- We use the native `clang-toolchain` package for darwin builds instead of `clang`, lining up with all our other toolchain packages.
Finally, we restructure the guix building hierarchy such that it looks something like:
```
guix-build-<short-hash-or-version-tag>
├── distsrc-<short-hash-or-version-tag>-${HOST}
│ ├── contrib
│ ├── depends
│ ├── src
│ └── ...
├── distsrc-<short-hash-or-version-tag>-...
└── output
├── dist-archive
│ └── bitcoin-<short-hash-or-version-tag>.tar.gz
├── *-linux-*
│ ├── bitcoin-<short-hash-or-version-tag>-*-linux-*-debug.tar.gz
│ └── bitcoin-<short-hash-or-version-tag>-*-linux-*.tar.gz
├── x86_64-apple-darwin18
│ ├── bitcoin-<short-hash-or-version-tag>-osx64.tar.gz
│ ├── bitcoin-<short-hash-or-version-tag>-osx-unsigned.dmg
│ └── bitcoin-<short-hash-or-version-tag>-osx-unsigned.tar.gz
└── x86_64-w64-mingw32
├── bitcoin-<short-hash-or-version-tag>-win64-debug.zip
├── bitcoin-<short-hash-or-version-tag>-win64-setup-unsigned.exe
├── bitcoin-<short-hash-or-version-tag>-win64.zip
└── bitcoin-<short-hash-or-version-tag>-win-unsigned.tar.gz
```
Separating guix builds by their version identifier (basically namespacing them) allows us to change the layout in the future without worry about potential naming conflicts.
ACKs for top commit:
sipa:
ACK 7476b46f18
laanwj:
ACK 7476b46f18
Tree-SHA512: 0e899aa941aafdf552b2a7e8a08131ee9283180bbef7334439e2461a02aa7235ab7b9ca9c149b80fc5d0a9f4bbd35bc80fcee26197c0836ba8eaf2d86ffa0386
This relatively easy change eliminates all runtime dependencies (except
for the kernel) for dmg, which is the only native build tool that gets
put in our output tarballs.
This allows much more flexibility when constructing the codesigning
environment, and is much more robust.
After reading
https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.70/autoconf.html#Specifying-Target-Triplets,
my understanding is that this change should mostly be a no-op, as
--build defaults to the output of config.guess, however, this may be
slightly more correct
> For historical reasons, whenever you specify --host, be sure to
> specify --build too; this will be fixed in the future.
and will quell some warnings in depends (#16354). If anything, this
also explicitly enables cross-compilation mode when `--host` differs
from `--build`.
As for "fixed in the future", this is the case for Autoconf 2.70+.
246774e264 depends: fix Qt precompiled headers bug (Igor Cota)
8e7ad4146d depends: disable Qt Vulkan support on Android (Igor Cota)
ba46adaa1a CI: add Android APK build to cirrus (Igor Cota)
7563720e30 CI: add Android APK build script (Igor Cota)
ebfb10cb75 Qt: add Android packaging support (Igor Cota)
Pull request description:
This PR is the third and final piece of the basic Android support puzzle - it depends on https://github.com/bitcoin/bitcoin/pull/16110 and is related to https://github.com/bitcoin/bitcoin/pull/16883. It introduces an `android` directory under `qt` and a simple way to build an Android package of `bitcoin-qt`:
1. Build depends for Android as described in the [README](https://github.com/bitcoin/bitcoin/blob/master/depends/README.md)
2. Configure with one of the resulting prefixes
3. Run `make && make apk` in `src/qt`
The resulting APK files will be in `android/build/outputs/apk`. You can install them manually or with [adb](https://developer.android.com/studio/command-line/adb). One can also open the `android` directory in Android Studio for that integrated development and debugging experience. `BitcoinQtActivity` is your starting point.
Under the hood makefile `apk` target:
1. Renames the `bitcoin-qt` binary to `libbitcoin-qt.so` and copies it over to a folder under `android/libs` depending on which prefix and corresponding [ABI](https://developer.android.com/ndk/guides/abis.html#sa) `bitcoin-qt` was built for
2. Takes `libc++_shared.so` from the Android NDK and puts in the same place. It [must be included](https://developer.android.com/ndk/guides/cpp-support) in the APK
3. Extracts Qt for Android Java support files from the `qtbase` archive in `depends/sources` to `android/src`
There is also just a tiny bit of `ifdef`'d code to make the Qt Widgets menus usable. It's not pretty but it works and is a stepping stone towards https://github.com/bitcoin/bitcoin/pull/16883.
ACKs for top commit:
MarcoFalke:
cr ACK 246774e264
laanwj:
Code review ACK 246774e264
Tree-SHA512: ba30a746576a167545223c35a51ae60bb0838818779fc152c210f5af1413961b2a6ab6af520ff92cbc8dcd5dcb663e81ca960f021218430c1f76397ed4cead6c
180dc3c886 build: miniupnpc 2.2.2 (fanquake)
Pull request description:
Creating the dll subdir is no-longer required.
We can drop our wingen patch.
One issue that came up in [#19867](https://github.com/bitcoin/bitcoin/pull/19867#discussion_r483516359):
> unrelated to this change but: why are we inserting the architecture in here, seems like something not necessary to reveal
> My assumption is that it was being inserted to make depends more deterministic. However I think we can improve this, as there's no reason to reveal more of the version information either. Could leave the version as is /2.0 and either drop the architecture, or insert something else?
I've dropped our `sed` and added a patch that just removes the OS string and miniupnpc version from the User-Agent. i.e:
```bash
# master
strings depends/x86_64-apple-darwin19.6.0/lib/libminiupnpc.a | rg -i User-Agent
User-Agent: x86_64-apple-darwin19.6.0, UPnP/1.1, MiniUPnPc/2.0.20180203
User-Agent: x86_64-apple-darwin19.6.0, UPnP/1.1, MiniUPnPc/2.0.20180203
# this PR
strings depends/x86_64-apple-darwin19.6.0/lib/libminiupnpc.a | rg -i User-Agent
User-Agent: UPnP/1.1
User-Agent: UPnP/1.1
```
Note that built unmodified (22c1386351), the User-Agent would be:
```bash
strings libminiupnpc.dylib | rg User-Agent
User-Agent: Darwin/19.6.0, UPnP/1.1, MiniUPnPc/2.2.0
User-Agent: Darwin/19.6.0, UPnP/1.1, MiniUPnPc/2.2.0
```
ACKs for top commit:
laanwj:
Code review ACK 180dc3c886
hebasto:
ACK 180dc3c886.
Tree-SHA512: b0b6e623dbc5499e28faedf992d84278d6a11887a45a3806957b9e08886c5e56044cdfa2e7d7ec81cb1dd55f89be99834367905315d6bc611ba530e91d889ad1
fanquake
