a58868d201 build: Makes rcc output always deterministic (Hennadii Stepanov)
Pull request description:
The Qt Resource Compiler ([rcc](https://doc.qt.io/qt-5/rcc.html)) has a command-line option `--format-version` which has the [default value](https://code.qt.io/cgit/qt/qtbase.git/tree/src/tools/rcc/main.cpp?h=5.12.10#n172) 2.
The only difference from `--format-version 1` is adding a [last modified timestamp](https://code.qt.io/cgit/qt/qtbase.git/tree/src/tools/rcc/rcc.cpp?h=5.12.10#n207) to the output file ([credits](https://github.com/bitcoin/bitcoin/pull/21654#issuecomment-819198228) to **fanquake**). That, in turn, forces us to use `QT_RCC_SOURCE_DATE_OVERRIDE=1` to get deterministic builds (#13732).
This change makes rcc output always deterministic by using `--format-version 1` option that makes usage of the
`QT_RCC_SOURCE_DATE_OVERRIDE` needless.
---
Also it improves interaction with ccache:
On master (f6c44e999b):
```
$ make && make clean && ccache --zero-stats && make && ccache --show-stats
...
cache directory /home/hebasto/.ccache
primary config /home/hebasto/.ccache/ccache.conf
secondary config (readonly) /etc/ccache.conf
stats updated Sun Apr 11 15:45:43 2021
stats zeroed Sun Apr 11 15:45:05 2021
cache hit (direct) 638
cache hit (preprocessed) 0
cache miss 1
cache hit rate 99.84 %
called for link 10
cleanups performed 0
files in cache 20023
cache size 13.2 GB
max cache size 15.0 GB
```
The missed file is always `qt/libbitcoinqt_a-qrc_bitcoin_locale.o`.
With this PR:
```
$ make && make clean && ccache --zero-stats && make && ccache --show-stats
...
cache directory /home/hebasto/.ccache
primary config /home/hebasto/.ccache/ccache.conf
secondary config (readonly) /etc/ccache.conf
stats updated Sun Apr 11 15:28:46 2021
stats zeroed Sun Apr 11 15:28:21 2021
cache hit (direct) 639
cache hit (preprocessed) 0
cache miss 0
cache hit rate 100.00 %
called for link 10
cleanups performed 0
files in cache 20012
cache size 13.2 GB
max cache size 15.0 GB
```
ACKs for top commit:
fanquake:
ACK a58868d201
Tree-SHA512: 52f4a3267f41883d13025c0de79b6da22e92d60c729e01b986935c6812bbfe7fadc40b742bd715bfdf09df94af6838d4fbbe8208c6123f366108e38c8e1121c5
b95f7f8ac0 build, qt, refactor: Drop sed commands for win32-g++/qmake.conf (Hennadii Stepanov)
Pull request description:
Such possibility is [available](https://codereview.qt-project.org/c/qt/qtbase/+/165348) since Qt 5.8.0.
ACKs for top commit:
fanquake:
ACK b95f7f8ac0
Tree-SHA512: e56a3d208a6bd5d42c722f8b344010fe7d1b6f7a28486613dfcb03f0403a47cee8476e2366eeaac401a19836cd09f782e8741a1e781ab4d78f72c500a30e4929
The Qt Resource Compiler (rcc) has a command-line option
`--format-version` which has the default value 2.
The only difference from `--format-version 1` is adding a last modified
timestamp to the output file. That, in turn, forces us to use
`QT_RCC_SOURCE_DATE_OVERRIDE=1` to get deterministic builds.
This change makes rcc output always deterministic by using
`--format-version 1` option that makes usage of the
`QT_RCC_SOURCE_DATE_OVERRIDE` needless. Also it improves interaction
with ccache.
Co-authored-by: fanquake <fanquake@gmail.com>
c799a19b4b build, qt: No longer need to set QT_RCC_TEST=1 for determinism (Hennadii Stepanov)
Pull request description:
The Qt Resource Compiler (rcc) output order relies on [`QHash`](https://doc.qt.io/qt-5/qhash.html):
> This randomization of `QHash` is enabled by default. Even though programs should never depend on a particular `QHash` ordering, there may be situations where you temporarily need deterministic behavior, for example for debugging or regression testing. To disable the randomization, define the environment variable `QT_HASH_SEED` to have the value 0.
Since #3620 we use `QT_RCC_TEST=1` to achieve a deterministic output.
Since Qt 5.3.1 hash seeding is disabled for all of the bootstrapped tools, including rcc. Therefore, `QT_RCC_TEST=1` is no longer needed.
See commit [5283a6c87beac5a43f612786fefd6e43f2c70bf6](5283a6c87b).
ACKs for top commit:
fanquake:
ACK c799a19b4b
Tree-SHA512: 9d116ac1e8c605ee3e8ed7f618586f0de85d8b06bbbb70fe8c298939ce203d2a7e97264a9afac037179993ab54c5f69a65ebb9ab27ca7f45acb963011bd45743
246774e264 depends: fix Qt precompiled headers bug (Igor Cota)
8e7ad4146d depends: disable Qt Vulkan support on Android (Igor Cota)
ba46adaa1a CI: add Android APK build to cirrus (Igor Cota)
7563720e30 CI: add Android APK build script (Igor Cota)
ebfb10cb75 Qt: add Android packaging support (Igor Cota)
Pull request description:
This PR is the third and final piece of the basic Android support puzzle - it depends on https://github.com/bitcoin/bitcoin/pull/16110 and is related to https://github.com/bitcoin/bitcoin/pull/16883. It introduces an `android` directory under `qt` and a simple way to build an Android package of `bitcoin-qt`:
1. Build depends for Android as described in the [README](https://github.com/bitcoin/bitcoin/blob/master/depends/README.md)
2. Configure with one of the resulting prefixes
3. Run `make && make apk` in `src/qt`
The resulting APK files will be in `android/build/outputs/apk`. You can install them manually or with [adb](https://developer.android.com/studio/command-line/adb). One can also open the `android` directory in Android Studio for that integrated development and debugging experience. `BitcoinQtActivity` is your starting point.
Under the hood makefile `apk` target:
1. Renames the `bitcoin-qt` binary to `libbitcoin-qt.so` and copies it over to a folder under `android/libs` depending on which prefix and corresponding [ABI](https://developer.android.com/ndk/guides/abis.html#sa) `bitcoin-qt` was built for
2. Takes `libc++_shared.so` from the Android NDK and puts in the same place. It [must be included](https://developer.android.com/ndk/guides/cpp-support) in the APK
3. Extracts Qt for Android Java support files from the `qtbase` archive in `depends/sources` to `android/src`
There is also just a tiny bit of `ifdef`'d code to make the Qt Widgets menus usable. It's not pretty but it works and is a stepping stone towards https://github.com/bitcoin/bitcoin/pull/16883.
ACKs for top commit:
MarcoFalke:
cr ACK 246774e264
laanwj:
Code review ACK 246774e264
Tree-SHA512: ba30a746576a167545223c35a51ae60bb0838818779fc152c210f5af1413961b2a6ab6af520ff92cbc8dcd5dcb663e81ca960f021218430c1f76397ed4cead6c
See:
- QTBUG-72903, commit 9864d2c6f3b628ca9f07a56b197e77bd43931cca
- QTBUG-78873, commit e55a61a77f0c87c05661a0335dfdb12673c6a27f
Could be dropped for Qt 5.14+.
This tries to invoke xcrun, which is not available when cross-compiling.
Given we are in control of the SDK versions being used, removing this
check has minimal-no effect.
remove fix_configure_mac.patch
Fixed upstream: https://bugreports.qt.io/browse/QTBUG-67286
remove fix_riscv64_arch.patch
Was fixed upstream in 6a39e49a6cdeb28a04a3657bb6a22f848d5dfa9d
remove fix_rcc_determinism.patch
Fixed upstream in https://bugreports.qt.io/browse/QTBUG-62511
remove freetype_back_compat.patch
By the time we ship a release with Qt 5.12, we'll certainly no-longer be
supporting Ubuntu 14.04 and Ubuntu 16.04 ships with FreeType 2.6.1,
which is new enough that using the symbol is no-longer an issue.
The renaming of FT_Get_X11_Font_Format() happened in FreeType 2.6
remove xkb-default.patch
This was removed upstream in d5abf545971da717014d316127045fc19edbcd65
Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
This change gets rid of multiple `../` that makes reasoning about the
script and its maintaining much easier.
Co-authored-by: fanquake <fanquake@gmail.com>
b685f60a08 build: mac_alias 2.1.1 (fanquake)
5d2cbdf772 macdeploy: use Python 3.6 (fanquake)
a42aa94c54 macdeploy: remove runHDIUtil in favor of directly calling subprocess.run (fanquake)
adaa26202b macdeploy: remove existing Bitcoin-Core.dmg if present (fanquake)
ccb0325b1b macdeploy: move qt_conf to where it's used (fanquake)
6390a04862 macdeploy: consolidate .DS_Store generation (fanquake)
32347cd56a macdeploy: assume plistlib is available (fanquake)
0ab4018c12 macdeploy: have a single level of logging output (fanquake)
827d382aa7 macdeploy: remove add-resources argument (fanquake)
464b34d4c3 macdeploy: remove codesigning argument (fanquake)
4d70d3d7fe build: automatically determine macOS translations (fanquake)
Pull request description:
This consolidates our macOS build code so that `.DS_Store` generation is the same when running `make deploy` for macOS when building on Linux and macOS, rather than maintaining two version of code that essentially do the same thing (just slightly differently).
It also removes unused code and any AppleScript usage, automates finding translation files and generally simplifies `macdeployqtplus`. It also gets rid of the annoying "popping up" behaviour during DMG generation, names the created image `Bitcoin-Core.dmg` rather than `Bitcoin-Qt.dmg`.
ACKs for top commit:
dergoegge:
ACK b685f60a08 - Less and cleaner code looks good. I tested this with `make deploy` and everything still works + the popup during DMG generation is gone.
Tree-SHA512: dcd38344e2dfcfa7ffbccf6226a71425c4d16b421a4881d5ee37b8e7ef393b3e8077262444c39b11912269d8cf688aba897e6518cba8361eb24a03fdd03b8caf
c23f6f84ef Add depends qt fix for ARM macs (Jonas Schnelli)
Pull request description:
With this, depends builds fine on macOS 11 on an Apple Silicon Mac (ARM64).
ACKs for top commit:
laanwj:
Code review ACK c23f6f84ef
Tree-SHA512: a8354cec99969cff9e7dab150c335050ddb4b3c93a9f12a4db5e8046f02b11ce692ac17c2b96cbbe7f380c1aa110b15b8d6d48d51bc9c560282c702e99fd8a8d
c82d15b6d1 depends: Do not force Precompiled Headers (PCH) for building Qt on Linux (Hennadii Stepanov)
Pull request description:
On CentOS 8 (Cirrus CI job) the forced `-pch` option breaks Qt build.
Removing `-pch` option does not affect build time for other systems:
- master (e2ff5e7b35):
```
$ time make -j 9 -C depends/ qt
...
Caching qt...
make: Leaving directory '/home/hebasto/guix/GitHub/bitcoin/depends'
real 4m22,359s
user 18m3,719s
sys 1m24,769s
```
- this PR:
```
$ time make -j 9 -C depends/ qt
...
Caching qt...
make: Leaving directory '/home/hebasto/guix/GitHub/bitcoin/depends'
real 4m14,862s
user 18m3,355s
sys 1m24,506s
```
Qt docs: https://doc.qt.io/qt-5/qmake-precompiledheaders.htmlFixes#20423
ACKs for top commit:
MarcoFalke:
review ACK c82d15b6d1
Tree-SHA512: 0f2a3712e90de881d00f8e56c363edde33dd4f5c117df5744ab4e51d0a8146331de7236bc8329d68ddd91535cd853e68ee80ef4cceb6a909786abfd8881b01e8
Rather than using OSX_QT_TRANSLATIONS which must be manually updated,
and we forget to update anyway, i.e: #19059, automatically find and copy
available translations from the translations directory.
2f5dfe4a7f depends: build qt in c++17 mode (fanquake)
104e859c97 builds: don't pass -silent to qt when building in debug mode (fanquake)
e2c500636c depends: build zeromq with -std=c++17 (fanquake)
2374f2fbef depends: build Boost with -std=c++17 (fanquake)
2dde55702d depends: build bdb with -std=c++17 (fanquake)
Pull request description:
In packages where we are passing `-std=c++11` switch to `-std=c++17`, or, `-std=c++1z` in the case of Qt.
This PR also contains a [commit](104e859c97) that improves debug output when building Qt for debugging (`DEBUG=1`).
Now we'll get output like this:
```bash
g++ -c -pipe -ffunction-sections -O2 -fPIC -std=c++11 -fno-exceptions <lots more> ../../corelib/kernel/qcoreapplication.cpp
```
rather than just:
```bash
compiling ../../corelib/kernel/qcoreapplication.cpp
```
Note that when you look at the DEBUG output for these changes when building Qt, you'll see objects being compiled with a mix of C++11 and C++17. The breakdown is roughly:
1. `qmake` built with `-std=c++11`:
```bash
Creating qmake...
make[1]: Entering directory '<trim>/qt/5.9.8-4110fa99945/qtbase/qmake'
g++ -c -o project.o -std=c++11 -ffunction-sections -O2 -g <trim> <trim>/qt/5.9.8-4110fa99945/qtbase/qmake/project.cpp
# when qmake, Qt also builds some of it's corelib, such as corelib/global/qmalloc.cpp
g++ -c -o qmalloc.o -std=c++11 -ffunction-sections -O2 -g <trim> <trim>/qt/5.9.8-4110fa99945/qtbase/src/corelib/global/qmalloc.cpp
```
2. `qmake` is run, and passed our build options, including `-c++std`:
```bash
make[1]: Entering directory '<trim>/qt/5.9.8-4110fa99945/qtbase'
<trim>qt/5.9.8-4110fa99945/qtbase/bin/qmake -o Makefile qtbase.pro -- -bindir <trim>/native/bin -c++std c++1z -confirm-license <trim>
```
3. After some cleaning and configuring, we actually start to build Qt, as well as it's tools and internal libs:
```bash
Building qt...
make[1]: Entering directory '<trim>/qt/5.9.8-4110fa99945/qtbase/src'
# build libpng, zlib etc
gcc -c -m64 -pipe -pipe -O1 <trim> -o .obj/png.o png.c
# build libQt5Bootstrap, using C++11, which again compiles qmalloc.cpp
make[2]: Entering directory '<trim>/qt/5.9.8-4110fa99945/qtbase/src/tools/bootstrap'
g++ -c -pipe -ffunction-sections -O2 -fPIC -std=c++11 <trim> -o .obj/qmalloc.o ../../corelib/global/qmalloc.cpp
# build a bunch of tools like moc, rcc, uic, qfloat16-tables, qdbuscpp2xml, using C++11
g++ -c -pipe -O2 -std=c++11 -fno-exceptions -Wall -W <trim> -o .obj/rcc.o rcc.cpp
# from here, Qt is compiled with -std=c++1z, including qmalloc.cpp, for the third and final time:
g++ -c -include .pch/Qt5Core <trim> -g -Og -fPIC -std=c++1z -fvisibility=hidden <trim> -o .obj/qmalloc.o global/qmalloc.cpp
```
4. Finally, build tools like `lrelease`, `lupdate`, etc, but back to using -std=c++11
```bash
make[1]: Entering directory '<trim>/qt/5.9.8-4110fa99945/qttools/src/linguist/lrelease'
g++ -c -pipe -O2 -std=c++11 -fno-exceptions -Wall -W <trim> -o .obj/translator.o ../shared/translator.cpp
```
If you dump the debug info from the built Qt libs, they should also tell you that they were compiled with `C++17`:
```bash
objdump -g bitcoin/depends/x86_64-pc-linux-gnu/lib/libQt5Core.a
GNU C++17 9.3.0 -m64 -mtune=generic -march=x86-64 -g -O1 -Og -std=c++17 -fPIC -fvisibility=hidden -fvisibility-inlines-hidden -fasynchronous-unwind-tables -fstack-protector-strong -fstack-clash-protection -fcf-protection
```
ACKs for top commit:
laanwj:
Code review ACK 2f5dfe4a7f
practicalswift:
cr ACK 2f5dfe4a7ff12b6b57427374142cdf7e266b73bc: patch looks correct
fjahr:
Code review ACK 2f5dfe4a7f
hebasto:
ACK 2f5dfe4a7f, I have reviewed the code and it looks OK, I agree it can be merged.
Tree-SHA512: fc5e9d7c7518c68349c8228fb1aead829850373efc960c9b8c079096a83d1dad19c62a9730fce5802322bf07e320960fd47851420d429eda0a87c307f4e8b03a
This means we'll get build output like this when building with DEBUG=1:
g++ -c -pipe -ffunction-sections -O2 -fPIC -std=c++11 -fno-exceptions <lots more> ../../corelib/kernel/qcoreapplication.cpp
rather than just:
compiling ../../corelib/kernel/qcoreapplication.cpp
Aplies a patch to Qt that fixes the non-determinism by modifying Qt. The
source of the non-determinism is how LLVM 8 optimizes qt_intersect_spans
when compiling. The particular optimization that seems to be causing the
problems is that a temp variable is being added for spans->y. For some
reason, when it does this, it chooses different instructions to use when
making that variable. We bypass this problem by patching
qt_intersect_spans to always make and use this local variable.
We generate our own Info.plist as part of make deploy, and as far as I
can tell, it doesn't seem to have an effect wether these are present
during qt's build.
I also can't find a single mention of the .app plist in the qt code,
whereas there are multiple instances of .lib.
This should mostly be a no-op, however it would seem to make more sense
that we pass through the XCODE_VERSION we now have in depends, rather
than leaving the version set to 4.3.
This is an alternative to #19751 that fixes the build without requiring
splitting out libpng. This patch can be dropped once we are building qt
5.12.0 or later.
Hennadii Stepanov
