Qt's configure grabs the path to xkb's data root during configure, but the
build changes in 5.8 apparently broke the handling for cross builds. As a
result, the string embedded in the binary depends on whether or not some files
are present in the builder's filesystem.
The "-xkb-config-root" configure setting is intended to allow manual overriding
but it is also broken. See: https://bugreports.qt.io/browse/QTBUG-60005
This has since been fixed upstream, so just hard-code the path for now. We can
drop this patch when we bump to a fixed Qt.
Also, fix the "-qt-xkbcommon-x11" config param which was renamed. This does not
appear to affect build results, presumably because auto-detection is working,
but it does not hurt to be explicit.
6b5506a286 Fix Qt's rcc determinism for depends/gitian (Fuzzbawls)
Pull request description:
With the update to Qt 5.9 having been merged, Qt's `rcc` tool now embeds a file's last modified time in it's output. Since the build system generates temporary files for all locale translations (`*.qm` files) at build time, the resulting `qrc_bitcoin_locale.cpp` file was always being generated in a non-deterministic way.
This is a backport of https://bugreports.qt.io/browse/QTBUG-62511, which is included in Qt versions 5.11+, that allows for an environment variable (`QT_RCC_SOURCE_DATE_OVERRIDE`) to override the behavior described above. This environment variable is in turn set in the gitian descriptors, as that is where determinism is vital for release purposes.
Prior to this, the `qt_libbitcoinqt_a-qrc_bitcoin_locale.o` object file (included into `libbitcoinqt.a`) was returning a different `sha256sum` for each and every build, regardless of file contents change, thus breaking determinism in the resulting binaries.
This should fix#13731
Tree-SHA512: 174017e41f9afc3950ef54a9419de81577ec900db9aec3c78ccd3d879c6aecaaeb944fde0615b933f43e6ca9d7898a27ec071cdd0b91cb772755a3012de96725
12c93cb85b depends: use MacOS friendly sed syntax in qt.mk (Sjors Provoost)
Pull request description:
I just got this error on master:
```
make clean-all
make -j5
[...]
/bitcoin/depends/sources/qttools-opensource-src-5.9.6.tar.xz: OK
Preprocessing qt...
sed: 1: "qtbase/mkspecs/bitcoin- ...": extra characters at the end of q command
make: *** /bitcoin/depends/work/build/x86_64-apple-darwin17.7.0/qt/5.9.6-b48df1b490b/.stamp_preprocessed] Error 1
```
Tree-SHA512: 1eaee535c0d637a67801c7dd48e9cd5a037823f4e162a6f50e375787045fdb95929f409a88c7d000d8781a4edc497b5ccb1fd23a24e29324ec633157fadd371e
f447a0a707 Remove program options from build system (Chun Kuan Lee)
11588c639e Replace boost program_options (Chun Kuan Lee)
Pull request description:
Concept from #12744, but without parsing negated options.
Tree-SHA512: 7f418744bb8934e313d77a5f162633746ef5d043de802b9c9cd9f7c1842e7e566eb5f171cd9e2cc13317281b2449c6fbd553fa4f09b837e6af2f5d2b2aabdca2
4b69984557 Add depends 32-bit arm support for bitcoin-qt (Sebastian Kung)
Pull request description:
Some hobbyists are used to using the desktop for interfacing with their raspberry pi. This commits adds qt to the arm-linux-gnueabihf target.
Tree-SHA512: cb03387267eb8f68dfd79735c2c01c5a119c406e5578805e60b377934da42d46cb34d35e45c8843979dfb4070859c553d09ae348b468d9731523f33307132fa8
974f0bf8e6 depends: Mention RISC-V known compilation issue with gcc-7.3.x (Wladimir J. van der Laan)
0d1f38c45f depends: update zmq config.guess/config.sub for riscv support (fanquake)
409481c465 depends: latest config.sub (fanquake)
d7005e9988 depends: latest config.guess (fanquake)
359e2e3525 depends: Add RISC-V support (Wladimir J. van der Laan)
Pull request description:
This adds support for riscv32 and riscv64 builds to the depends system.
The change consists of documentation and build system changes. The most significant change is an update of `config.sub` and `config.guess` inside zeromq patch, as the current version does not recognize the `riscv*` host tuples (there's no new version of ZeroMQ yet with newer ones).
Good thing: RISC-V 64-bit toolchain packages can be installed out of the box on Ubuntu 18.04+.
I would also like to add RISC-V 64-bit executables to gitian, but this will not be possible until #12511 .
Tree-SHA512: 358ed72ee9e4ae44e7d305c09a4ff5ce5460eeb7ed915eb25d39c8f43b61e7b347f51bf0ae5d83ddb4ce8876dea7703c926b3baa3cccb4932b3bc17160d801bb
aff16fd depends: Add 'make clean' and 'make clean-all' rules (Henrik Jonsson)
Pull request description:
It's useful to have a standard way to clean up the work done by the
depends system when testing changes to it.
Tree-SHA512: d56686ca810c2531d852049420f242178d124d45a2508feeac0acf80408e8a95fc9915fb777ff0cbc4edc496990f5fdecf699c87d7986164c74b8fbf418ebf49
It's useful to have a standard way to clean up the work done by the
depends system when testing changes to it.
The `make clean-all` rule removes build artifacts for all
supported architectures (in addition to sources/), while `make clean`
only removes artifacts for current architecture (`BUILD`).
4ef82f1 depends: biplist 1.0.3 (fanquake)
Pull request description:
biplist should now be usable for reproducible builds without any patching.
One change has been incorporated, the two remaining changes were rejected upstream: https://bitbucket.org/wooster/biplist/pull-requests/9/make-biplist-ordering-deterministic/diff#comment-None
testing on gitian cc @jonasschnelli
Tree-SHA512: a3ef3ecad08b09f7a34d927bc4e3d8604099e9acb2c984bbe741df6162f4014f40e9eb2fd28309fc79d3dd2bb82f14bfd473925b90048e5fd135a471726a4836
cc879675e1 depends: Remove ccache (fanquake)
Pull request description:
After discussion with @theuni, we can possibly just remove ccache from depends entirely.
Related to #12606
Tree-SHA512: ae0a60c8d97467fa41d617daa48ed22159cf32613808634a983304901dd5ed27124e77868d2314004e5144f7b35ba1333f720bb12daec4c5ca03aaf29d593ef2
The patch Bitcoin Core has been maintaining for mac_alias was pulled by the mac_alias maintainer in commit 4f31cb084c1c6a8626128b0b00842020b6db9037. Delete the patch and remove the patch from the depends system.
Note that this PR won't be complete until a new version of mac_alias containing the path has been released, and the depends system is updated to reflect the new version.
31a013563 Add required package dependencies for depends cross compilation [skip-ci] (Jonas Schnelli)
Pull request description:
Stumbled over this during a setup of a new depends compile system.
Related to #8913.
Tree-SHA512: 67e2fdf9ca3cbedeb02982fa73771dd36978b319e9291ea5a41ede7fdf772c4505ccc9523b48fe66ead927f141efefbdf1e3eaa19a9d8a1304861a8ede040056
This is broken for a number of reasons, including:
- g++ understands "-static-libstdc++ -lstdc++" to mean "link against whatever
libstdc++ exists, probably shared", which in itself is buggy.
- another stdlib (libc++ for example) may be in use
4f92b5f Run Qt wallet tests on travis (Russell Yanofsky)
Pull request description:
Currently these test failures are not caught by travis leading to bugs like:
https://github.com/bitcoin/bitcoin/pull/10506
Tree-SHA512: db1e4ff5b17bcd6fd000a3d21aa74f6b7e4c194e0663c1896a100612671910a7cdadd896b59642420ea016598895b54a8468914847ebefef105a3c47c311d4b2
ld64 is threaded, and uses a worker for each CPU to parse input files. But
there's a bug in the parser causing dependencies to be calculated differently
based on which files have already been parsed.
As a result, builders with more CPUs are more likely to see non-determinism.
This looks to have been fixed in a newer version of ld64, so just disable
threading for now. There's no noticible slowdown.
This contains a few hacks very specific to Qt's buildsystem. These can be
reverted once we split the build between native and target builds.
Qt's build contains a circular dependency when not using a system zlib.
By far the easiest fix is to switch to a system zlib, rather than Qt's own.
However, that confuses Qt's cross build which assumes that when using a system
zlib, it should also find a system (native) zlib for native tools. The build
breaks if that zlib is not present.
To solve this:
1. Always use a system zlib rather than the one provided by qt
2. Set force_bootstrap, which instructs the build tools to be built as though
we're cross-compiling (build != target)
3. For build tools, use qt's internal zlib so that a native zlib is not
required.
Step 3 means that if any zlib headers are found by the native build, it will
confuse Qt's internal zlib build. So we also need to make sure that the target
headers/libs aren't found. To do so, specify that our
cflags/cxxflags/cppflags/ldflags only apply for non-host builds.
qt5.7 changed the location of some of its symbols, creating a circular
dependency in Qt5Core. Rather than trying to fix that up, build our own zlib
rather than having it built for us.
Their buildsystem insists on using the installed ltranslate, but gets confused
about how to find it. Since we manually control the build order, just drop the
dependency.
Add a patch that seems to be necessary for compatibilty of libevent
2.0.22 with recent mingw-w64 gcc versions (at least GCC 5.3.1 from Ubuntu
16.04).
Without this patch the Content-Length in the HTTP header ends up as
`Content-Length: zu`, causing communication between the RPC
client and server to break down. See discussion in #8653.
Source: https://sourceforge.net/p/levent/bugs/363/
Thanks to @sstone for the suggestion.
- create a script to handle split debug. This will also eventually need to check
targets, and use dsymutil for osx.
- update config.guess/config.sub for bdb for aarch64.
- temporarily disable symbol checks for arm/aarch64
- quit renaming to linux32/linux64 and use the host directly
This also adds a hack to work around an Ubuntu bug in the gcc-multilib package:
https://bugs.launchpad.net/ubuntu/+source/gcc-defaults-armhf-cross/+bug/1347820
The problem is that gcc-multilib conflicts with the aarch toolchain.
gcc-multilib installs a symlink that points
/usr/include/asm -> /usr/include/x86_64-linux-gnu/asm.
Without this link, gcc -m32 can't find asm/errno.h (and others), since
/usr/include/x86_64-linux-gnu isn't in its default include path. But
/usr/include/i386-linux-gnu is (though it doesn't exist on disk).
So work around the problem by linking
/usr/include/i386-linux-gnu/asm -> /usr/include/x86_64-linux-gnu/asm.
The symlink fix is actually quite reasonable, but echoing the password into
sudo is nasty, and should probably be addressed in gitian itself. It makes more
sense to enable passwordless sudo for the build user by default.
7e7eb27 gitian: create debug packages for linux/windows (Cory Fields)
ad38204 gitian: use CONFIG_SITE rather than hijacking the prefix (Cory Fields)
b676f38 depends: allow for CONFIG_SITE to be used rather than stealing prefix (Cory Fields)
This does not break any existing prefix behavior, only makes new behavior work.
For example:
CONFIG_SITE=$PWD/depends/x86_64-pc-linux-gnu/share/config.site ./configure --prefix=/
These add very simple sanity checks to ensure that the build/host toolchains
have not changed since the last run. If they have, all ids will change and
packages will be rebuilt.
For more complicated usage (like parsing dpkg), HOST_ID_SALT/BUILD_ID_SALT may
be used to introduce arbitrary data to the ids.
2016/01/24:
Change miniwget to return HTTP status code
Increments API_VERSION to 16
2016/01/22:
Improve UPNPIGD_IsConnected() to check if WAN address is not private.
Parse HTTP response status line in miniwget.c
149641e Travis: Use Blue Box VMs for IPv6 loopback support (Luke Dashjr)
c01f08d Bugfix: depends/Travis: Use --location (follow redirects) and --fail [on HTTP error response] with curl (Luke Dashjr)
5d1148c Travis: Use curl rather than wget for Mac SDK (Luke Dashjr)
1ecbb3b depends: Use curl for fetching on Linux (Luke Dashjr)
Currently Travis's wget fails fetching qrencode:
Fetching qrencode...
ERROR: no certificate subject alternative name matches
requested host name `fukuchi.org'.
To connect to fukuchi.org insecurely, use `--no-check-certificate'.
OpenSSL: error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error
Unable to establish SSL connection.
make: *** [/home/travis/build/luke-jr/bitcoin/depends/sources/download-stamps/.stamp_fetched-qrencode-qrencode-3.4.4.tar.bz2.hash] Error 4
Remove sed-based qt PIDLIST_ABSOLUTE workaround, replace by a patch that
works for both old (such as used by Travis and Ubuntu Precise) and new
mingw (Ubuntu Trusty).
This passes `-Wa,--noexecstack` to the assembler when building
platform-specific assembly files, to signal that a non-executable stack
can be used. This is the same approach as used by Debian
(see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=430583)
This version of miniupnpc fixes a buffer overflow in the XML (ugh)
parser during initial network discovery.
http://talosintel.com/reports/TALOS-2015-0035/
The commit fixing the vulnerability is:
79cca974a4
Reported by timothy on IRC.
This should fix the spurious comparison tool failures.
See discussion here: https://github.com/bitcoin/bitcoin/pull/6305
The race fix was cherry-picked on top of the version we're currently using, so
it should be functionally identical otherwise.
This should be functionally identical to what's in place now. It was built from
be0eef7744
That commit is the same as this pruned commit in TheBlueMatt's repo:
https://github.com/TheBlueMatt/bitcoinj/commit/0f7b5d8
Now we'll be able to trust the line numbers in the stack traces.
Boost assumes variadic templates are always available in GCC 4.4+, but
they aren't since we don't build with -std=c++11.
This applies the patch that fixed the issue in boost 1.57:
eec8085549
See also: https://svn.boost.org/trac/boost/ticket/10500
Documentation more readable when viewed on Github.
Some extra changes by @laanwj:
- Make README.usage the default README. This is more convenient from a
user perspective. Link to other documentation in this default README
- Add list of popular targets for cross compilation, change default to
Win64 instead of Win32
In some cases (Travis), sources and build caches may be moved around in-between
builds, and we can't necessarily trust that everything is still intact.
This introduces pre-build checks that verify against stashed checksums.
Note that this will cause all sources to be re-downloaded, since cached sources
weren't trustworthy before this.
See here for background: https://bugreports.qt.io/browse/QTBUG-34748
libxcb temporarily had an abi breakage which caused crashes when qt was
compiled against a non-compatible version. Building qt with -qt-xcb should have
shielded us from this issue, except that incompatible headers were used when
building qt's wrapper.
Make sure those headers aren't picked up by qt's build.
Details:
qt's build adds a wrapper around the xcb libs when -qt-xcb is used. This is
done to avoid having to link to a handful of different libs, which may not be
api/abi stable. This build depends on include-order, so that its files are
found before the real libxcb headers.
Our build (for other reasons related to qt's complicated build-system) injects
our prefix into CXXFLAGS. Because libxcb is found in this path, that reverses
the include-order, negating the purpose of the wrapper.
To fix, libxcb's includes are simply moved to a subdir. pkg-config ensures that
they're still found properly when needed.
To make things even more interesting, this behavior in qt's .pro files is broken:
INCLUDEPATH += $$QMAKE_CFLAGS_XCB
The INCLUDEPATH variable is processed by qmake which automatically prefixes each
entry with "-I". The QMAKE_CFLAGS_XCB variable comes from pkg-config and
already contains -I, making the path look like "-I-I/path/to/xcb/headers".
To work around that, CFLAGS/CXXFLAGS are used here rather than INCLUDEPATH.
tl;dr: Update to the newer stable toolchain and SDK for OSX without giving up
any backwards compatibility. We can move to clang 3.5 as a next step which
allows use to use libc++ and the 10.10 sdk, but we'll need to find a build that
works in gitian/travis first.
Switch to a new, better maintained fork of cctools:
https://github.com/tpoechtrager/cctools-port
I've forked this and will be working on it some as well:
https://github.com/theuni/cctools-port
This brings in:
cctools v862
ld64: v241.9
It also fixes 64bit builds, so there's no longer any need to use a 32bit clang.
Since clang is no longer tied to an old/crusty 32bit build, clang has been
upgraded to 3.3. Unfortunately, there's a bug in 3.4 that breaks builds. 3.5
works fine, but there are no binary builds compatible with precise, which is
currently used for gitian and travis. We could always build our own if
necessary.
After updating to stable clang/linker/cctools, it's possible to use a more
recent SDK. The current SDK (10.7) through the most recent 10.10 have all been
built/tested successfully, both with and without 10.6 compatibility. However,
10.10 requires clang 3.5.
SDKs >= 10.9 use libc++ rather than libstdc++. This is verified working as well.
Broken hash logic caused all depends on some platforms (osx at least) to end up
with the same build-id. Without this fix, nothing will be rebuilt when recipes
or dependencies change.
Since the last commit will force rebuilds of all depends, take the opportunity
to clean up a few other things that would trigger rebuilds as well.
- Move source stamps to the sources dir so that SOURCES_PATH is respected for
"make download".
- Only print "fetching..." when actually downloading a file.
- Avoid using non-deterministic paths for the recipe hash (patch location).
This should ensure that all builders get the same resulting build-ids.
- Use a per-package source paths. This will allow for removing old source files
in the future.
- Use a host-agnostic path for downloads which gets cleaned up properly.
Fixes default hidden symbol visibility for our linux->osx cross build. Without
this change, the check for working -fvisibility=hidden fails, and all symbols
are visible by default.
Ugly as this is, it's just a simple find/replace to fix a bug in Qt's configure.
They assume in an "XPLATFORM_MAC" block that the builder is capable of running
osx programs. This should be "BUILD_ON_MAC" instead.
Descriptors now make use of the dependencies builder, so results are cached.
A very new version (>= e9741525c) of Gitian should be used in order to take
advantage of caching.
We're not ready to switch to a static qt5 for Linux yet due to missing plugin
support. This adds a recipe for building a shared qt4 that we build and link
against, but don't distribute.
make USE_LINUX_STATIC_QT5=1 can be used to build static qt5 as before.
tl;dr: This solves boost visibility problems for default/release build configs
on non-Linux platforms.
When Bitcoin builds against boost's header-only classes, it ends up with
objects containing symbols that the upstream boost libs also have. Since
Bitcoin builds by default with hidden symbol visibility, it can end up trying
to link against a copy of the same symbols with default visibility.
This is not a problem on Linux because 3rd party static libs are un-exported
by default (--exclude-libs,ALL), but that is not available for MinGW and OSX.
Those platforms (and maybe others?) end up confused about which version to use.
The OSX linker spews hundreds of: "ld: warning: direct access in <foo> to
global weak symbol guard variable for <bar> means the weak symbol cannot be
overridden at runtime. This was likely caused by different translation units
being compiled with different visibility settings."
MinGW's linker complains similarly.
Since the default symbol visibility for Bitcoin is hidden and releases are
built that way as well, build Boost with hidden visibility. Linux builds Boost
this way also, but only for the sake of continuity.
This means that the linker confusion logic is reversed, so the problem will
will now be encountered if Bitcoin is built with --disable-reduce-exports, but
that's better than the current situation.
Bumps the OpenSSL version to the latest release, and kills SSL2. (SSL3 was already killed here, so I'm not sure why SSL2 was left around?)
No other changes.
0b17964 Bugfix: Replace bashisms with standard sh in tests/tools (Luke Dashjr)
ab72068 Bugfix: Replace bashisms with standard sh in gitian descriptors (Luke Dashjr)
b77b4ed Bugfix: Replace bashisms with standard sh to fix build on non-BASH systems (Luke Dashjr)
d6b0539 travis: add non-default shell testing to travis. (Cory Fields)
Newer mingw supports the features necessary to enable this api, whereas older
versions didn't. However once enabled (automatically by configure), it triggers
an unrelated build bug.
Since it was not enabled previously anyway, and we don't depend on the
functionality, just disable it across the board.
Some sources are renamed after download, since the filenames don't play nice
with (for example) gitian. This fixes the rename.
Needed for OSX build as it renames a file.
b144a74 depends: bump miniupnpc to 1.9.20140701. (Cory Fields)
f628127 depends: bump openssl to 1.0.1i (Cory Fields)
9f7f504 build: add -DMINIUPNP_STATICLIB for new version (Cory Fields)
Cory Fields