xorriso and its mkisofs/genisoimage emulation alter-ego xorrisofs are
more maintained, and has the right toggles for us to achieve output
determinism without using blunt tools like faketime.
In this commit, we use xorrisofs from the build environment rather than
building it ourselves using depends. This is not necessary and can be
changed in the future.
From https://wiki.debian.org/genisoimage?action=recall&rev=11 :
> The classical command line interface for production of ISO 9660
> filesystem images is the option set established by program mkisofs.
> For reasons of licensing and other problems with its author, Debian
> ships a fork of mkisofs, called genisoimage, which was split off in
> 2006 and then developed independently.
>
> Meanwhile, genisoimage gets no new features and not even bug fixes. It
> is first choice only if its options -udf or -hfs are needed.
>
> Replacement in most uses cases, especially for bootable ISO 9660
> filesystems, archiving, and backup, is xorrisofs which starts the -as
> mkisofs emulation mode of program xorriso.
Without the fix, the tests immediately abort:
terminate called after throwing an instance of 'std::runtime_error'
what(): locale::facet::_S_create_c_locale name not valid
./qt/test/test_bitcoin-qt: line 2: 116150 Aborted
This avoids timeouts when signing a large raw transaction
https://cirrus-ci.com/task/5009228131729408?command=ci#L4981
test_framework.authproxy.JSONRPCException: 'signrawtransactionwithwallet' RPC took longer than 120.000000 seconds. Consider using larger timeout for calls that take longer to return. (-344)
0e2a5e448f tests: dumping and minimizing of script assets data (Pieter Wuille)
4567ba034c tests: add generic qa-asset-based script verification unit test (Pieter Wuille)
f06e6d0345 tests: functional tests for Schnorr/Taproot/Tapscript (Pieter Wuille)
3c226639eb tests: add BIP340 Schnorr signature support to test framework (Pieter Wuille)
206fb180ec --- [TAPROOT] Tests --- (Pieter Wuille)
d7ff237f29 Activate Taproot/Tapscript on regtest (BIP 341, BIP 342) (Pieter Wuille)
e9a021d7e6 Make Taproot spends standard + policy limits (Pieter Wuille)
865d2c37e2 --- [TAPROOT] Regtest activation and policy --- (Pieter Wuille)
72422ce396 Implement Tapscript script validation rules (BIP 342) (Johnson Lau)
330de894a9 Use ScriptExecutionData to pass through annex hash (Pieter Wuille)
8bbed4b7ac Implement Taproot validation (BIP 341) (Pieter Wuille)
0664f5fe1f Support for Schnorr signatures and integration in SignatureCheckers (BIP 340) (Pieter Wuille)
5de246ca81 Implement Taproot signature hashing (BIP 341) (Johnson Lau)
9eb590894f Add TaggedHash function (BIP 340) (Pieter Wuille)
450d2b2371 --- [TAPROOT] BIP340/341/342 consensus rules --- (Pieter Wuille)
5d62e3a68b refactor: keep spent outputs in PrecomputedTransactionData (Pieter Wuille)
8bd2b4e784 refactor: rename scriptPubKey in VerifyWitnessProgram to exec_script (Pieter Wuille)
107b57df9f scripted-diff: put ECDSA in name of signature functions (Pieter Wuille)
f8c099e220 --- [TAPROOT] Refactors --- (Pieter Wuille)
Pull request description:
This is an implementation of the Schnorr/taproot consensus rules proposed by BIPs [340](https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki), [341](https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki), and [342](https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki).
See the list of commits [below](https://github.com/bitcoin/bitcoin/pull/19953#issuecomment-691815830). No signing or wallet support of any kind is included, as testing is done entirely through the Python test framework.
This is a successor to https://github.com/bitcoin/bitcoin/pull/17977 (see discussion following [this comment](https://github.com/bitcoin/bitcoin/pull/17977#issuecomment-682285983)), and will have further changes squashed/rebased. The history of this PR can be found in #19997.
ACKs for top commit:
instagibbs:
reACK 0e2a5e448f
benthecarman:
reACK 0e2a5e4
kallewoof:
reACK 0e2a5e448f
jonasnick:
ACK 0e2a5e448f almost only looked at bip340/libsecp related code
jonatack:
ACK 0e2a5e448f modulo the last four commits (tests) that I plan to finish reviewing tomorrow
fjahr:
reACK 0e2a5e448f
achow101:
ACK 0e2a5e448f
Tree-SHA512: 1b00314450a2938a22bccbb4e177230cf08bd365d72055f9d526891f334b364c997e260c10bc19ca78440b6767712c9feea7faad9a1045dd51a5b96f7ca8146e
This adds a unit test that does generic script verification tests,
with positive/negative witnesses/scriptsigs, under various flags.
The test data is large (several MB) so it's stored in the qa-assets
repo.
fdf697fe75 ci: Increase CCACHE_SIZE in some builds on Travis (Hennadii Stepanov)
Pull request description:
This PR will decrease build time for the longest Travis builds.
Up to ~20 min improvement with warmed up cache is expected.
Top commit has no ACKs.
Tree-SHA512: ee3a22162d03537be11e60b53de043247f6a65e35e630e0807b758a846b7e05ef2059d18846644aafa3cd5dc7d3e3f56a8ccb94cc71b95227debc9201d3142a2
870f0cd2a0 build: Add MemorySanitizer (MSan) in Travis to detect use of uninitialized memory (practicalswift)
Pull request description:
Add MemorySanitizer (MSan) in Travis to detect use of uninitialized memory.
First UBSan, then ASan followed by TSan... and now: yes, the wait is over -- **MSan is finally here!** :)
Some historical context:
* 2017: Continuous compilation with Clang Thread Safety analysis enabled (#10866, #10923)
* 2018: Continuous testing with trapping on signed integer overflows (`-ftrapv`) (#12686)
* 2018: Continuous testing of use of locale dependent functions (#13041)
* 2018: Continuous testing of format strings (#13705)
* 2018: Continuous compilation with MSVC `TreatWarningAsError` (#14151)
* 2018: Continuous testing under UndefinedBehaviorSanitizer – UBSan (#14252, #14673, #17006)
* 2018: Continuous testing under AddressSanitizer – ASan (#14794, #17205, #17674)
* 2018: Continuous testing under ThreadSanitizer – TSan (#14829)
* 2019: Continuous testing in an unsigned char environment (`-funsigned-char`) (#15134)
* 2019: Continuous compile-time testing of assumptions we're making (#15391)
* 2019: Continuous testing of fuzz test cases under Valgrind (#17633, #18159, #18166)
* 2020: Finally... MemorySanitizer – MSAN! :)
What is the next step? What tools should we add to CI to keep bugs from entering `master`? :)
ACKs for top commit:
MarcoFalke:
ACK 870f0cd2a0
Tree-SHA512: 38327c8b75679d97d469fe42e704cacd1217447a5a603701dd8a58ee50b3be2c10248f8d68a479ed081c0c4b254589d3081c9183f991640b06ef689061f75578
a92e48b02d test: move TEST_RUNNER_EXTRA into native tsan setup (fanquake)
Pull request description:
`feature_block.py` is being run in the tsan job, i.e [here](https://travis-ci.org/github/bitcoin/bitcoin/jobs/703122309), even though it should be excluded. My hasty assumption is that this will fix it. In any case, all other instances of `TEST_RUNNER_EXTRA` seem to have moved out of `.travis.yml` and into the different CI configurations.
ACKs for top commit:
MarcoFalke:
ACK a92e48b02d
practicalswift:
ACK a92e48b02d -- patch looks correct
hebasto:
ACK a92e48b02d, I have reviewed the code and it looks OK, I agree it can be merged.
Tree-SHA512: 86057bef2cc87c6acdbbf94f8cd7a5147510448c3e67aacde8daf247e3ccf649cfc5afbbd10693e084f426042d98150616c0e49bfa5f32b949dff9cebd2fd95d
adf543d714 darwin: pass mlinker-version so that clang enables new features (Cory Fields)
2418f739f7 macos: Bump to xcode 11.3.1 and 10.15 SDK (Cory Fields)
5c2c835433 depends: bump MacOS toolchain (Cory Fields)
85b5e42088 contrib: macdeploy: Remove historical extraction notes (Carl Dong)
351beb5c9a contrib: macdeploy: Use apple-sdk-tools instead of xar+pbzx (Carl Dong)
fbcfcf6954 native_cctools: Don't use libc++ from pinned clang (Carl Dong)
3381e4a189 Adapt rest of tooling to new SDK naming scheme (Carl Dong)
b3394ab235 contrib: macdeploy: Correctly generate macOS SDK (Carl Dong)
Pull request description:
This PR achieves 3 main things:
1. It simplifies the macOS SDK generation by putting the logic inside a (semi-)portable python3 script `gen-sdk`
2. It transitions us to using `libc++` headers extracted from the `Xcode.app`, which is more correct as those headers better match the `.tbd` library stubs we use from the `MacOSX.sdk` (located under the same `Xcode.app`). Previously, we used `libc++` headers copied from our downloaded, pinned clang (see `native_cctools.mk`).
3. It bumps the macOS toolchain in a way that fulfills all of the following constraints:
1. The new SDK should support compiling with C++17 (our current one doesn't)
2. The new toolchain should not change our minimum supported macOS version (`-mmacosx-version-min`)
3. The new toolchain should expect to use a version of `cctools` that is supported by https://github.com/tpoechtrager/cctools-port
For the constraints in (3), you can reference [this chart](https://en.wikipedia.org/wiki/Xcode#Xcode_7.0_-_11.x_(since_Free_On-Device_Development)) to see that the newest toolchain we can use with our `cctools-port` is `11.3.1`, and the rest of the constraints were tested with local builds.
#### But [the other Wikipedia chart](https://en.wikipedia.org/wiki/Xcode#Xcode_11.x_(since_SwiftUI_framework)) says that the "min macOS to run" for Xcode 11.3.1 is 10.14.4, doesn't that violate constraint (ii)?
This confused me at first too, but the "min macOS to run" is for the Xcode.app App itself. The SDK still supports 10.12, as evident in a few plist files and as proven through local builds.
#### Why bundle all of this together in a single PR?
We need (1) and (2) together, because if we don't, manually adding the `libc++` headers and writing that out in a `README.md` is going to result in a lot of user error, so it's great to have these together to be more correct and also make it easier on the user at the same time.
We need (3) together with everything else because bumping (or in the case of (1), renaming) the SDK requires some human coordination and may break some builds. And since it's not that complicated a change, it makes sense to do it together with the rest.
ACKs for top commit:
theuni:
ACK adf543d714.
fanquake:
ACK adf543d714 - I'll take a look at the linker issue.
Tree-SHA512: 3813b69ebfe9610bee14100f26296fb5438d9bf0dd184ea55e6c38f5ebd94f7c171d98b150fc9e52fde626533f347f7ec51a2b72b79859d946284f578c1084a3
0012471391 build: turn on --enable-c++17 by --enable-fuzz (Vasil Dimov)
Pull request description:
Fuzzing code uses C++17 specific code (e.g. std::optional), so it is not
possible to compile with --enable-fuzz and without --enable-c++17.
Thus, turn on --enable-c++17 whenever --enable-fuzz is used.
ACKs for top commit:
hebasto:
ACK 0012471391, tested on Linux Mint 19.3 (x86_64); verified that it fails to compile with `--enable-fuzz` and without `--enable-c++17` on master.
Tree-SHA512: 290531ea8d79de3b9251ea4ad21e793478b18150cc0124eea1e50c3a4ed92bab89c3e70ed0aa526906f8723ea952cdba4268f1560ae4be9bd25b9e4f9b97436c
Fuzzing code uses C++17 specific code (e.g. std::optional), so it is not
possible to compile with --enable-fuzz and without --enable-c++17.
Thus, turn on --enable-c++17 whenever --enable-fuzz is used.
losh11