5ca90f8b59 scripts: add MACHO lazy bindings check to security-check.py (fanquake)
Pull request description:
This is a slightly belated follow up to #17686 and some discussion with Cory. It's not entirely clear if we should make this change due to the way the macOS dynamic loader appears to work. However I'm opening this for some discussion. Also related to #17768.
#### Issue:
[`LD64`](https://opensource.apple.com/source/ld64/) doesn't set the [MH_BINDATLOAD](https://opensource.apple.com/source/xnu/xnu-6153.11.26/EXTERNAL_HEADERS/mach-o/loader.h.auto.html) bit in the header of MACHO executables, when building with `-bind_at_load`. This is in contradiction to the [documentation](https://opensource.apple.com/source/ld64/ld64-450.3/doc/man/man1/ld.1.auto.html):
```bash
-bind_at_load
Sets a bit in the mach header of the resulting binary which tells dyld to
bind all symbols when the binary is loaded, rather than lazily.
```
The [`ld` in Apples cctools](https://opensource.apple.com/source/cctools/cctools-927.0.2/ld/layout.c.auto.html) does set the bit, however the [cctools-port](https://github.com/tpoechtrager/cctools-port/) that we use for release builds, bundles `LD64`.
However; even if the linker hasn't set that bit, the dynamic loader ([`dyld`](https://opensource.apple.com/source/dyld/)) doesn't seem to ever check for it, and from what I understand, it looks at a different part of the header when determining whether to lazily load symbols.
Note that our release binaries are currently working as expected, and no lazy loading occurs.
#### Example:
Using a small program, we can observe the behaviour of the dynamic loader.
Conducted using:
```bash
clang++ --version
Apple clang version 11.0.0 (clang-1100.0.33.17)
Target: x86_64-apple-darwin18.7.0
ld -v
@(#)PROGRAM:ld PROJECT:ld64-530
BUILD 18:57:17 Dec 13 2019
LTO support using: LLVM version 11.0.0, (clang-1100.0.33.17) (static support for 23, runtime is 23)
TAPI support using: Apple TAPI version 11.0.0 (tapi-1100.0.11)
```
```cpp
#include <iostream>
int main() {
std::cout << "Hello World!\n";
return 0;
}
```
Compile and check the MACHO header:
```bash
clang++ test.cpp -o test
otool -vh test
...
Mach header
magic cputype cpusubtype caps filetype ncmds sizeofcmds flags
MH_MAGIC_64 X86_64 ALL LIB64 EXECUTE 16 1424 NOUNDEFS DYLDLINK TWOLEVEL WEAK_DEFINES BINDS_TO_WEAK PIE
# Run and dump dynamic loader bindings:
DYLD_PRINT_BINDINGS=1 DYLD_PRINT_TO_FILE=no_bind.txt ./test
Hello World!
```
Recompile with `-bind_at_load`. Note still no `BINDATLOAD` flag:
```bash
clang++ test.cpp -o test -Wl,-bind_at_load
otool -vh test
Mach header
magic cputype cpusubtype caps filetype ncmds sizeofcmds flags
MH_MAGIC_64 X86_64 ALL LIB64 EXECUTE 16 1424 NOUNDEFS DYLDLINK TWOLEVEL WEAK_DEFINES BINDS_TO_WEAK PIE
...
DYLD_PRINT_BINDINGS=1 DYLD_PRINT_TO_FILE=bind.txt ./test
Hello World!
```
If we diff the outputs, you can see that `dyld` doesn't perform any lazy bindings when the binary is compiled with `-bind_at_load`, even if the `BINDATLOAD` flag is not set:
```diff
@@ -1,11 +1,27 @@
+dyld: bind: test:0x103EDF030 = libc++.1.dylib:__ZNKSt3__16locale9use_facetERNS0_2idE, *0x103EDF030 = 0x7FFF70C9FA58
+dyld: bind: test:0x103EDF038 = libc++.1.dylib:__ZNKSt3__18ios_base6getlocEv, *0x103EDF038 = 0x7FFF70CA12C2
+dyld: bind: test:0x103EDF068 = libc++.1.dylib:__ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE6sentryC1ERS3_, *0x103EDF068 = 0x7FFF70CA12B6
+dyld: bind: test:0x103EDF070 = libc++.1.dylib:__ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE6sentryD1Ev, *0x103EDF070 = 0x7FFF70CA1528
+dyld: bind: test:0x103EDF080 = libc++.1.dylib:__ZNSt3__16localeD1Ev, *0x103EDF080 = 0x7FFF70C9FAE6
<trim>
-dyld: lazy bind: test:0x10D4AC0C8 = libsystem_platform.dylib:_strlen, *0x10D4AC0C8 = 0x7FFF73C5C6E0
-dyld: lazy bind: test:0x10D4AC068 = libc++.1.dylib:__ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE6sentryC1ERS3_, *0x10D4AC068 = 0x7FFF70CA12B6
-dyld: lazy bind: test:0x10D4AC038 = libc++.1.dylib:__ZNKSt3__18ios_base6getlocEv, *0x10D4AC038 = 0x7FFF70CA12C2
-dyld: lazy bind: test:0x10D4AC030 = libc++.1.dylib:__ZNKSt3__16locale9use_facetERNS0_2idE, *0x10D4AC030 = 0x7FFF70C9FA58
-dyld: lazy bind: test:0x10D4AC080 = libc++.1.dylib:__ZNSt3__16localeD1Ev, *0x10D4AC080 = 0x7FFF70C9FAE6
-dyld: lazy bind: test:0x10D4AC070 = libc++.1.dylib:__ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE6sentryD1Ev, *0x10D4AC070 = 0x7FFF70CA1528
```
Note: `dyld` also has a `DYLD_BIND_AT_LAUNCH=1` environment variable, that when set, will force any lazy bindings to be non-lazy:
```bash
dyld: forced lazy bind: test:0x10BEC8068 = libc++.1.dylib:__ZNSt3__113basic_ostream
```
#### Thoughts:
After looking at the dyld source, I can't find any checks for `MH_BINDATLOAD`. You can see the flags it does check for, such as MH_PIE or MH_BIND_TO_WEAK [here](https://opensource.apple.com/source/dyld/dyld-732.8/src/ImageLoaderMachO.cpp.auto.html).
It seems that the lazy binding of any symbols depends on whether or not [lazy_bind_size](https://opensource.apple.com/source/xnu/xnu-6153.11.26/EXTERNAL_HEADERS/mach-o/loader.h.auto.html) from the `LC_DYLD_INFO_ONLY` load command is > 0. Which was mentioned in [#17686](https://github.com/bitcoin/bitcoin/pull/17686#issue-350216254).
#### Changes:
This PR is one of [Corys commits](7b6ba26178), that I've rebased and modified to make build. I've also included an addition to the `security-check.py` script to check for the flag.
However, given the above, I'm not entirely sure this patch is the correct approach. If the linker no-longer inserts it, and the dynamic loader doesn't look for it, there might be little benefit to setting it. Or, maybe this is an oversight from Apple and needs some upstream discussion. Looking for some thoughts / Concept ACK/NACK.
One alternate approach we could take is to drop the patch and modify security-check.py to look for `lazy_bind_size` == 0 in the `LC_DYLD_INFO_ONLY` load command, using `otool -l`.
ACKs for top commit:
theuni:
ACK 5ca90f8b59
Tree-SHA512: 444022ea9d19ed74dd06dc2ab3857a9c23fbc2f6475364e8552d761b712d684b3a7114d144f20de42328d1a99403b48667ba96885121392affb2e05b834b6e1c
0eeb0468e7 net: Hardcoded seeds update for 0.20 (Wladimir J. van der Laan)
Pull request description:
Update hardcoded seeds from http://bitcoin.sipa.be/seeds.txt.gz,
according to release process.
Output from makeseeds.py:
```
IPv4 IPv6 Onion Pass
1364173 244127 2454 Initial
1364173 244127 2454 Skip entries with invalid address
1129552 213117 2345 After removing duplicates
1129548 213117 2345 Skip entries from suspicious hosts
338216 191944 2249 Enforce minimal number of blocks
336851 188993 2189 Require service bit 1
6998 1520 150 Require minimum uptime
5682 1290 89 Require a known and recent user agent
5622 1279 89 Filter out hosts with multiple bitcoin ports
512 146 89 Look up ASNs and limit results per ASN and per net
```
Top commit has no ACKs.
Tree-SHA512: ce1c2cda18dd5bd22586a5283a0877f3bd890437cc29dc1d85452ba4a4d28032f591c8b37f3329e8e649556cf83750b6949a068fad76d1773853d93014609da0
Update hardcoded seeds from seeds_emzy.txt seeds_lukejr.txt
seeds_sipa.txt seeds_sjors.txt, according to release process.
Output from makeseeds.py:
```
IPv4 IPv6 Onion Pass
1364173 244127 2454 Initial
1364173 244127 2454 Skip entries with invalid address
1129552 213117 2345 After removing duplicates
1129548 213117 2345 Skip entries from suspicious hosts
338216 191944 2249 Enforce minimal number of blocks
336851 188993 2189 Require service bit 1
6998 1520 150 Require minimum uptime
5682 1290 89 Require a known and recent user agent
5622 1279 89 Filter out hosts with multiple bitcoin ports
512 146 89 Look up ASNs and limit results per ASN and per net
```
332f373a9d [scripts] previous_release: improve failed download error message (Sebastian Falbesoner)
Pull request description:
Currently, if the earlier release build/fetch script `previous_release.sh` is invoked with the option `-b` (intending to fetch a binary package from `https://bitcoin.org`) and the download fails, the user sees the following confusing output:
```
$ contrib/devtools/previous_release.sh -r -b v0.9.5
[...]
gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
```
This implies that the download worked, but the archive is corrupted, when in reality the HTML document containing the delivery fail reason (most likely 404 Not Found) is saved and tried to get unpacked. In contrast to wget, curl is a bit stubborn and needs explicit instructions to react to server errors via the flag `-f` (outputs error message and returns error code, ideal for scripts): https://curl.haxx.se/docs/manpage.html#-f
On the PR branch, the output on failed download looks now the following:
```
$ contrib/devtools/previous_release.sh -r -b v0.9.5
[...]
curl: (22) The requested URL returned error: 404 Not Found
Download failed.
```
ACKs for top commit:
fanquake:
ACK 332f373a9d
Tree-SHA512: 046c931ad9e78aeb2d13faa4866d46122ed325aa142483547c2b04032d03223ed2411783b00106fcab0cd91b2f78691531ac526ed7bb3ed7547b6e2adbfb2e93
3e0df92bf2 Update with new Windows code signing certificate (Andrew Chow)
Pull request description:
The current Windows code signing certificate is about expire (on March 26th 2020). As I have volunteered to take over the Windows code signing duties, I've purchased a new Windows code signing certificate with the same CA and under the same organization (Bitcoin Core Code Signing Association).
A signature by the old certificate over the new certificate has been provided to me. This signature can be verified using
```
openssl cms -verify -inform pem -purpose any -content path/to/new/win-codesign.cert -CAfile path/to/old/win-codesign.cert -certfile path/to/old/win-codesign.cert
```
The verification should succeed and the new certificate will be printed out. This can be compared to the contents of `win-codesign.cert`.
```
-----BEGIN PKCS7-----
MIIC3AYJKoZIhvcNAQcCoIICzTCCAskCAQExDzANBglghkgBZQMEAgEFADALBgkq
hkiG9w0BBwExggKkMIICoAIBATCBkTB8MQswCQYDVQQGEwJHQjEbMBkGA1UECBMS
R3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHEwdTYWxmb3JkMRgwFgYDVQQKEw9T
ZWN0aWdvIExpbWl0ZWQxJDAiBgNVBAMTG1NlY3RpZ28gUlNBIENvZGUgU2lnbmlu
ZyBDQQIRALWcUnSOxv9FQW3xdaMDO6swDQYJYIZIAWUDBAIBBQCggeQwGAYJKoZI
hvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMjAwMzI0MjA0ODM3
WjAvBgkqhkiG9w0BCQQxIgQgtLkmnuSQyczDlJSnJeqbi61p3iJ/rpFABrY8JWBO
o74weQYJKoZIhvcNAQkPMWwwajALBglghkgBZQMEASowCwYJYIZIAWUDBAEWMAsG
CWCGSAFlAwQBAjAKBggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcN
AwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwDQYJKoZIhvcNAQEBBQAEggEA
XaCl3Q8HwI9VpLCb9OY9eQh0QOPyl1KWEc3TP3UvwZwR4/gXkfPOKKf19UnS8eRB
48SgUKRMYWoDYfSVUJRMda9BLkbJbQlHG3LFXhSY2alajpPXEHcMto/XPhVAmqzL
w6aSNY0Gaorow696JHpetpKqAAlL1r2GjeaPYi2aZyIAifuhay/qwA+ig0SqzGOw
UdgFZWMyS5yanq8/WlLCCql6kKOzT4tEqUaleD7R1q8BTcG2+fmhWR8WwJLpIV6y
7GAqt0Cocu8sYpTNBNk8iKHxzZ2hMZKJpH9lHZuiJ/9vSercrvDy2R4/MG+KnBWb
OyiFAt2mC51+63RhLOMJfg==
-----END PKCS7-----
```
ACKs for top commit:
laanwj:
ACK 3e0df92bf2
theuni:
ACK 3e0df92bf2.
Tree-SHA512: 4210f4db1e805ab11231fbae49ea197257c6f7e44f1f6219685b63831704984d824ac2f9e0a3b1bd2655953af72636a474f077cb859fb35852551f5a9f8fbde3
e4d366788b build: Drop needless EXTRA_DIST content (Hennadii Stepanov)
6c4da59f5b build: Drop SOURCEDIST reordering (Hennadii Stepanov)
5e6b8b3912 build: Use git archive as source tarball (Hennadii Stepanov)
Pull request description:
This PR:
- is an alternative to #17104
- closes#16734
- closes#6753
The idea is clear described by some developers:
- [MarcoFalke](https://github.com/bitcoin/bitcoin/pull/17097#issuecomment-540691850):
> This whole concept of explicitly listing each and every file manually (or with a fragile wildcard) is an obvious sisyphean task. I'd say all we need to do is run git archive and be done with it forever, see #16734, #6753, #11530 ...
- [laanwj](https://github.com/bitcoin/bitcoin/pull/17097#issuecomment-540706025):
> I agree, I've never been a fan of it. I don't think we have any files in the git repository we don't want to ship in the source tarball.
---
The suggested changes have a downside which is pointed by [**luke-jr**](https://github.com/bitcoin/bitcoin/pull/17104#issuecomment-540828045):
> ... but the distfile needs to include autogen-generated files.
This means that a user is not able to run `./configure && make` right away. One must run `./autogen.sh` at first.
Here are opinions about mandatory use of `./autogen.sh`:
- [ryanofsky](https://github.com/bitcoin/bitcoin/issues/16734#issuecomment-534139356):
> It's probably ok to require autogen. I think historically configure scripts were supposed to work on obscure unix systems that would just have a generic shell + make tool + c compiler, and not necessarily need gnu packages like m4 which are needed for autogen.
- [laanwj](https://github.com/bitcoin/bitcoin/issues/16734#issuecomment-540729483):
> I also think it's fine to require autogen. What is one dependency more, if you're building from source.
---
~Also this PR provides Windows users with ZIP archives of the sources. Additionally the commit ID is stored in these ZIP files as a file comment:~
---
Note for reviewers: please verify is `git archive` output deterministic?
ACKs for top commit:
MarcoFalke:
re-ACK e4d366788b, only change is adding two dots in a the path 🛳
laanwj:
ACK e4d366788b
Tree-SHA512: d1153d3ca4a580696019b92be3555ab004d197d9a2146aacff9d3150eb7093b7d40eebd6eea12d861d93ff62d62b68706e04e64dbe5ea796ff6757486e462193
before:
------------------------------------------------------------
$ contrib/devtools/previous_release.sh -r -b v0.9.5
[...]
gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
------------------------------------------------------------
now:
------------------------------------------------------------
$ contrib/devtools/previous_release.sh -r -b v0.9.5
[...]
curl: (22) The requested URL returned error: 404 Not Found
Download failed.
------------------------------------------------------------
Previously, Guix would produce a gcc which did not know to use the SSP
function from glibc, and required a gcc make flag for it to do so, in my
attempt to fix it upstream I realized that this is no longer the case.
This can be verified by performing a Guix build and doing
readelf -s ... | grep __stack_chk
to check that symbols are coming from glibc, and doing
readelf -d ... | grep NEEDED | grep ssp
to see that libssp.so is not being depended on
fae9084ac5 build: Skip i686 build by default in guix and gitian (MarcoFalke)
fa55a2554c depends: Remove reference to win32 (MarcoFalke)
Pull request description:
Closes#17504
Now that we no longer provide downloads for i686 on our website (https://bitcoincore.org/en/download/), there is no need to build them by default.
i686 can still be built in depends (tested by ci/travis) and in guix/gitian by setting the appropriate `HOSTS`.
ACKs for top commit:
practicalswift:
ACK fae9084ac5 -- patch looks correct
dongcarl:
ACK fae9084ac5 patch looks correct
laanwj:
Code review ACK fae9084ac5
hebasto:
ACK fae9084ac5, I have reviewed the code and it looks OK, I agree it can be merged.
Tree-SHA512: b000c19a2cd2a596a52028fa298c4022c24cfdfc1bdb3795a90916d0a00a32e4dd22278db93790b6a11724e08ea8451f4f05c77bc40d1664518e11a8c82d6e29
677fb8e923 test: Add ubsan surpression for crc32c (Wladimir J. van der Laan)
8e68bb1dde build: Disable msvc warning 4722 for leveldb build (Aaron Clauson)
be23949765 build: MSVC changes for leveldb update (Aaron Clauson)
9ebdf04757 build: CRC32C build system integration (Wladimir J. van der Laan)
402252a808 build: Add LCOV exception for crc32c (Wladimir J. van der Laan)
3a037d0067 test: Add crc32c exception to various linters and generation scripts (Wladimir J. van der Laan)
84ff1b2076 test: Add crc32c to subtree check linter (Wladimir J. van der Laan)
7cf13a5134 doc: Add crc32c subtree to developer notes (Wladimir J. van der Laan)
24d02a9ac0 build: Update build system for new leveldb (Wladimir J. van der Laan)
2e1819311a Squashed 'src/crc32c/' content from commit 224988680f7673cd7c769963d4035cb315aa3388 (Wladimir J. van der Laan)
66480821b3 Squashed 'src/leveldb/' changes from f545dfabff4c2e9836efed094dba99a34fbc6b88..f8ae182c1e5176d12e816fb2217ae33a5472fdd7 (Wladimir J. van der Laan)
Pull request description:
This updates leveldb to currently newest upstream commit 0c40829872:
- CRC32C hardware acceleration is now an external library [crc32c](https://github.com/google/crc32c). This adds acceleration on ARM, and should be faster on x86 because of using prefetch. It also makes it easy to support similar instruction sets on other platforms in the future.
- Thread handling uses C++11, instead of platform specific code.
- Native windows environment was added. No need to maintain our own hacky one, anymore.
- Upstream now builds using CMake. This doesn't mean we need to use that (phew), but internal configuration changed to a a series of checks, instead of OS profiles. This means the blanket error "Cannot build leveldb for $host. Please file a bug report' is removed.
All changes: a53934a3ae...0c40829872
Pretty much all our changes have been subsumed by upstream, so we figured it was cleaner to start over with a new branch from upstream with the still-relevant patches applied: https://github.com/bitcoin-core/leveldb/tree/bitcoin-fork-new
There's quite some testing to be done (see below). See https://github.com/bitcoin-core/leveldb/issues/25 and https://github.com/bitcoin-core/leveldb/pull/26 for more history and context.
TODO:
- [x] Subtree `crc32c`
- [x] Make linters happy about crc32 subtree
- [x] Integrate `crc32c` library into build system
- [x] MSVC build system
ACKs for top commit:
sipa:
ACK 677fb8e923
Tree-SHA512: 37ee92a750e053e924bc4626b12bb3fd81faa9f8c5ebaa343931fee810c45ba05aa6051fdea82535fa351bf2be7297801b98af9469865fc5ead771650a5d6240
317fb96de9 Add search for first blk file with pruned node (Rjected)
Pull request description:
<!--
*** Please remove the following help text before submitting: ***
Pull requests without a rationale and clear improvement may be closed
immediately.
-->
<!--
Please provide clear motivation for your patch and explain how it improves
Bitcoin Core user experience or Bitcoin Core developer experience
significantly:
* Any test improvements or new tests that improve coverage are always welcome.
* All other changes should have accompanying unit tests (see `src/test/`) or
functional tests (see `test/`). Contributors should note which tests cover
modified code. If no tests exist for a region of modified code, new tests
should accompany the change.
* Bug fixes are most welcome when they come with steps to reproduce or an
explanation of the potential issue as well as reasoning for the way the bug
was fixed.
* Features are welcome, but might be rejected due to design or scope issues.
If a feature is based on a lot of dependencies, contributors should first
consider building the system outside of Bitcoin Core, if possible.
* Refactoring changes are only accepted if they are required for a feature or
bug fix or otherwise improve developer experience significantly. For example,
most "code style" refactoring changes require a thorough explanation why they
are useful, what downsides they have and why they *significantly* improve
developer experience or avoid serious programming bugs. Note that code style
is often a subjective matter. Unless they are explicitly mentioned to be
preferred in the [developer notes](/doc/developer-notes.md), stylistic code
changes are usually rejected.
-->
When bitcoind is running in pruned mode, producing a hashlist with `./linearize-hashes.py linearize.cfg > hashlist.txt` and then executing `linearize-data.py linearize.cfg` will produce:
```
Read 313001 hashes
Input file /home/dan/.bitcoin/blocks/blk00000.dat
Premature end of block data
```
This happens because `linearize-data` starts by attempting to process `blk00000.dat` regardless of whether or not `blk00000.dat` actually exists - this may not be the case if working with a pruned node.
This PR adds a function which finds the first block file that does exist, and calls that function when the `BlockDataCopier` is initialized.
This is a refactor of #16431.
<!--
Bitcoin Core has a thorough review process and even the most trivial change
needs to pass a lot of eyes and requires non-zero or even substantial time
effort to review. There is a huge lack of active reviewers on the project, so
patches often sit for a long time.
-->
ACKs for top commit:
darosior:
ACK 317fb96de9
laanwj:
Code review ACK 317fb96de9
theStack:
Code review ACK 317fb96de9
Tree-SHA512: fc8014282df6cfe7b267e64db8ce7d82b86b758c302fbfea4a3c39b62d93512f5c2e31a0de4e9c5ec18fc0268c917f011257d37b45afaef6033eec90e4aa585f
b3c4d9bac6 test: rename test suite name "tx_validationcache_tests" to match filename (Sebastian Falbesoner)
Pull request description:
Quoting `src/test/README.md`, '`Adding test cases`':
> "The file naming convention is `<source_filename>_tests.cpp`
> and such files should wrap their tests in a test suite
> called `<source_filename>_tests`."
Currently the unit test source file `txvalidationcache_tests.cpp` contains a unit test suite with the name `tx_validationcache_tests`, which is fixed by this PR. The following shell script shows that this is the only mismatch and for all other unit test source files the test suite names are correct:
```
#!/bin/bash
shopt -s globstar
for test_full_filename in **/*_tests.cpp; do
test_name_file=`basename $test_full_filename .cpp`
test_name_suite=`sed -n "s/^.*TEST_SUITE(\(.*_tests\).*$/\1/p" $test_full_filename`
if [ $test_name_file != $test_name_suite ]; then
echo "TestFilename: $test_name_file != TestSuitname: $test_name_suite"
fi
done
```
ACKs for top commit:
practicalswift:
ACK b3c4d9bac6 -- expected naming is better than unexpected naming :)
kristapsk:
ACK b3c4d9bac6
Tree-SHA512: 29d409b1eb22057ee2cc407508e2580d2bc03f412401df11b8ecf77be5ada6bda8f7d2cb5338c5e079490fa12242c1fd6230a09e47252c1b0d9fe535a828ca4c
88c83636d5 guix: Update documentation for time-machine (Carl Dong)
e6050884fd guix: Pin Guix using `guix time-machine` (Carl Dong)
Pull request description:
An alternative to #16519, pinning our version of Guix and eliminating a `guix pull` and changing the default Guix profile of builders.
I think this method might be superior, as it:
- Eliminates the possibility of future changes to the `guix environment` command line interface breaking our builds
- Eliminates the need to set up a separate channel repo
It is a more general pinning solution than #16519.
-----
The reason why I didn't originally propose this is because `guix time-machine` is a recent addition to Guix, only available since `f675f8dec73d02e319e607559ed2316c299ae8c7`
ACKs for top commit:
fanquake:
ACK 88c83636d5
Tree-SHA512: 85e03b0987ffa86da73e02801e1cd8b7622698d70c4ba4e60561611be1e9717d661c2811a59b3e137b1b8eef2d0ba37c313867d035ebc89c3bd06a23a078064a
Quoting src/test/README.md, 'Adding test cases':
"The file naming convention is `<source_filename>_tests.cpp`
and such files should wrap their tests in a test suite
called `<source_filename>_tests`."
Currently the unit test source file txvalidationcache_tests.cpp contains a unit
test suite with the name tx_validationcache_tests, which is fixed by this commit.
The following shell script shows that this is the only mismatch and for all other
unit test source files the test suite names are correct:
#!/bin/bash
shopt -s globstar
for test_full_filename in **/*_tests.cpp; do
test_name_file=`basename $test_full_filename .cpp`
test_name_suite=`sed -n "s/^.*TEST_SUITE(\(.*_tests\).*$/\1/p" $test_full_filename`
if [ $test_name_file != $test_name_suite ]; then
echo "TestFilename: $test_name_file != TestSuitname: $test_name_suite"
fi
done
2d23082cbe bump test timeouts so that functional tests run in valgrind (Micky Yun Chan)
Pull request description:
ci/tests: Bump timeouts so all functional tests run on travis in valgrind #17763
Top commit has no ACKs.
Tree-SHA512: 5a8c6e2ea02b715facfcb58c761577be15ae58c45a61654beb98c2c2653361196c2eec521bcae4a9a1bab8e409d6807de771ef4c46d3d05996ae47a22d499d54
c491368d8c scripts: add MACHO dylib checking to symbol-check.py (fanquake)
76bf97213f scripts: fix check-symbols & check-security argument passing (fanquake)
Pull request description:
Based on #17857.
This adds dynamic library checks for MACHO executables to symbol-check.py. The script has been modified to function more like `security-check.py`. The error output is now also slightly different. i.e:
```bash
# Linux x86
bitcoin-cli: symbol operator new[](unsigned long) from unsupported version GLIBCXX_3.4
bitcoin-cli: export of symbol vtable for std::basic_ios<char, std::char_traits<char> > not allowed
bitcoin-cli: NEEDED library libstdc++.so.6 is not allowed
bitcoin-cli: failed IMPORTED_SYMBOLS EXPORTED_SYMBOLS LIBRARY_DEPENDENCIES
# RISCV (skips exported symbols checks)
bitcoin-tx: symbol operator new[](unsigned long) from unsupported version GLIBCXX_3.4
bitcoin-tx: NEEDED library libstdc++.so.6 is not allowed
bitcoin-tx: failed IMPORTED_SYMBOLS LIBRARY_DEPENDENCIES
# macOS
Checking macOS dynamic libraries...
libboost_filesystem.dylib is not in ALLOWED_LIBRARIES!
bitcoind: failed DYNAMIC_LIBRARIES
```
Compared to `v0.19.0.1` the macOS allowed dylibs has been slimmed down somewhat:
```diff
src/qt/bitcoin-qt:
/usr/lib/libSystem.B.dylib
-/System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
-/System/Library/Frameworks/Security.framework/Versions/A/Security
-/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics
-/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL
-/System/Library/Frameworks/AGL.framework/Versions/A/AGL
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/usr/lib/libc++.1.dylib
-/System/Library/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
/System/Library/Frameworks/CoreText.framework/Versions/A/CoreText
/System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO
/usr/lib/libobjc.A.dylib
```
ACKs for top commit:
laanwj:
ACK c491368d8c
Tree-SHA512: f8624e4964e80b3e0d34e8d3cc33f3107938f3ef7a01c07828f09b902b5ea31a53c50f9be03576e1896ed832cf2c399e03a7943a4f537a1e1c705f3804aed979
fae75306ba scripted-diff: Set gitian arch back to amd64 (MarcoFalke)
Pull request description:
This was required to allow gitian builds on non-amd64 architecture, however, it seems to break the current builds (with lxc), see https://github.com/bitcoin/bitcoin/pull/17409#issuecomment-554099626
Also, the gititan builds wouldn't be deterministic across arches anyway, see #17468
So instead of wasting more time on this, revert the change and hope that guix allows to compile on non-amd64 architectures.
Top commit has no ACKs.
Tree-SHA512: 801e9a30ae1b0882ef45d5eb3a3cf80f3ace3b99db046069dbd95b6162119e977e3cf3134287d1ac5d09483906206acc71e1ac34d6b74dbc533d46aaf73f5cc2
e1c582cbaa contrib: makeseeds: Read suspicious hosts from a file instead of hardcoding (Sanjay K)
Pull request description:
referring to: https://github.com/bitcoin/bitcoin/issues/17020
good first issue: reading SUSPICIOUS_HOSTS from a file.
I haven't changed the base hosts that were included in the original source, just made it readable from a file.
ACKs for top commit:
practicalswift:
ACK e1c582cbaa -- diff looks correct
Tree-SHA512: 18684abc1c02cf52d63f6f6ecd98df01a9574a7c470524c37e152296504e2e3ffbabd6f3208214b62031512aeb809a6d37446af82c9f480ff14ce4c42c98e7c2
fac86ac7b3 scripted-diff: Add missed copyright headers (Hennadii Stepanov)
6fde9d5e47 script: Update EXLUDE list in copyright_header.py (Hennadii Stepanov)
1998152f15 script: Add empty line after C++ copyright (Hennadii Stepanov)
071f2fc204 script: Add ability to insert copyright to *.sh (Hennadii Stepanov)
Pull request description:
This PR improves `contrib/devtools/copyright_header.py` script and adds copyright headers to the files in `src` and `test` directories with two exceptions:
- [`src/reverse_iterator.h`](https://github.com/bitcoin/bitcoin/blob/master/src/reverse_iterator.h) (added to exceptions)
- [`src/test/fuzz/FuzzedDataProvider.h`](https://github.com/bitcoin/bitcoin/blob/master/src/test/fuzz/FuzzedDataProvider.h) (added to exceptions)
On master 5622d8f3156a293e61d0964c33d4b21d8c9fd5e0:
```
$ ./contrib/devtools/copyright_header.py report . | grep zero
25 with zero copyrights
```
With this PR:
```
$ ./contrib/devtools/copyright_header.py report . | grep zero
2 with zero copyrights
```
~I am uncertain about our copyright policy with `build_msvc` and `contrib` directories content, so they are out of scope of this PR.~
ACKs for top commit:
MarcoFalke:
ACK fac86ac7b3
Tree-SHA512: d7832c4a7a1a3b7806119775b40ec35d7982f49ff0e6199b8cee4c0e0a36e68d51728b6ee9924b1c161df4bc6105bd93391b79d42914357fa522f499cb113fa8
582e66b6e7 doc: Added regtest config for linearize script (Gr0kchain)
Pull request description:
Updated the example-linearize.cfg file to include support for the regtest chain network config which is used by the ./linearize-data.py
Problem:
Without the regtest magic, genesis hash and path config, the `linearize-data.py` script cannot generate a bootstrap.dat file.
Example:
./linearize-data.py ./linearize.cfg
Read 102 hashes
Genesis block not found in hashlist
Solution:
Added netmagic, genesis and input example parameters to file.
Resolution
1. Starting bitcoind in regtest mode
2. bitcoin-cli generatetoaddress 101 $(bitcoin-cli getnewaddress)
3. ./linearize-hashes.py ./linearize.cfg > ./hashlist.txt
4. ./linearize-data.py ./linearize.cfg
```
$ ./linearize-data.py ./linearize.cfg
Read 102 hashes
Input file /Users/gr0kchain/.bitcoin/regtest/blocks/blk00000.dat
Output file /Users/gr0kchain/Downloads/bootstrap.dat
Done (102 blocks written)
```
ACKs for top commit:
fanquake:
ACK 582e66b6e7
Tree-SHA512: 699e92e740e68e2e5190ba37538efbbe3e4d4e725ebd6af704a0cf5517683b691754f7ea097bf840845d2b53b793c63258d406e9bd37922db810cf58bed053c3
-BEGIN VERIFY SCRIPT-
s() { contrib/devtools/copyright_header.py insert "$1"; }
s build_msvc/bitcoin_config.h
s build_msvc/msvc-autogen.py
s build_msvc/testconsensus/testconsensus.cpp
s contrib/devtools/circular-dependencies.py
s contrib/devtools/gen-manpages.sh
s contrib/filter-lcov.py
s contrib/gitian-build.py
s contrib/install_db4.sh
s src/crypto/sha256_avx2.cpp
s src/crypto/sha256_sse41.cpp
s src/fs.cpp
s src/qt/test/addressbooktests.cpp
s src/qt/test/addressbooktests.h
s src/qt/test/util.cpp
s src/qt/test/util.h
s src/qt/test/wallettests.cpp
s src/qt/test/wallettests.h
s src/test/blockchain_tests.cpp
s test/functional/combine_logs.py
s test/lint/lint-locale-dependence.sh
sed -i '1G' test/lint/lint-shebang.sh
s test/lint/lint-shebang.sh
-END VERIFY SCRIPT-
fanquake