9edd713c18 build: Fix MSVC linker /SubSystem option for bitcoin-qt.exe (Hennadii Stepanov)
Pull request description:
On master (6f3fbc062f), running `bitcoin-qt.exe`, which was built with MSVC, causes a terminal window open along with the GUI.
This PR fixes such behavior. See Microsoft [docs](https://docs.microsoft.com/en-us/cpp/build/reference/subsystem-specify-subsystem?view=msvc-160).
It is still possible to use the `-printtoconsole` option for debug builds.
ACKs for top commit:
sipsorcery:
tACK 9edd713c18.
Tree-SHA512: 02f2874b13e484f98344f6a7e3b01fa82a78a39865787c77bd674ead22a84a7f98a1849ccad26bd2b8c8603b3e29dcc1633b0ad731ce7d61be2d6b1f9584839c
5c7ee1b2da libsecp256k1 no longer has --with-bignum= configure option (Pieter Wuille)
bdca9bcb6c Squashed 'src/secp256k1/' changes from 3967d96bf1..efad3506a8 (Pieter Wuille)
cabb566123 Disable certain false positive warnings for libsecp256k1 msvc build (Pieter Wuille)
Pull request description:
This updates our src/secp256k1 subtree to the latest upstream master. The changes include:
* The introduction of safegcd-based modular inverses, reducing ECDSA signing time by 25%-30% and ECDSA verification time by 15%-17%.
* [Original paper](https://gcd.cr.yp.to/papers.html) by Daniel J. Bernstein and Bo-Yin Yang
* [Implementation](https://github.com/bitcoin-core/secp256k1/pull/767) by Peter Dettman; [final](https://github.com/bitcoin-core/secp256k1/pull/831) version
* [Explanation](https://github.com/bitcoin-core/secp256k1/blob/master/doc/safegcd_implementation.md) of the algorithm using Python snippets
* [Analysis](https://github.com/sipa/safegcd-bounds) of the maximum number of iterations the algorithm needs
* [Formal proof in Coq](https://medium.com/blockstream/a-formal-proof-of-safegcd-bounds-695e1735a348) by Russell O'Connor, for a high-level equivalent algorithm
* Removal of libgmp as an (optional) dependency (which wasn't used in the Bitcoin Core build)
* CI changes (Travis -> Cirrus)
* Build system improvements
ACKs for top commit:
laanwj:
Tested ACK 5c7ee1b2da
Tree-SHA512: ad8ac3746264d279556a4aa7efdde3733e114fdba8856dd53218588521f04d83950366f5c1ea8fd56329b4c7fe08eedf8e206f8f26dbe3f0f81852e138655431
fac30eec42 refactor: Replace &foo[0] with foo.data() (MarcoFalke)
faece47c47 refactor: Avoid &foo[0] on C-Style arrays (MarcoFalke)
face961109 refactor: Use only one temporary buffer in CreateObfuscateKey (MarcoFalke)
fa05dddc42 refactor: Use CPubKey vector constructor where possible (MarcoFalke)
fabb6dfe6e script: Replace address-of idiom with vector data() method (Guido Vranken)
Pull request description:
The main theme of this refactor is to replace `&foo[0]` with `foo.data()`.
The first commit is taken from #21781 with the rationale:
* In CSignatureCache::ComputeEntryECDSA, change the way a vector pointer is resolved to prevent invoking undefined behavior if the vector is empty.
The other commits aim to remove all `&foo[0]`, where `foo` is any kind of byte representation. The rationale:
* Sometimes alternative code without any raw data pointers is easier to read (refer to the respective commit message for details)
* If the raw data pointer is needed, `foo.data()` should be preferred, as pointed out in the developer notes. This addresses the instances that have been missed in commit 592404f03f, and https://github.com/bitcoin/bitcoin/pull/9804
ACKs for top commit:
laanwj:
Code review ACK fac30eec42
practicalswift:
cr ACK fac30eec42c486ec1bfd696293040a7aa0f04625: patch looks correct
promag:
Code review ACK fac30eec42.
Tree-SHA512: e7e73146edbc78911a8e8c728b0a1c6b0ed9a88a008e650aa5dbffe72425bd42c76df70199a9cf7e02637448d7593e0eac52fd0f91f59240283e1390ee21bfa5
46b025e00d test: add new python linter to check file names and permissions (windsok)
6f6bb3ebc7 test: fix file permissions on various scripts (windsok)
Pull request description:
Adds a new python linter test which tests for correct filenames and file permissions in the repository.
Replaces the existing tests in the `test/lint/lint-filenames.sh` and `test/lint/lint-shebang.sh` linter tests, as well as adding some new and increased testing. This increased coverage is intended to catch issues such as in #21728 and https://github.com/bitcoin/bitcoin/pull/16807/files#r345547050
Summary of tests:
* Checks every file in the repository against an allowed regexp to make sure only lowercase or uppercase alphanumerics (a-zA-Z0-9), underscores (_), hyphens (-), at (@) and dots (.) are used in repository filenames.
* Checks only source files (*.cpp, *.h, *.py, *.sh) against a stricter allowed regexp to make sure only lowercase alphanumerics (a-z0-9), underscores (_), hyphens (-) and dots (.) are used in source code filenames. Additionally there is an exception regexp for directories or files which are excepted from matching this regexp (This should replicate the existing `test/lint/lint-filenames.sh` test)
* Checks all files in the repository match an allowed executable or non-executable file permission octal. Additionally checks that for executable files, the file contains a shebang line.
* Checks that for executable `.py` and `.sh` files, the shebang line used matches an allowable list of shebangs (This should replicate the existing `test/lint/lint-shebang.sh` test)
* Checks every file that contains a shebang line to ensure it has an executable permission
Additionally updates the permissions on various files to comply with the new tests.
Fixes#21729
ACKs for top commit:
practicalswift:
cr re-ACK 46b025e00df40724175735eb5606ac73067cb3b8: patch still looks correct
kiminuo:
code review ACK 46b025e00d if `contrib/gitian-descriptors/assign_DISTNAME` permission change is deemed OK.
laanwj:
Code review ACK 46b025e00d
Tree-SHA512: 1c8201a2cee0d9cbce15652b68cec9a6458a8b493fcd5392f98560aca0b1a12e668baab65a47100f116f626dadc3f591deb47f7368468c6a46c6c712c2533455
This PR was motivated by a comment in GUI PR (257) regarding a suggested improvement not being supported by VS2017.
When checking whether master can still be built with the VS2017 toolset ABI issues were encountered. Most likely due to the pre-compiled Qt binaries that are used.
It does not seem worth the effort to try and support VS2017, which would most likely require additional Qt binaries, or lengthy instructions on how to build static Qt binaries on Windows (which is very error prone and tedious).
Added advisory note about build not working with earlier Visual Studio versions.
Fixed grammar.
9bd3f35003 build: adds switch for disabling random base addresses in MSVC (Ethan Heilman)
Pull request description:
In m4 builds we have the --disable-hardening switch that can be given in `./configure` to turn off randomized addresses. This PR provides a simple way of turning off randomized addresses in MSVC builds.
This PR:
* Adds this option the common-init project file so that it can be globally set across the project
* Documents this switch in msvc build readme
I have run the following test to verify this works
I ran the msvc build with `<RandomizedBaseAddress>true</RandomizedBaseAddress>` then checked `bitcoind.exe` and `bitcoin-cli.exe` with `dumpbin.exe`:
bitcoind
```
> .\dumpbin.exe /headers src/bitcoind.exe
Microsoft (R) COFF/PE Dumper Version 14.16.27045.0
...
OPTIONAL HEADER VALUES
20B magic # (PE32+)
14.28 linker version
AE4600 size of code
345C00 size of initialized data
0 size of uninitialized data
6BED74 entry point (00000001406BED74) mainCRTStartup
1000 base of code
140000000 image base (0000000140000000 to 0000000140E2DFFF)
1000 section alignment
200 file alignment
6.00 operating system version
0.00 image version
6.00 subsystem version
0 Win32 version
E2E000 size of image
400 size of headers
0 checksum
3 subsystem (Windows CUI)
8160 DLL characteristics
High Entropy Virtual Addresses
Dynamic base
NX compatible
Terminal Server Aware
```
bitcoin-cli
```
> .\dumpbin.exe /headers src/bitcoin-cli.exe
Microsoft (R) COFF/PE Dumper Version 14.16.27045.0
...
OPTIONAL HEADER VALUES
20B magic # (PE32+)
14.28 linker version
1E3E00 size of code
92C00 size of initialized data
0 size of uninitialized data
104384 entry point (0000000140104384) mainCRTStartup
1000 base of code
140000000 image base (0000000140000000 to 0000000140279FFF)
1000 section alignment
200 file alignment
6.00 operating system version
0.00 image version
6.00 subsystem version
0 Win32 version
27A000 size of image
400 size of headers
0 checksum
3 subsystem (Windows CUI)
8160 DLL characteristics
High Entropy Virtual Addresses
Dynamic base
NX compatible
Terminal Server Aware
```
Then I built with `<RandomizedBaseAddress>false</RandomizedBaseAddress>` then checked `bitcoind.exe` and `bitcoin-cli.exe` with `dumpbin.exe` and observed that `Dynamic base` was longer listed in `OPTIONAL HEADER VALUES`
bitcoind
```
PS C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.16.27023\bin\HostX64\x64> .\dumpbin.exe /headers C:\Users\e0\Documents\GitHub\bitcoin021noaslr/src/bitcoind.exe
Microsoft (R) COFF/PE Dumper Version 14.16.27045.0
...
OPTIONAL HEADER VALUES
20B magic # (PE32+)
14.28 linker version
AE4600 size of code
33FE00 size of initialized data
0 size of uninitialized data
6BED74 entry point (00000001406BED74) mainCRTStartup
1000 base of code
140000000 image base (0000000140000000 to 0000000140E27FFF)
1000 section alignment
200 file alignment
6.00 operating system version
0.00 image version
6.00 subsystem version
0 Win32 version
E28000 size of image
400 size of headers
0 checksum
3 subsystem (Windows CUI)
8120 DLL characteristics
High Entropy Virtual Addresses
NX compatible
Terminal Server Aware
```
bitcoin-cli
```
> .\dumpbin.exe /headers src/bitcoin-cli.exe
Microsoft (R) COFF/PE Dumper Version 14.16.27045.0
...
OPTIONAL HEADER VALUES
20B magic # (PE32+)
14.28 linker version
1E3E00 size of code
90C00 size of initialized data
0 size of uninitialized data
104384 entry point (0000000140104384) mainCRTStartup
1000 base of code
140000000 image base (0000000140000000 to 0000000140277FFF)
1000 section alignment
200 file alignment
6.00 operating system version
0.00 image version
6.00 subsystem version
0 Win32 version
278000 size of image
400 size of headers
0 checksum
3 subsystem (Windows CUI)
8120 DLL characteristics
High Entropy Virtual Addresses
NX compatible
Terminal Server Aware
```
ACKs for top commit:
sipsorcery:
ACK 9bd3f35003.
practicalswift:
cr ACK 9bd3f35003c2e9eff74766d57a71d6b391ab602a: patch looks correct
Tree-SHA512: ddffdb4ff8a09c7cfef61c07a5db2a2828e9e3aa795ad8e5a1bf51ab489a68b40f87f6694518c5e0b8858c0fad4f93bb947b052e6b9d5e55eb38e764b746fc02
`vcpkg integrate install` must be executed so that msbuild will automatically install external dependencies.
It was removed in 712f95d332
It was originally added in 7644567758
7c543bc27b build: Enable safe warnings for msvc builds (Hennadii Stepanov)
Pull request description:
Disabling of some specific warnings no longer needed.
ACKs for top commit:
MarcoFalke:
ACK 7c543bc27b if green
practicalswift:
ACK 7c543bc27b if green: compiler warnings are good, compiler safety warnings are great
sipsorcery:
tACK 7c543bc27b.
Tree-SHA512: a8253a5a0ccc22b8622ae66efae77a163c54bbd6dbd7e28b043ad4eb474b2c8e5c7b22d3870b15a2b9b91bade0a5ae5bfc5ee765d132dc32ddbbe690fb0146d9
These were added as part of #9366 to fix issues with Protobuf.
Now that we no-longer use Protobuf, there's no reason to maintain a
duplicate set of byteswap tests for qt.
This adds a `-daemonwait` flag that does the same as `-daemon` except
it, from a user perspective, backgrounds the process only after
initialization is complete.
This can be useful when the process launching bitcoind wants to
guarantee that either the RPC server is running, or that initialization
failed, before continuing. The exit code indicates the initialization
result.
This replaces the use of the libc function `daemon()` by a custom
implementation which is inspired by the glibc implementation, but also
creates a pipe from the child to the parent process for communication.
An additional advantage of having our own `daemon()` implementation is
that no MACOS-specific pragmas are needed anymore to silence a
deprecation warning.
f75e0c1edd doc: add external-signer.md (Sjors Provoost)
d4b0107d68 rpc: send: support external signer (Sjors Provoost)
245b4457cf rpc: signerdisplayaddress (Sjors Provoost)
7ebc7c0215 wallet: ExternalSigner: add GetDescriptors method (Sjors Provoost)
fc5da520f5 wallet: add GetExternalSigner() (Sjors Provoost)
259f52cc33 test: external_signer wallet flag is immutable (Sjors Provoost)
2655197e1c rpc: add external_signer option to createwallet (Sjors Provoost)
2700f09c41 rpc: signer: add enumeratesigners to list external signers (Sjors Provoost)
07b7c940a7 rpc: add external signer RPC files (Sjors Provoost)
8ce7767071 wallet: add ExternalSignerScriptPubKeyMan (Sjors Provoost)
157ea7c614 wallet: add external_signer flag (Sjors Provoost)
f3e6ce78fb test: add external signer test (Sjors Provoost)
8cf543f96d wallet: add -signer argument for external signer command (Sjors Provoost)
f7eb7ecc67 test: framework: add skip_if_no_external_signer (Sjors Provoost)
87a97941f6 configure: add --enable-external-signer (Sjors Provoost)
Pull request description:
Big picture overview in [this gist](https://gist.github.com/Sjors/29d06728c685e6182828c1ce9b74483d).
This PR lets `bitcoind` call an arbitrary command `-signer=<cmd>`, e.g. a hardware wallet driver, where it can fetch public keys, ask to display an address, and sign a transaction (using PSBT under the hood).
It's design to work with https://github.com/bitcoin-core/HWI, which supports multiple hardware wallets. Any command with the same arguments and return values will work. It simplifies the manual procedure described [here](https://github.com/bitcoin-core/HWI/blob/master/docs/bitcoin-core-usage.md).
Usage is documented in [doc/external-signer.md](
https://github.com/Sjors/bitcoin/blob/2019/08/hww-box2/doc/external-signer.md), which also describes what protocol a different signer binary should conform to.
Use `--enable-external-signer` to opt in, requires Boost::Process:
```
Options used to compile and link:
with wallet = yes
with gui / qt = no
external signer = yes
```
It adds the following RPC methods:
* `enumeratesigners`: asks <cmd> for a list of signers (e.g. devices) and their master key fingerprint
* `signerdisplayaddress <address>`: asks <cmd> to display an address
It enhances the following RPC methods:
* `createwallet`: takes an additional `external_signer` argument and fetches keys from device
* `send`: automatically sends transaction to device and waits
Usage TL&DR:
* clone HWI repo somewhere and launch `bitcoind -signer=../HWI/hwi.py`
* check if you can see your hardware device: `bitcoin-cli enumeratesigners`
* create wallet and auto import keys `bitcoin-cli createwallet "hww" true true "" true true true`
* display address on device: `bitcoin-cli signerdisplayaddress ...`
* to spend, use `send` RPC and approve transaction on device
Prerequisites:
- [x] #21127 load wallet flags before everything else
- [x] #21182 remove mostly pointless BOOST_PROCESS macro
Potentially useful followups:
- GUI support: bitcoin-core/gui#4
- bumpfee support
- (automatically) verify (a subset of) keys on the device after import, through message signing
ACKs for top commit:
laanwj:
re-ACK f75e0c1edd
Tree-SHA512: 7db8afd54762295c1424c3f01d8c587ec256a72f34bd5256e04b21832dabd5dc212be8ab975ae3b67de75259fd569a561491945750492f417111dc7b6641e77f
This option replaces --with-boost-process
This prepares external signer support to be disabled by default.
It adds a configure option to enable this feature and to check
if Boost::Process is present.
This also exposes ENABLE_EXTERNAL_SIGNER to the test suite via test/config.ini
Tidied up debug and release configuration blocks in common project file to avoid duplication.
Updated appveyor config to use latest Visual Studio 2019 image.
Changed appveyor config file hash to use a new version of Qt pre-compiled binaries built for Visual Studio 2019 v16.8.1.
Bumped vcpkg version to tag '2020.11-1' for binary caching feature.
See #20392 for related discussion.
d52f502b1e Fix mock SQLiteDatabases (Andrew Chow)
99309ab3e9 Allow disabling BDB in configure with --without-bdb (Andrew Chow)
ee47f11f73 GUI: Force descriptor wallets when BDB is not compiled (Andrew Chow)
71e40b33bd RPC: Require descriptors=True for createwallet when BDB is not compiled (Andrew Chow)
6ebc41bf9c Enforce salvage is only for BDB wallets (Andrew Chow)
a58b719cf7 Do not compile BDB things when USE_BDB is defined (Andrew Chow)
b33af48210 Include wallet/bdb.h where it is actually being used (Andrew Chow)
Pull request description:
Adds a `--without-bdb` option to `configure` which disables the compilation of the BDB stuff. Legacy wallets will not be created when BDB is not compiled. A legacy-sqlite wallet can be loaded, but we will not create them.
Based on #20156 to resolve the situation where both `--without-sqlite` and `--without-bdb` are provided. In that case, the wallet is disabled and `--disable-wallet` is effectively set.
ACKs for top commit:
laanwj:
Code review ACK d52f502b1e
Tree-SHA512: 5a92ba7a542acc2e27003e9d4e5940e0d02d5c1f110db06cdcab831372bfd83e8d89c269caff31dd5bff062c1cf5f04683becff12bd23a33be731676f346553d
Removes the leading 0 from the version number. The minor version, which
we had been using as the major version, is now the major version. The
revision, which we had been using as the minor version, is now the minor
version. The revision number is dropped. The build number is promoted to
being part of the version number. This also avoids issues where it was
accidentally not included in the version number.
The CLIENT_VERSION remains the same format as previous as previously,
the Major version was 0 so that was never a factor in CLIENT_VERSION.
0e2a5e448f tests: dumping and minimizing of script assets data (Pieter Wuille)
4567ba034c tests: add generic qa-asset-based script verification unit test (Pieter Wuille)
f06e6d0345 tests: functional tests for Schnorr/Taproot/Tapscript (Pieter Wuille)
3c226639eb tests: add BIP340 Schnorr signature support to test framework (Pieter Wuille)
206fb180ec --- [TAPROOT] Tests --- (Pieter Wuille)
d7ff237f29 Activate Taproot/Tapscript on regtest (BIP 341, BIP 342) (Pieter Wuille)
e9a021d7e6 Make Taproot spends standard + policy limits (Pieter Wuille)
865d2c37e2 --- [TAPROOT] Regtest activation and policy --- (Pieter Wuille)
72422ce396 Implement Tapscript script validation rules (BIP 342) (Johnson Lau)
330de894a9 Use ScriptExecutionData to pass through annex hash (Pieter Wuille)
8bbed4b7ac Implement Taproot validation (BIP 341) (Pieter Wuille)
0664f5fe1f Support for Schnorr signatures and integration in SignatureCheckers (BIP 340) (Pieter Wuille)
5de246ca81 Implement Taproot signature hashing (BIP 341) (Johnson Lau)
9eb590894f Add TaggedHash function (BIP 340) (Pieter Wuille)
450d2b2371 --- [TAPROOT] BIP340/341/342 consensus rules --- (Pieter Wuille)
5d62e3a68b refactor: keep spent outputs in PrecomputedTransactionData (Pieter Wuille)
8bd2b4e784 refactor: rename scriptPubKey in VerifyWitnessProgram to exec_script (Pieter Wuille)
107b57df9f scripted-diff: put ECDSA in name of signature functions (Pieter Wuille)
f8c099e220 --- [TAPROOT] Refactors --- (Pieter Wuille)
Pull request description:
This is an implementation of the Schnorr/taproot consensus rules proposed by BIPs [340](https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki), [341](https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki), and [342](https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki).
See the list of commits [below](https://github.com/bitcoin/bitcoin/pull/19953#issuecomment-691815830). No signing or wallet support of any kind is included, as testing is done entirely through the Python test framework.
This is a successor to https://github.com/bitcoin/bitcoin/pull/17977 (see discussion following [this comment](https://github.com/bitcoin/bitcoin/pull/17977#issuecomment-682285983)), and will have further changes squashed/rebased. The history of this PR can be found in #19997.
ACKs for top commit:
instagibbs:
reACK 0e2a5e448f
benthecarman:
reACK 0e2a5e4
kallewoof:
reACK 0e2a5e448f
jonasnick:
ACK 0e2a5e448f almost only looked at bip340/libsecp related code
jonatack:
ACK 0e2a5e448f modulo the last four commits (tests) that I plan to finish reviewing tomorrow
fjahr:
reACK 0e2a5e448f
achow101:
ACK 0e2a5e448f
Tree-SHA512: 1b00314450a2938a22bccbb4e177230cf08bd365d72055f9d526891f334b364c997e260c10bc19ca78440b6767712c9feea7faad9a1045dd51a5b96f7ca8146e
This enables the schnorrsig module in libsecp256k1, adds the relevant types
and functions to src/pubkey, as well as in higher-level `SignatureChecker`
classes. The (verification side of the) BIP340 test vectors is also added.
The vcpkg tool has introduced a proper way to use manifests, https://devblogs.microsoft.com/cppblog/vcpkg-accelerate-your-team-development-environment-with-binary-caching-and-manifests/. This PR replaces the custom text file mechanism with the new manifest approach.
It is planned that vckpg manifests will include the ability to version dependencies in the future. Dependency versions would solve a number of issues that currently require workarounds with the appveyor CI.
Set vcpkg manifest version to 1 to avoid any perception it's related to any release or other version numbering.
The way we currently test with AC_CHECK_DECLS do not work with Clang:
```bash
configure:21492: clang++-10 -std=c++11 -c -g -O2 -DHAVE_BUILD_INFO -D__STDC_FORMAT_MACROS conftest.cpp >&5
conftest.cpp💯10: error: builtin functions must be directly called
(void) __builtin_clz;
^
1 error generated.
```
This also removes the __builtin_clz() check, as we don't actually use
it anywhere, and it's trvial to re-add detection if we do start using
it at some point.
Add a "PSBT Operations" dialog, reached from the "Load PSBT..." menu
item, giving options to sign or broadcast the loaded PSBT as
appropriate, as well as copying the result to the clipboard or saving
it to a file.
MarcoFalke
