Merge #16367: Multiprocess build support
pull/764/headb919efadffdepends: Use default macos clang compiler (Russell Yanofsky)d54f64c6c7Add multiprocess travis configuration (Russell Yanofsky)787f40668dSet LD_LIBRARY_PATH consistently in travis tests (Russell Yanofsky)d630646662libmultiprocess depends build (Russell Yanofsky)e6e44eedd5Multiprocess build changes (Russell Yanofsky) Pull request description: This PR is part of the [process separation project](https://github.com/bitcoin/bitcoin/projects/10). This splits autotools, depends build, and travis changes out of #10102, so code changes and build system changes can be reviewed separately. ACKs for top commit: hebasto: re-ACKb919efadff, rebased only since my [previous](https://github.com/bitcoin/bitcoin/pull/16367#issuecomment-605514556) review. Tree-SHA512: ebc5e403cc99a0d9629ed7fe1595e01d57e6d1255cbf03968a3196ff6f528f734c78060fdc065724ee1f923bcc5aa2b29470fcb36a7f15957eb57c76d58178a4
MarcoFalke
5 years ago
commit
1b30761360
@ -0,0 +1,14 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# Copyright (c) 2020 The Bitcoin Core developers
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
|
||||
export LC_ALL=C.UTF-8
|
||||
|
||||
export CONTAINER_NAME=ci_native_multiprocess
|
||||
export PACKAGES="cmake python3"
|
||||
export DEP_OPTS="MULTIPROCESS=1"
|
||||
export GOAL="install"
|
||||
export BITCOIN_CONFIG=""
|
||||
export TEST_RUNNER_ENV="BITCOIND=bitcoin-node"
|
||||
@ -0,0 +1,17 @@
|
||||
package=capnp
|
||||
$(package)_version=$(native_$(package)_version)
|
||||
$(package)_download_path=$(native_$(package)_download_path)
|
||||
$(package)_file_name=$(native_$(package)_file_name)
|
||||
$(package)_sha256_hash=$(native_$(package)_sha256_hash)
|
||||
|
||||
define $(package)_config_cmds
|
||||
$($(package)_autoconf)
|
||||
endef
|
||||
|
||||
define $(package)_build_cmds
|
||||
$(MAKE)
|
||||
endef
|
||||
|
||||
define $(package)_stage_cmds
|
||||
$(MAKE) DESTDIR=$($(package)_staging_dir) install
|
||||
endef
|
||||
@ -0,0 +1,18 @@
|
||||
package=libmultiprocess
|
||||
$(package)_version=$(native_$(package)_version)
|
||||
$(package)_download_path=$(native_$(package)_download_path)
|
||||
$(package)_file_name=$(native_$(package)_file_name)
|
||||
$(package)_sha256_hash=$(native_$(package)_sha256_hash)
|
||||
$(package)_dependencies=native_$(package) boost capnp
|
||||
|
||||
define $(package)_config_cmds
|
||||
cmake -DCMAKE_INSTALL_PREFIX=$($($(package)_type)_prefix)
|
||||
endef
|
||||
|
||||
define $(package)_build_cmds
|
||||
$(MAKE)
|
||||
endef
|
||||
|
||||
define $(package)_stage_cmds
|
||||
$(MAKE) DESTDIR=$($(package)_staging_dir) install
|
||||
endef
|
||||
@ -0,0 +1,13 @@
|
||||
package=native_boost
|
||||
$(package)_version=$(boost_version)
|
||||
$(package)_download_path=$(boost_download_path)
|
||||
$(package)_file_name=$(boost_file_name)
|
||||
$(package)_sha256_hash=$(boost_sha256_hash)
|
||||
|
||||
define $(package)_config_cmds
|
||||
./bootstrap.sh --prefix=$($($(package)_type)_prefix) --with-libraries=headers
|
||||
endef
|
||||
|
||||
define $(package)_stage_cmds
|
||||
./b2 -d0 --prefix=$($(package)_staging_prefix_dir) install
|
||||
endef
|
||||
@ -0,0 +1,18 @@
|
||||
package=native_capnp
|
||||
$(package)_version=0.7.0
|
||||
$(package)_download_path=https://capnproto.org/
|
||||
$(package)_download_file=capnproto-c++-$($(package)_version).tar.gz
|
||||
$(package)_file_name=capnproto-cxx-$($(package)_version).tar.gz
|
||||
$(package)_sha256_hash=c9a4c0bd88123064d483ab46ecee777f14d933359e23bff6fb4f4dbd28b4cd41
|
||||
|
||||
define $(package)_config_cmds
|
||||
$($(package)_autoconf)
|
||||
endef
|
||||
|
||||
define $(package)_build_cmds
|
||||
$(MAKE)
|
||||
endef
|
||||
|
||||
define $(package)_stage_cmds
|
||||
$(MAKE) DESTDIR=$($(package)_staging_dir) install
|
||||
endef
|
||||
@ -0,0 +1,18 @@
|
||||
package=native_libmultiprocess
|
||||
$(package)_version=1d630f536d43d1fdba028f2cfccf49bc27da92db
|
||||
$(package)_download_path=https://github.com/chaincodelabs/libmultiprocess/archive
|
||||
$(package)_file_name=$($(package)_version).tar.gz
|
||||
$(package)_sha256_hash=f43183ccf59fa7596d113e56518563673f5823a721974b4058d8778ef53bd476
|
||||
$(package)_dependencies=native_boost native_capnp
|
||||
|
||||
define $(package)_config_cmds
|
||||
cmake -DCMAKE_INSTALL_PREFIX=$($($(package)_type)_prefix)
|
||||
endef
|
||||
|
||||
define $(package)_build_cmds
|
||||
$(MAKE)
|
||||
endef
|
||||
|
||||
define $(package)_stage_cmds
|
||||
$(MAKE) DESTDIR=$($(package)_staging_dir) install
|
||||
endef
|
||||
@ -0,0 +1,34 @@
|
||||
# Multiprocess Bitcoin
|
||||
|
||||
On unix systems, the `--enable-multiprocess` build option can be passed to `./configure` to build new `bitcoin-node`, `bitcoin-wallet`, and `bitcoin-gui` executables alongside existing `bitcoind` and `bitcoin-qt` executables.
|
||||
|
||||
`bitcoin-node` is a drop-in replacement for `bitcoind`, and `bitcoin-gui` is a drop-in replacement for `bitcoin-qt`, and there are no differences in use or external behavior between the new and old executables. But internally (after [#10102](https://github.com/bitcoin/bitcoin/pull/10102)), `bitcoin-gui` will spawn a `bitcoin-node` process to run P2P and RPC code, communicating with it across a socket pair, and `bitcoin-node` will spawn `bitcoin-wallet` to run wallet code, also communicating over a socket pair. This will let node, wallet, and GUI code run in separate address spaces for better isolation, and allow future improvements like being able to start and stop components independently on different machines and environments.
|
||||
|
||||
## Next steps
|
||||
|
||||
Specific next steps after [#10102](https://github.com/bitcoin/bitcoin/pull/10102) will be:
|
||||
|
||||
- [ ] Adding `-ipcbind` and `-ipcconnect` options to `bitcoin-node`, `bitcoin-wallet`, and `bitcoin-gui` executables so they can listen and connect to TCP ports and unix socket paths. This will allow separate processes to be started and stopped any time and connect to each other.
|
||||
- [ ] Adding `-server` and `-rpcbind` options to the `bitcoin-wallet` executable so wallet processes can handle RPC requests directly without going through the node.
|
||||
- [ ] Supporting windows, not just unix systems. The existing socket code is already cross-platform, so the only windows-specific code that needs to be written is code spawning a process and passing a socket descriptor. This can be implemented with `CreateProcess` and `WSADuplicateSocket`. Example: https://memset.wordpress.com/2010/10/13/win32-api-passing-socket-with-ipc-method/.
|
||||
- [ ] Adding sandbox features, restricting subprocess access to resources and data. See [https://eklitzke.org/multiprocess-bitcoin](https://eklitzke.org/multiprocess-bitcoin).
|
||||
|
||||
## Debugging
|
||||
|
||||
After [#10102](https://github.com/bitcoin/bitcoin/pull/10102), the `-debug=ipc` command line option can be used to see requests and responses between processes.
|
||||
|
||||
## Installation
|
||||
|
||||
The multiprocess feature requires [Cap'n Proto](https://capnproto.org/) and [libmultiprocess](https://github.com/chaincodelabs/libmultiprocess) as dependencies. A simple way to get starting using it without installing these dependencies manually is to use the [depends system](../depends) with the `MULTIPROCESS=1` [dependency option](../depends#dependency-options) passed to make:
|
||||
|
||||
```
|
||||
cd <BITCOIN_SOURCE_DIRECTORY>
|
||||
make -C depends NO_QT=1 MULTIPROCESS=1
|
||||
./configure --prefix=$PWD/depends/x86_64-pc-linux-gnu
|
||||
make
|
||||
src/bitcoin-node -regtest -printtoconsole -debug=ipc
|
||||
```
|
||||
|
||||
The configure script will pick up settings and library locations from the depends directory, so there is no need to pass `--enable-multiprocess` as a separate flag when using the depends system (it's controlled by the `MULTIPROCESS=1` option).
|
||||
|
||||
Alternately, you can install [Cap'n Proto](https://capnproto.org/) and [libmultiprocess](https://github.com/chaincodelabs/libmultiprocess) packages on your system, and just run `./configure --enable-multiprocess` without using the depends system. The configure script will be able to locate the installed packages via [pkg-config](https://www.freedesktop.org/wiki/Software/pkg-config/). See [Installation](https://github.com/chaincodelabs/libmultiprocess#installation) section of the libmultiprocess readme for install steps. See [build-unix.md](build-unix.md) and [build-osx.md](build-osx.md) for information about installing dependencies in general.
|
||||
Loading…
Reference in new issue