diff --git a/doc/mweb/light-clients.md b/doc/mweb/light-clients.md new file mode 100644 index 0000000000..ff12ecd54b --- /dev/null +++ b/doc/mweb/light-clients.md @@ -0,0 +1,21 @@ +# Light Clients + +Since addresses are not stored on MWEB outputs, identifying coins belonging to a wallet is no longer as simple as sharing a [bloom filter](https://github.com/bitcoin/bips/blob/master/bip-0037.mediawiki) with a full node, and letting them check for outputs that _probabilistically_ belong to the wallet. + +There are 2 ways that existing light clients identify coins belonging to a wallet: + +## 1. Server-Side Filtering + +[BIP37 - Bloom Filters](https://github.com/bitcoin/bips/blob/master/bip-0037.mediawiki): Light wallets build a bloom filter for a list of addresses belonging to them, and share it with a full node. That full node then does the work of checking for outputs that _probabilistically_ belong to the wallet. + +#### MWEB Equaivalent + +Since addresses are not stored on MWEB outputs, we cannot identify a wallet's transactions using server-side filtering through probabilistic data structures like bloom filters. The only server-side filtering currently supported for MWEB is filtering using knowledge of private view keys. Light wallets would share their private view keys with a full node, which would then churn through all outputs to find any belonging to the wallet. Unfortunately, this is not a probabilistic process, so full nodes will know with 100% certainty outputs belong to the wallet. To make matters worse, the node will also learn the _value_ of the wallet's outputs. This method is ideal for those who want to use MWEB from a mobile wallet connected to _their own_ full node on a desktop or server, but due to the weakened privacy, you should avoid using this method when connected to untrusted nodes. + +## 2. Client-Side Filtering + +[BIP157 - Client Side Block Filtering](https://github.com/bitcoin/bips/blob/master/bip-0157.mediawiki): The reverse, where full nodes build a compact filter for each block, and light wallets use those filters to identify blocks that may contain the wallet's transactions. + +#### MWEB Equivalent + +For those who don't have a trusted full node to connect to, we could use something similar to client side block filtering. More precisely, we could perform client side _output_ filtering. Though no method has yet been designed for building filters as compact as those defined in [BIP158 - Compact Block Filters for Light Clients](https://github.com/bitcoin/bips/blob/master/bip-0158.mediawiki), nodes can provide light clients with _compact_ MWEB outputs, allowing them to avoid downloading the large rangeproofs associated to each output. In fact, we can be even more data efficient, and download only outputs that are verifiably unspent. Following the process defined in [LIP-0006](https://github.com/DavidBurkett/lips/blob/LIP0006/LIP-0006.mediawiki), light clients can download and check all UTXOs from multiple nodes _in parallel_, avoiding the privacy and verifiability shortcomings of the server-side filtering approaches. \ No newline at end of file diff --git a/doc/mweb/pmmr.md b/doc/mweb/pmmr.md new file mode 100644 index 0000000000..f61630b8b3 --- /dev/null +++ b/doc/mweb/pmmr.md @@ -0,0 +1,175 @@ +# Prunable Merkle Mountain Ranges (PMMRs) + +Merkle Mountain Ranges (MMRs), an invention of Peter Todd for use in [Open Timestamps](https://github.com/opentimestamps/opentimestamps-server/blob/master/doc/merkle-mountain-range.md), are append-only binary trees that can be used to commit to a collection of elements. In addition to supporting efficient insertion, they also come in a prunable form,called Prunable Merkle Mountain Ranges (PMMRs), which support pruning of entire subtrees. They are used in Litecoin\'s MWEB to build TXO commitments. + +## Example 1 + +``` +Height 4: 30 + / \ + / \ + / \ + / \ + / \ + / \ + / \ + / \ + / \ + / \ + / \ + / \ + / \ + / \ + / \ + / \ + / \ + / \ + / \ +Height 3: 14 29 45 + / \ / \ / \ + / \ / \ / \ + / \ / \ / \ + / \ / \ / \ + / \ / \ / \ + / \ / \ / \ + / \ / \ / \ + / \ / \ / \ + / \ / \ / \ +Height 2: 06 13 21 28 37 44 52 + / \ / \ / \ / \ / \ / \ / \ + / \ / \ / \ / \ / \ / \ / \ + / \ / \ / \ / \ / \ / \ / \ + / \ / \ / \ / \ / \ / \ / \ +Height 1: 02 05 09 12 17 20 24 27 33 36 40 43 48 51 55 + / \ / \ / \ / \ / \ / \ / \ / \ / \ / \ / \ / \ / \ / \ / \ + / \ / \ / \ / \ / \ / \ / \ / \ / \ / \ / \ / \ / \ / \ / \ +Height 0: 00 01 03 04 07 08 10 11 15 16 18 19 22 23 25 26 31 32 34 35 38 39 41 42 46 47 49 50 53 54 56 + || || || || || || || || || || || || || || || || || || || || || || || || || || || || || || || +Leaves: 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 + +Peaks: 30, 45, 52, 55, 56 +``` + +## Glossary + +* Leaf - An item to commit to +* Leaf Index - 0-based insertion position of leaves, represented as a 64-bit unsigned integer +* Node - A leaf or the parent of leaves or nodes +* Position - 0-based index of nodes, represented as a 64-bit unsigned integer +* Size - The number of nodes in an unpruned MMR. In our example with 31 leaves (L0-L30), there are 57 nodes (N0-N56) +* Sibling - The other leaf or node that shares the same parent (e.g. N14 & N29) +* Prune - When a leaf is spent, it shall be marked as pruned. A parent node whose children are both pruned shall also be marked as pruned. +* Compact - To remove or "forget" a node +* Peak - A node with no parent +* HASH(x || y) - The blake3 hash of `x` concatenated with `y` + +## Insertion + +To append a new leaf `L` to an MMR, add a node at height 0 to the right of the last leaf. If the leaf to the left of it has no parent, add a parent node to the 2 leaves. If the node to the left of the newly-added parent node also has no parent, add a parent node for them. Repeat until the last node added does not have a parent-less sibling. + +Let\'s walk through an example. Start with an MMR with 3 leaves (L00, L01, and L02): + +``` + N02 + / \ + / \ +N00 N01 N03 + | | | +L00 L01 L02 +``` + +To add a 4th leaf (L03), append a node (N04) to right of the last leaf node (N03): + +``` + N02 + / \ + / \ +N00 N01 N03 N04 + | | | | +L00 L01 L02 L03 +``` + +The node to the left (N03) of our added node (N04) does not have a parent. So let\'s give them a parent node (N05). + +``` + N02 N05 + / \ / \ + / \ / \ +N00 N01 N03 N04 + | | | | +L00 L01 L02 L03 +``` + +The node to the left (N02) of our added parent node (N05) also does not have a parent. So let\'s add another parent node (N06). + +``` + N06 + / \ + / \ + / \ + / \ + N02 N05 + / \ / \ + / \ / \ +N00 N01 N03 N04 + | | | | +L00 L01 L02 L03 +``` + +There is no node to the left of node our added parent node (N06), so we are finished updating the MMR. + +## Hashing + +All nodes in an MMR are just hashes that commit to an item or a tree of items (in the case of parent nodes). + +To calculate the hash of a leaf `L` containing data `D`: + +``` +Node(L) = HASH(L.position || D) +``` + +For leaf `L03` with data `0x0102030405`, the hash would be: + +``` +HASH(uint64_t(4) || 0x0102030405) == blake3(0x00000000000000040102030405) +``` + +To calculate the hash for a parent node `P` with left child `C_L` and right child `C_R`: + +``` +Node(P) = HASH(P.position || Node(C_L) || Node(C_R)) +``` + +For parent node `N05` of left child `N03` with data `0x0102030405` and right child `N04` with data `0xaabbccdd`, the hash would be: + +``` +HASH(uint64_t(05) || N03 || N04) == blake3(0x0000000000000005 || blake3(0x00000000000000030102030405) || blake3(0x0000000000000004aabbccdd)) +``` + +## PMMR Root + +Similar to how a single hash, the merkle root, can commit to a merkle tree, a single hash, the PMMR root, can be calculated that commits to an entire PMMR. Since there's not a single peak for MMRs the way there is with merkle trees, calculating the root is a little more complicated. + +To calculate the root hash, we start with the rightmost (lowest) peak. We\'ll call this `root_hash`. Then, we work up the mountain range until we reach the highest peak, each step calculating `root_hash = HASH(mmr_size || higher_peak_hash || root_hash)`, where `||` is just the concatenate symbol. This process is called "bagging the peaks." + +In *example 1*, we have 5 disparate trees, and therefore 5 different peaks. So we start by calculating the `mmr_size`, which in our case would be 57 (N0-N56). Then, we assign `root_hash` as the lowest peak (56), and then bag peak 55, then 52, then 45, and finally peak 30: + +``` +root_hash = N56.hash; +root_hash = HASH(57 || N55.hash || root_hash); +root_hash = HASH(57 || N52.hash || root_hash); +root_hash = HASH(57 || N45.hash || root_hash); +root_hash = HASH(57 || N30.hash || root_hash); +``` + +Without temporary variables, this would be: +``` +root_hash = HASH(57 || N30.hash || HASH(57 || N45.hash || HASH(57 || N52.hash || HASH(57 || N55.hash || N56.hash)))); +``` + +## TODO + +1. Describe difference between pruning and compacting +2. Document PruneList +3. Explain merkle proofs +4. Document `Segment`s and how they\'re built & verified \ No newline at end of file diff --git a/doc/mweb/stealth-addresses.md b/doc/mweb/stealth-addresses.md new file mode 100644 index 0000000000..291ced1af2 --- /dev/null +++ b/doc/mweb/stealth-addresses.md @@ -0,0 +1,81 @@ +# Stealth Addresses + +Transacting on the MWEB happens via stealth addresses. Stealth addresses are supported using the *Dual-Key Stealth Address Protocol (DKSAP)*. These addresses consist of 2 secp256k1 public keys (**Ai**, **Bi**) where **Ai** is the scan pubkey used for identifying outputs, and **Bi** is the spend pubkey. + +### Notation +* **G** = Curve generator point +* Bold uppercase letter (**A**, **Ai**, etc.) represents a secp256k1 public key +* Bold lowercase letter (**a**, **ai**, etc.) represents a scalar +* A letter in single quotes ('K') represents the ASCII value of the character literal +* HASH32(x | y | z) represents the standard 32-byte `BLAKE3` hash of the conjoined serializations of `x`, `y`, and `z` +* HASH64(m) represents the `BLAKE3` hash of `m` using a 64-byte output digest +* SWITCH(**v**, **r**) represents the pedersen commitment generated using a blind switch with value **v** and key **r**. + +### Address Generation + +Unique stealth addresses should be generated deterministically from a wallet's seed using the following formula (using **G** = generator point): + +1. Generate master scan keypair (**a**, **A**) using HD keychain path `m/1/0/100'` +2. Generate master spend keypair (**b**, **B**) using HD keychain path `m/1/0/101'` +3. Choose the lowest unused address index **i** +4. Calculate one-time spend keypair (**bi**, **Bi**) as:
+ **bi** = **b** + HASH32(**A** | **i** | **a**)
+ **Bi** = **bi\*G** where **G** refers to the curve generator point +5. Calculate one-time scan keypair (**ai**, **Ai**) as:
+ **ai** = **a\*bi**
+ **Ai** = **ai\*G** where **G** refers to the curve generator point + +### Outputs + +Outputs consist of the following data: + +* **Co** - The pedersen commitment to the value. +* Output message, **m**, consisting of: + * **Ko** - The receiver's one-time public key. + * **Ke** - The key exchange public key. + * **Ks** - The sender's public key. + * **t** - The view tag. This is the first byte of the shared secret. + * **v'** - The masked value. + * **n'** - The masked nonce. +* A signature of the **m** using the sender's key **ks**. +* A rangeproof of **Co** that also commits to the **m**. + +### Output Construction + +To create an output for value **v** to a receiver's stealth address (**Ai**,**Bi**): + +1. Generate a random sender keypair (**ks**, **Ks**). # TODO: Can we generate this deterministically? +2. Derive the nonce **n** as the first 16 bytes of HASH32('N'|**ks**) +3. Derive the sending key **s** = HASH32('S'|**Ai**|**Bi**|**v**|**n**) +4. Derive the shared secret **e** = HASH32('D'|**s*Ai**). The view tag **t** is the first byte of **e**. +5. Calculate the receiver's one-time public key **Ko** = **Bi** + HASH32('O'|**e**)***G** +6. Calculate the key exchange pubkey **Ke** = **s*Bi** +7. Derive the 64-byte mask **m** = HASH64(**e**) +8. Encrypt the value using **v'** = **v** ⊕ **m[32,39]** +9. Encrypt the nonce using **n'** = **n** ⊕ **m[40,55]** +10. Calculate the commitment **Co** = SWITCH(**v**, **m[0,31]**). +11. Generate the rangeproof for **Co**, committing also to **m**. +12. Sign **m** using the sender key **ks**. + + +### Output Identification + +NOTE: the wallet must keep a map **Bi**->**i** of all used spend pubkeys and the next few unused ones. + +To check if an output belongs to a wallet: + +1. Calculate the ECDHE shared secret **e** = HASH32('D'|**a*Ke**) +2. If the first byte of **e** does not match the view tag **t**, the output does not belong to the wallet. +3. Calculate the one-time spend pubkey: **Bi** = **Ko** - HASH('O'|**e**)***G** +4. Lookup the index **i** that generates **Bi** from the wallet's map **Bi**->**i**. If not found, the output does not belong to the wallet. +5. Derive the 64-byte mask **m** = HASH64(**e**) +6. Decrypt the value **v** = **v'** ⊕ **m[32,39]** where **m[32,39]** refers to bytes 32-39 (0-based big-endian) of the 64 byte mask **m**. +7. Verify that SWITCH(**v**,**m[0-31]**) ?= **Co** +8. Decrypt the nonce **n** = **n'** ⊕ **m[40,55]**. +9. Calculate the send key **s** = HASH32('S'|**Ai**|**Bi**|**v**|**n**) +10. Verify that **Ke** ?= **s*Bi**. + +If all verifications succeed, the output belongs to the wallet, and is safe to use. + +The spend key can be recovered by **ko** = HASH32('O'|**e**) + **ai**. + diff --git a/doc/mweb/store.md b/doc/mweb/store.md new file mode 100644 index 0000000000..d5b9c80a46 --- /dev/null +++ b/doc/mweb/store.md @@ -0,0 +1,73 @@ +# Data Storage + +In addition to extending the existing `CBlock` and `CTransaction` objects already used in Litecoin, the following data stores are created or modified for MWEB. + +### CBlockUndo + +After a new block is connected, a `rev.dat` file is created that describes how to remove or "undo" the block from the chain state. +When performing a reorg, the undo file can be processed, which will revert the UTXO set back to the way it was just before the block. + +To support undo-ing MWEB blocks, we''ve added a new `mw::BlockUndo` object to `CBlockUndo`, which gets serialized at the end. +This contains the following fields: + +* `prev_header` - the MWEB header in the previous block +* `utxos_spent` - vector of `UTXO`s that were spent in the block +* `utxos_added` - vector of coin IDs (hashes) that were added in the block + +To revert the block from the chain state, the node adds back the `UTXO`s in `utxos_spent`, removes the matching `UTXO`s in `utxos_added`, and sets `prev_header` as the MWEB chain tip. + +NOTE: For backward compatibility, when deserializing a `CBlockUndo`, we first must look up the size of the `CBlockUndo` object. +After deserializing vtxundo (the vector of `CTxUndo`s), if more data remains, assume it\'s the `mw::BlockUndo` object. +If no more data remains, assume the `CBlockUndo` does not have MWEB data. +An `UnserializeBlockUndo` function was added to handle this. + +### UTXOs +##### CoinDB (leveldb) + +Litecoin's leveldb instance is used to maintain a UTXO table (prefix: 'U') with `UTXO` objects, consisting of the following data fields: + +* output_hash (key) - The hash of the output. +* block_height - The block height the UTXO was included. +* leaf_index - The index of the leaf in the output PMMR. +* output - The full `Output` object, including the rangeproof and owner data. + +### PMMRs +##### MMR Info (leveldb) +Litecoin's leveldb instance is used to maintain an MMR Info table (prefix: "M") with `MMRInfo` objects consisting of the following data fields: + +* version - A version byte that allows for future schema upgrades. +* index (key) - File number of the PMMR files. +* pruned_hash - Hash of latest header this PMMR represents. +* compact_index - File number of the PruneList bitset. +* compacted_hash - Hash of the header this MMR was compacted for. You cannot rewind beyond this point. + +Each time the PMMRs are flushed to disk, a new MMRInfo object is written to the DB and marked as the latest. + +##### Leaves (leveldb) +Litecoin's leveldb instance is used to maintain MMR leaf tables (prefix: 'O' for outputs) to store uncompacted PMMR leaves consisting of the following data fields: + +* leaf_index (key) - The zero-based leaf position. +* leaf - The raw leaf data committed to by the PMMR. + +Leaves spent before the horizon will be removed during compaction. + +##### MMR Hashes (file) + +Stored in file `.dat` where `` refers to 'O' for the output PMMR, and `` is a 6-digit number that matches the `index` value of the latest `MMRInfo` object. +Example: If the latest `MMRInfo` object has an `index` of 123, the matching output PMMR hash file will be named `O000123.dat`. + +The hash file consists of un-compacted leaf hashes and their parent hashes. + +##### Leafset (file) + +Stored in file `leaf.dat`. + +The leafset file consists of a bitset indicating which leaf indices of the output PMMR are unspent. +Example: If the PMMR contains 5 leaves where leaf indices 0, 1, and 2 are spent, but 3 and 4 are unspent, the file will contain a single byte of 00011000 = 0x18. + +##### PruneList (file) + +Stored in file `prun.dat`. + +The prunelist file consists of a bitset indicating which nodes of the output PMMR are not included in the output PMMR hash file. +Example: If nodes 0, 1, 3, and 4 are compacted (not included in PMMR), then the first byte of the prune list bitset will be 11011000 = 0xD8. \ No newline at end of file diff --git a/src/libmw/include/mw/consensus/Params.h b/src/libmw/include/mw/consensus/Params.h index 43728f89fd..9c36b8a48c 100644 --- a/src/libmw/include/mw/consensus/Params.h +++ b/src/libmw/include/mw/consensus/Params.h @@ -10,7 +10,7 @@ MW_NAMESPACE /// Consensus parameters /// Any change to these will cause a hardfork! /// -static constexpr size_t BYTES_PER_WEIGHT = 42; +static constexpr size_t BYTES_PER_WEIGHT = 42; // For any 'extra' data added to inputs, outputs, or kernels static constexpr size_t BASE_KERNEL_WEIGHT = 2; static constexpr size_t STEALTH_EXCESS_WEIGHT = 1; diff --git a/src/libmw/include/mw/mmr/LeafSet.h b/src/libmw/include/mw/mmr/LeafSet.h index e31c811f74..522090725a 100644 --- a/src/libmw/include/mw/mmr/LeafSet.h +++ b/src/libmw/include/mw/mmr/LeafSet.h @@ -11,84 +11,85 @@ class ILeafSet { public: - using Ptr = std::shared_ptr; + using Ptr = std::shared_ptr; - virtual ~ILeafSet() = default; + virtual ~ILeafSet() = default; - virtual uint8_t GetByte(const uint64_t byteIdx) const = 0; - virtual void SetByte(const uint64_t byteIdx, const uint8_t value) = 0; + virtual uint8_t GetByte(const uint64_t byteIdx) const = 0; + virtual void SetByte(const uint64_t byteIdx, const uint8_t value) = 0; - void Add(const mmr::LeafIndex& idx); - void Remove(const mmr::LeafIndex& idx); - bool Contains(const mmr::LeafIndex& idx) const noexcept; - mw::Hash Root() const; - void Rewind(const uint64_t numLeaves, const std::vector& leavesToAdd); - const mmr::LeafIndex& GetNextLeafIdx() const noexcept { return m_nextLeafIdx; } - BitSet ToBitSet() const; + void Add(const mmr::LeafIndex& idx); + void Remove(const mmr::LeafIndex& idx); + bool Contains(const mmr::LeafIndex& idx) const noexcept; + mw::Hash Root() const; + void Rewind(const uint64_t numLeaves, const std::vector& leavesToAdd); + const mmr::LeafIndex& GetNextLeafIdx() const noexcept { return m_nextLeafIdx; } + uint64_t GetNumNodes() const noexcept { return GetNextLeafIdx().GetPosition(); } + BitSet ToBitSet() const; - virtual void ApplyUpdates( - const uint32_t file_index, - const mmr::LeafIndex& nextLeafIdx, - const std::unordered_map& modifiedBytes - ) = 0; + virtual void ApplyUpdates( + const uint32_t file_index, + const mmr::LeafIndex& nextLeafIdx, + const std::unordered_map& modifiedBytes + ) = 0; protected: - uint8_t BitToByte(const uint8_t bit) const; + uint8_t BitToByte(const uint8_t bit) const; - ILeafSet(const mmr::LeafIndex& nextLeafIdx) - : m_nextLeafIdx(nextLeafIdx) { } + ILeafSet(const mmr::LeafIndex& nextLeafIdx) + : m_nextLeafIdx(nextLeafIdx) { } - mmr::LeafIndex m_nextLeafIdx; + mmr::LeafIndex m_nextLeafIdx; }; class LeafSet : public ILeafSet { public: - using Ptr = std::shared_ptr; + using Ptr = std::shared_ptr; - static LeafSet::Ptr Open(const FilePath& leafset_dir, const uint32_t file_index); - static FilePath GetPath(const FilePath& leafset_dir, const uint32_t file_index); + static LeafSet::Ptr Open(const FilePath& leafset_dir, const uint32_t file_index); + static FilePath GetPath(const FilePath& leafset_dir, const uint32_t file_index); - uint8_t GetByte(const uint64_t byteIdx) const final; - void SetByte(const uint64_t byteIdx, const uint8_t value) final; + uint8_t GetByte(const uint64_t byteIdx) const final; + void SetByte(const uint64_t byteIdx, const uint8_t value) final; - void ApplyUpdates( - const uint32_t file_index, - const mmr::LeafIndex& nextLeafIdx, - const std::unordered_map& modifiedBytes - ) final; - void Flush(const uint32_t file_index); - void Cleanup(const uint32_t current_file_index) const; + void ApplyUpdates( + const uint32_t file_index, + const mmr::LeafIndex& nextLeafIdx, + const std::unordered_map& modifiedBytes + ) final; + void Flush(const uint32_t file_index); + void Cleanup(const uint32_t current_file_index) const; private: - LeafSet(FilePath dir, MemMap&& mmap, const mmr::LeafIndex& nextLeafIdx) + LeafSet(FilePath dir, MemMap&& mmap, const mmr::LeafIndex& nextLeafIdx) : ILeafSet(nextLeafIdx), m_dir(std::move(dir)), m_mmap(std::move(mmap)) {} - FilePath m_dir; - MemMap m_mmap; - std::unordered_map m_modifiedBytes; + FilePath m_dir; + MemMap m_mmap; + std::unordered_map m_modifiedBytes; }; class LeafSetCache : public ILeafSet { public: - using Ptr = std::shared_ptr; - using UPtr = std::unique_ptr; + using Ptr = std::shared_ptr; + using UPtr = std::unique_ptr; - LeafSetCache(const ILeafSet::Ptr& pBacked) - : ILeafSet(pBacked->GetNextLeafIdx()), m_pBacked(pBacked) { } + LeafSetCache(const ILeafSet::Ptr& pBacked) + : ILeafSet(pBacked->GetNextLeafIdx()), m_pBacked(pBacked) { } - uint8_t GetByte(const uint64_t byteIdx) const final; - void SetByte(const uint64_t byteIdx, const uint8_t value) final; + uint8_t GetByte(const uint64_t byteIdx) const final; + void SetByte(const uint64_t byteIdx, const uint8_t value) final; - void ApplyUpdates( - const uint32_t file_index, - const mmr::LeafIndex& nextLeafIdx, - const std::unordered_map& modifiedBytes - ) final; - void Flush(const uint32_t file_index); + void ApplyUpdates( + const uint32_t file_index, + const mmr::LeafIndex& nextLeafIdx, + const std::unordered_map& modifiedBytes + ) final; + void Flush(const uint32_t file_index); private: - ILeafSet::Ptr m_pBacked; - std::unordered_map m_modifiedBytes; + ILeafSet::Ptr m_pBacked; + std::unordered_map m_modifiedBytes; }; \ No newline at end of file diff --git a/src/libmw/include/mw/mmr/MMR.h b/src/libmw/include/mw/mmr/MMR.h index ec537b29c3..cffea52bc0 100644 --- a/src/libmw/include/mw/mmr/MMR.h +++ b/src/libmw/include/mw/mmr/MMR.h @@ -56,6 +56,13 @@ public: /// The next leaf index. virtual mmr::LeafIndex GetNextLeafIdx() const noexcept = 0; + /// + /// Gets the number of nodes in the MMR, including those that have been pruned. + /// eg. If the MMR contains leaves 0, 1, and 2, this will return 4. + /// + /// The number of (pruned and unpruned) nodes in the MMR. + uint64_t GetNumNodes() const noexcept { return GetNextLeafIdx().GetPosition(); } + /// /// Gets the number of leaves in the MMR, including those that have been pruned. /// eg. If the MMR contains leaves 0, 1, and 2, this will return 3. diff --git a/src/libmw/include/mw/mmr/MMRUtil.h b/src/libmw/include/mw/mmr/MMRUtil.h index e7678adf84..b6a28027bf 100644 --- a/src/libmw/include/mw/mmr/MMRUtil.h +++ b/src/libmw/include/mw/mmr/MMRUtil.h @@ -4,13 +4,14 @@ #include #include -class ILeafSet; +class IMMR; class MMRUtil { public: static mw::Hash CalcParentHash(const mmr::Index& index, const mw::Hash& left_hash, const mw::Hash& right_hash); static std::vector CalcPeakIndices(const uint64_t num_nodes); + static boost::optional CalcBaggedPeak(const IMMR& mmr, const mmr::Index& peak_idx); static BitSet BuildCompactBitSet(const uint64_t num_leaves, const BitSet& unspent_leaf_indices); static BitSet DiffCompactBitSet(const BitSet& prev_compact, const BitSet& new_compact); diff --git a/src/libmw/include/mw/mmr/Segment.h b/src/libmw/include/mw/mmr/Segment.h index 3d99ae89e1..1c7897a0b9 100644 --- a/src/libmw/include/mw/mmr/Segment.h +++ b/src/libmw/include/mw/mmr/Segment.h @@ -45,15 +45,8 @@ private: static std::set CalcHashIndices( const ILeafSet& leafset, const std::vector& peak_indices, - const mmr::LeafIndex& first_leaf_idx, - const mmr::LeafIndex& last_leaf_idx - ); - - static boost::optional BagNextLowerPeak( - const IMMR& mmr, - const std::vector& peak_indices, - const mmr::Index& peak_idx, - const uint64_t num_nodes + const LeafIndex& first_leaf_idx, + const LeafIndex& last_leaf_idx ); }; diff --git a/src/libmw/src/mmr/IMMR.cpp b/src/libmw/src/mmr/IMMR.cpp index 996bd42018..cd3b0dc59d 100644 --- a/src/libmw/src/mmr/IMMR.cpp +++ b/src/libmw/src/mmr/IMMR.cpp @@ -5,7 +5,7 @@ using namespace mmr; mw::Hash IMMR::Root() const { - const uint64_t num_nodes = mmr::LeafIndex::At(GetNumLeaves()).GetPosition(); + const uint64_t num_nodes = GetNumNodes(); // Find the "peaks" std::vector peak_indices = MMRUtil::CalcPeakIndices(num_nodes); diff --git a/src/libmw/src/mmr/MMRUtil.cpp b/src/libmw/src/mmr/MMRUtil.cpp index 72422852be..8e6245ba61 100644 --- a/src/libmw/src/mmr/MMRUtil.cpp +++ b/src/libmw/src/mmr/MMRUtil.cpp @@ -1,4 +1,5 @@ #include +#include #include #include @@ -42,6 +43,31 @@ std::vector MMRUtil::CalcPeakIndices(const uint64_t num_nodes) return peak_indices; } +boost::optional MMRUtil::CalcBaggedPeak(const IMMR& mmr, const mmr::Index& peak_idx) +{ + mmr::Index next_node = mmr.GetNextLeafIdx().GetNodeIndex(); + + // Find the "peaks" + std::vector peak_indices = MMRUtil::CalcPeakIndices(mmr.GetNumNodes()); + + // Bag 'em + boost::optional bagged_peak; + for (auto iter = peak_indices.crbegin(); iter != peak_indices.crend(); iter++) { + mw::Hash peakHash = mmr.GetHash(*iter); + if (bagged_peak) { + bagged_peak = MMRUtil::CalcParentHash(next_node, peakHash, *bagged_peak); + } else { + bagged_peak = peakHash; + } + + if (*iter == peak_idx) { + return bagged_peak; + } + } + + return boost::none; +} + BitSet MMRUtil::BuildCompactBitSet(const uint64_t num_leaves, const BitSet& unspent_leaf_indices) { BitSet compactable_node_indices(num_leaves * 2); @@ -61,7 +87,7 @@ BitSet MMRUtil::BuildCompactBitSet(const uint64_t num_leaves, const BitSet& unsp Index last_node = Index::At(next_leaf.GetPosition() - 1); uint64_t height = 1; - while ((std::pow(2, height + 1) - 2) <= next_leaf.GetPosition()) { + while ((uint64_t(2) << height) - 2 <= next_leaf.GetPosition()) { SiblingIter iter(height, last_node); while (iter.Next()) { Index right_child = iter.Get().GetRightChild(); @@ -143,7 +169,7 @@ BitSet MMRUtil::CalcPrunedParents(const BitSet& unspent_leaf_indices) SiblingIter::SiblingIter(const uint64_t height, const Index& last_node) : m_height(height), m_lastNode(last_node), - m_baseInc((uint64_t)std::pow(2, height + 1) - 1), + m_baseInc((uint64_t(2) << height) - 1), m_siblingNum(0), m_next() { diff --git a/src/libmw/src/mmr/MemMMR.cpp b/src/libmw/src/mmr/MemMMR.cpp index 1d0c361bdc..eced43ee2a 100644 --- a/src/libmw/src/mmr/MemMMR.cpp +++ b/src/libmw/src/mmr/MemMMR.cpp @@ -45,5 +45,5 @@ void MemMMR::Rewind(const uint64_t numLeaves) { assert(numLeaves <= m_leaves.size()); m_leaves.resize(numLeaves); - m_hashes.resize(GetNextLeafIdx().GetPosition()); + m_hashes.resize(GetNumNodes()); } \ No newline at end of file diff --git a/src/libmw/src/mmr/PMMRCache.cpp b/src/libmw/src/mmr/PMMRCache.cpp index ba4c8b49c7..b6136b9958 100644 --- a/src/libmw/src/mmr/PMMRCache.cpp +++ b/src/libmw/src/mmr/PMMRCache.cpp @@ -75,7 +75,7 @@ void PMMRCache::Rewind(const uint64_t numLeaves) m_leaves.erase(iter, m_leaves.end()); } - const uint64_t numNodes = GetNextLeafIdx().GetPosition() - m_firstLeaf.GetPosition(); + const uint64_t numNodes = GetNumNodes() - m_firstLeaf.GetPosition(); if (m_nodes.size() > numNodes) { m_nodes.erase(m_nodes.begin() + numNodes, m_nodes.end()); } diff --git a/src/libmw/src/mmr/Segment.cpp b/src/libmw/src/mmr/Segment.cpp index 6b31d31b70..7087f2869e 100644 --- a/src/libmw/src/mmr/Segment.cpp +++ b/src/libmw/src/mmr/Segment.cpp @@ -25,8 +25,7 @@ Segment SegmentFactory::Assemble(const IMMR& mmr, const ILeafSet& leafset, const ++leaf_idx; } - const uint64_t num_nodes = leafset.GetNextLeafIdx().GetPosition(); - std::vector peak_indices = MMRUtil::CalcPeakIndices(num_nodes); + std::vector peak_indices = MMRUtil::CalcPeakIndices(leafset.GetNumNodes()); assert(!peak_indices.empty()); // Populate hashes @@ -37,16 +36,19 @@ Segment SegmentFactory::Assemble(const IMMR& mmr, const ILeafSet& leafset, const } // Determine the lowest peak that can be calculated using the hashes we've already provided - auto peak = *std::find_if( + auto peak = std::find_if( peak_indices.begin(), peak_indices.end(), [&last_leaf_idx](const Index& peak_idx) { return peak_idx > last_leaf_idx.GetNodeIndex(); }); // Bag the next lower peak (if there is one), so the root can still be calculated - segment.lower_peak = BagNextLowerPeak(mmr, peak_indices, peak, num_nodes); + if (peak != peak_indices.end() && peak++ != peak_indices.end()) { + segment.lower_peak = MMRUtil::CalcBaggedPeak(mmr, *peak); + } return segment; } + std::set SegmentFactory::CalcHashIndices( const ILeafSet& leafset, const std::vector& peak_indices, @@ -98,11 +100,8 @@ std::set SegmentFactory::CalcHashIndices( assert(peak_iter != peak_indices.end()); Index peak = *peak_iter; - auto on_mountain_right_edge = [prev_peak, peak](const Index& idx) -> bool { - const uint64_t adjustment = prev_peak - .map([](const Index& i) { return i.GetPosition() + 1; }) - .value_or(0); - return (idx.GetPosition() - adjustment) >= ((peak.GetPosition() - adjustment) - peak.GetHeight()); + auto on_mountain_right_edge = [peak](const Index& idx) -> bool { + return idx.GetPosition() >= peak.GetPosition() - peak.GetHeight(); }; idx = last_leaf_idx.GetNodeIndex(); @@ -117,31 +116,4 @@ std::set SegmentFactory::CalcHashIndices( } return proof_indices; -} - -boost::optional SegmentFactory::BagNextLowerPeak( - const IMMR& mmr, - const std::vector& peak_indices, - const mmr::Index& peak_idx, - const uint64_t num_nodes) -{ - boost::optional lower_peak = boost::none; - - if (peak_idx != peak_indices.back()) { - // Bag peaks until we reach the next lowest - for (auto iter = peak_indices.crbegin(); iter != peak_indices.crend(); iter++) { - if (*iter == peak_idx) { - break; - } - - mw::Hash peakHash = mmr.GetHash(*iter); - if (lower_peak) { - lower_peak = MMRUtil::CalcParentHash(Index::At(num_nodes), peakHash, *lower_peak); - } else { - lower_peak = peakHash; - } - } - } - - return lower_peak; } \ No newline at end of file diff --git a/src/libmw/test/tests/mmr/Test_Segment.cpp b/src/libmw/test/tests/mmr/Test_Segment.cpp index a6f6aaf337..465ae461c3 100644 --- a/src/libmw/test/tests/mmr/Test_Segment.cpp +++ b/src/libmw/test/tests/mmr/Test_Segment.cpp @@ -7,7 +7,6 @@ #include #include #include -#include #include @@ -26,7 +25,7 @@ struct MMRWithLeafset { ILeafSet::Ptr leafset; }; -static mmr::Leaf DeterministicLeaf(const size_t i) +static mmr::Leaf DeterministicLeaf(const uint64_t i) { std::vector serialized{ uint8_t(i >> 24), @@ -36,7 +35,7 @@ static mmr::Leaf DeterministicLeaf(const size_t i) return mmr::Leaf::Create(mmr::LeafIndex::At(i), serialized); } -static MMRWithLeafset BuildDetermininisticMMR(const size_t num_leaves) +static MMRWithLeafset BuildDetermininisticMMR(const uint64_t num_leaves) { auto mmr = std::make_shared(); auto leafset = LeafSet::Open(GetDataDir(), 0); @@ -48,32 +47,6 @@ static MMRWithLeafset BuildDetermininisticMMR(const size_t num_leaves) return MMRWithLeafset{mmr, leafset}; } -static boost::optional CalcBaggedPeak(const IMMR::Ptr& mmr, const mmr::Index& peak_idx) -{ - const uint64_t num_nodes = mmr->GetNextLeafIdx().GetPosition(); - - // Find the "peaks" - std::vector peak_indices = MMRUtil::CalcPeakIndices(num_nodes); - - // Bag 'em - boost::optional bagged_peak; - for (auto iter = peak_indices.crbegin(); iter != peak_indices.crend(); iter++) { - mw::Hash peakHash = mmr->GetHash(*iter); - if (bagged_peak) { - bagged_peak = MMRUtil::CalcParentHash(Index::At(num_nodes), peakHash, *bagged_peak); - } else { - bagged_peak = peakHash; - } - - BOOST_TEST_MESSAGE("peak(" << iter->GetPosition() << "): " << bagged_peak); - if (*iter == peak_idx) { - return bagged_peak; - } - } - - return bagged_peak; -} - BOOST_FIXTURE_TEST_SUITE(TestSegment, MWEBTestingSetup) BOOST_AUTO_TEST_CASE(AssembleSegment) @@ -101,7 +74,7 @@ BOOST_AUTO_TEST_CASE(AssembleSegment) }; BOOST_REQUIRE_EQUAL_COLLECTIONS(segment.hashes.begin(), segment.hashes.end(), expected_hashes.begin(), expected_hashes.end()); - boost::optional expected_lower_peak = CalcBaggedPeak(mmr, mmr::Index::At(21)); + boost::optional expected_lower_peak = MMRUtil::CalcBaggedPeak(*mmr, mmr::Index::At(21)); BOOST_REQUIRE_EQUAL(expected_lower_peak, segment.lower_peak); // Verify PMMR root can be fully recomputed diff --git a/src/net_processing.cpp b/src/net_processing.cpp index fe2f6ca9c6..41b5bc4fb3 100644 --- a/src/net_processing.cpp +++ b/src/net_processing.cpp @@ -1884,7 +1884,6 @@ static void ProcessGetMWEBUTXOs(CNode& pfrom, const ChainstateManager& chainman, } auto mweb_cache = temp_view.GetMWEBCacheView(); - auto pLeafset = mweb_cache->GetLeafSet(); mmr::Segment segment = mmr::SegmentFactory::Assemble( *mweb_cache->GetOutputPMMR(), diff --git a/src/protocol.cpp b/src/protocol.cpp index edb2c33f41..4e8263acdf 100644 --- a/src/protocol.cpp +++ b/src/protocol.cpp @@ -92,6 +92,8 @@ const static std::string allNetMessageTypes[] = { NetMsgType::WTXIDRELAY, NetMsgType::MWEBHEADER, NetMsgType::MWEBLEAFSET, + NetMsgType::GETMWEBUTXOS, + NetMsgType::MWEBUTXOS, }; const static std::vector allNetMessageTypesVec(allNetMessageTypes, allNetMessageTypes+ARRAYLEN(allNetMessageTypes));