Pull additional nodes from virtu's crawler. Data includes sufficient
Onion and I2P nodes to align the uptime requirements for these networks
to that of clearnet nodes (i.e., 50%). Data also includes more than
three times the number of CJDNS nodes currently hardcoded into
nodes_main_manual.txt, so hardcoded nodes becomes obsolete.
The seeders now produce onion and i2p seeds, so there is no need to keep these
in the manual list.
Although should also be produced, there are not enough
good ones detected by the seeder, so we keep the manual seeds for them.
The crawlers are not guaranteed to output nodes in a random order, so
shuffle the ips list after parsing to break any biasing that may be
caused by the output order.
Update the user agent regex to match all 3 digits of the version number,
not just the first 2 digits.
Also updates it to include 24.2, 25.2, 26.1, 27.0, 27.1, 27.99, 28.0 and
28.99.
fa52e13ee8 test: Remove struct.pack from almost all places (MarcoFalke)
fa826db477 scripted-diff: test: Use int.to_bytes over struct packing (MarcoFalke)
faf2a975ad test: Use int.to_bytes over struct packing (MarcoFalke)
faf3cd659a test: Normalize struct.pack format (MarcoFalke)
Pull request description:
`struct.pack` has many issues:
* The format string consists of characters that may be confusing and may need to be looked up in the documentation, as opposed to using easy to understand self-documenting code.
This lead to many test bugs, which weren't hit, which is fine, but still confusing. Ref: https://github.com/bitcoin/bitcoin/pull/29400, https://github.com/bitcoin/bitcoin/pull/29399, https://github.com/bitcoin/bitcoin/pull/29363, fa3886b7c6, ...
Fix all issues by using the built-in `int` helpers `to_bytes` via a scripted diff.
Review notes:
* For `struct.pack` and `int.to_bytes` the error behavior is the same, although the error messages are not identical.
* Two `struct.pack` remain. One for float serialization in a C++ code comment, and one for native serialization.
ACKs for top commit:
achow101:
ACK fa52e13ee8
rkrux:
tACK [fa52e13](fa52e13ee8)
theStack:
Code-review ACK fa52e13ee8
Tree-SHA512: ee80d935b68ae43d1654b047e84ceb80abbd20306df35cca848b3f7874634b518560ddcbc7e714e2e7a19241e153dee64556dc4701287ae811e26e4f8c57fe3e
6abe772a17 contrib: Add asmap-tool (Fabian Jahr)
Pull request description:
This adds `asmap.py` and `asmap-tool.py` from sipa's `nextgen` branch: https://github.com/sipa/asmap/tree/nextgen
The motivation is that we should maintain the tooling for de- and encoding asmap files within the bitcoin core repository because it is not possible to use an asmap file that is not encoded.
We already had an earlier version of `asmap.py` within the seeds contrib tools. The newer version only had a small amount of changes and is still compatible, so the old version is removed from contrib/seeds and the new version is made available to `makeseeds.py`.
ACKs for top commit:
virtu:
ACK [6abe772](6abe772a17)
0xB10C:
ACK 6abe772a17
achow101:
ACK 6abe772a17
brunoerg:
ACK 6abe772a17
Tree-SHA512: cc2a82ffa4eb46fa0ce4ca769dd82f8d0d2f37fc3652aa748eeb060e1142f9da4035008fe89433e2fd524a4dc153b7b9c085748944b49137b37009b0c0be8afb
Since Python 3.9, type hinting has become a little less awkward, as for
collection types one doesn't need to import the corresponding
capitalized types (`Dict`, `List`, `Set`, `Tuple`, ...) anymore, but can
use the built-in types directly. [1] [2]
This commit applies the replacement for all Python scripts (i.e. in the
contrib and test folders) for the basic types:
- typing.Dict -> dict
- typing.List -> list
- typing.Set -> set
- typing.Tuple -> tuple
[1] https://docs.python.org/3.9/whatsnew/3.9.html#type-hinting-generics-in-standard-collections
[2] https://peps.python.org/pep-0585/#implementation for a list of type
a2bef805c1 kernel: update m_assumed_* chain params for 25.x (fanquake)
4128e01dba kernel: update chainTxData for 25.x (fanquake)
00b2b114b4 kernel: update nMinimumChainWork & defaultAssumeValid for 25.x (fanquake)
07fcc0a82c doc: update references to kernel/chainparams.cpp (fanquake)
Pull request description:
Update chainparams pre `25.x` branch off.
Co-Author in the commits as a PR (#27223) had previously been opened too-early to do the same.
Note: Remember that some variance is expected in the `m_assumed_*` sizes.
ACKs for top commit:
achow101:
ACK a2bef805c1
josibake:
ACK a2bef805c1
gruve-p:
ACK a2bef805c1
dergoegge:
ACK a2bef805c1 on the new mainnet params
Tree-SHA512: 0b19c2ef15c6b15863d6a560a1053ee223057c7bfb617ffd3400b1734cee8f75bc6fd7f04d8f8e3f5af6220659a1987951a1b36945d6fe17d06972004fd62610
3cc989da5c Fix checking bad dns seeds without casting (Yusuf Sahin HAMZA)
Pull request description:
- Since seed lines comes with `str` type, comparing `good` column directly with **0** (`int` type) in the if statement was not working at all. This is fixed by casting `int` type to the values in the `good` column of seeds text file.
- Lines that starts with comment in the seeds text file are now ignored.
- If statement for checking bad seeds are moved to the top of the `parseline` function as if a seed is bad; there is no point of going forward from there.
Since this bug-fix eliminates bad seeds over **550k** in the first place, in my case; particular job for parsing all seeds speed is up by **600%** and whole script's speed is up by **%30**.
Note that **stats** in the terminal are not going to include bad seeds after this fix, which would be the same if this bug were never there before.
ACKs for top commit:
achow101:
ACK 3cc989da5c
jonatack:
ACK 3cc989da5c
Tree-SHA512: 13c82681de4d72de07293f0b7f09721ad8514a2ad99b0584d1c94fa5f2818821df2000944f9514d6a222a5dccc82856d16c8c05aa36d905cfa7d4610c629fd38
1c07500dbb contrib: make DNS seeds file an argument in CLI (brunoerg)
Pull request description:
Instead of using `makeseeds.py` this way:
```sh
python3 makeseeds.py -a asmap-filled.dat < seeds_main.txt > nodes_main.txt
```
We could use the DNS seeds file as an argument since it is a required one. It improves the way the script handles it when that file is missing as well as makes this script more friendly.
E.g:
```sh
python3 makeseeds.py -a asmap-filled.dat -s seeds_main.txt > nodes_main.txt
```
ACKs for top commit:
vincenzopalazzo:
ACK 1c07500dbb
Tree-SHA512: bddf728d5d376659155f5bbeb1fa0d42aa273ec4a0cf5687f4d3f3be85625f541d392f30008e3c9d2c65967cb882deb36af34330994727771be73c9adeb521e0
Since seed lines comes with 'str' type, comparing it directly with 0
('int' type) in the if statement was not working at all. This is fixed
by casting 'int' type to the values in the 'good' column of seeds text file.
Lines that starts with comment in the seeds text file are now ignored.
If statement for checking bad seeds are moved to the top of the 'parseline'
function as if seed is bad, there is no point of going forward from there.
2ef33e936e contrib: update testnet torv3 hardcoded seeds (Jon Atack)
Pull request description:
As a follow-up to https://github.com/bitcoin/bitcoin/issues/13550 and #22060, replace the mostly unreachable testnet torv3 hardcoded seeds from v0.22 with new ones that are consistently reachable recently and that have service bit 1 set.
This needs to be done before v24.0 to make sure onion-only testnet nodes can still connect to the network.
Ways to test:
- Re-generate `src/chainparamsseeds.h` with `cd contrib/seeds && python3 generate-seeds.py . > ../../src/chainparamsseeds.h`, check if git tree stays the same
- Re-compile and create a new testnet node with `bitcoind -testnet -dnsseed=0 -onlynet=onion -proxy=127.0.0.1:9050` (or delete `~/.bitcoin/testnet3/peers.dat` and launch bitcoind with `-testnet -dnsseed=0`). Make sure there are no `addnode=` in your `bitcoin.conf`. The debug log should print "Adding fixed seeds". Check if the node is able to connect to the network and get blocks with for ex. `watch -t ./src/bitcoin-cli -testnet -rpcwait -netinfo 4`
- Check the addrman contains the seeds by running for ex. `bitcoin-cli -rpcwait -testnet getnodeaddresses 0 onion | jq -r '.[] | (.address + ":" + (.port|tostring) + " " + (.services|tostring))' | sort`
- Check if the addresses are connectable, for ex. with this python script by laanwj:
```python3
#!/usr/bin/env python3
import pprint
import subprocess
with open('contrib/seeds/nodes_test.txt') as f:
for line in (line for line in (line.rstrip().split('#', 1)[0] for line in f) if line):
pprint.pprint(line)
subprocess.call(["nc", "-v", "-x", "127.0.0.1:9050", "-z"] + line.split(':'))
```
Thanks to satsie (Stacie Waleyko) for help with the list.
ACKs for top commit:
satsie:
ACK 2ef33e936e
laanwj:
ACK 2ef33e936e
Tree-SHA512: 72d27ecba243089bd49c11e921855fba626a1e09ae9b17508254a3bbec4bec341ed6c3d5a4eabc2d37f20bafb8a47ecc7d125e0dda956512a9525ad83273ffd6
Add an argument `-a` to provide a asmap file to do the IP to ASN
lookups.
This speeds up the script greatly, and makes the output deterministic.
Also removes the dependency on `dns.lookup`.
I've annotated the output with ASxxxx comments to provide a way to
verify the functionality.
For now I've added instructions in README.md to download and use the
`demo.map` from the asmap repository. When we have some other mechanism
for distributing asmap files we could switch to that.
This continues #24824. I've removed all the fallbacks and extra
complexity, as everyone will be using the same instructions anyway.
Co-authored-by: Pieter Wuille <pieter.wuille@gmail.com>
Co-authored-by: James O'Beirne <james.obeirne@pm.me>
Co-authored-by: russeree <reese.russell@ymail.com>
I have some qualms with maintaining a suspicious hosts list as part as
the repository. But also, it's stale and irrelevant. I've checked the
entire list and none of them is connectable. Only one still appars in
`nodes_main.txt` but with low uptime and an old subversion string so it
wouldn't be picked in the first place.
Documentation:
- Use https URL for bitcoin.sipa.be (http sends a redirect, fooling
curl).
- Add explicit step to add manual seeds.
Code:
- Change PATTERN_ONION to v3 (effectively means that the no onion hosts
are delivered).
- Add versions to PATTERN_AGENT filter.
- Print specific message on resolve exception.
An IPv6 address from fc00::/8 could be either from the CJDNS network or
from a private-unroutable-reserved segment of IPv6. A seed node with
such an address must be from the CJDNS network, otherwise other peers
will not be able to connect to it.
virtu