[Net] asmap to improve IP bucketing in addrman - backports #2480
Conversation
furszy
force-pushed
the
2021_ASN_IP_based_bucketing
branch
2 times, most recently
from
33ae75b
to
7fead2c
Compare
|
rebased on master, ready to go. |
random-zebra
added
the
Needs Release Notes
furszy
force-pushed
the
2021_ASN_IP_based_bucketing
branch
from
7fead2c
to
334df2e
Compare
|
Updated per zebra's feedback 👌, fixed the two commented test cases :). |
furszy
force-pushed
the
2021_ASN_IP_based_bucketing
branch
2 times, most recently
from
13915de
to
97e0497
Compare
furszy
force-pushed
the
2021_ASN_IP_based_bucketing
branch
from
97e0497
to
a7138cf
Compare
|
done, squashed. I missed it between all of the open works. |
|
IMHO, I think eb3eba0 should be dropped and the issue fixed directly in upstream. The extra indentation goes against the styling policies/guidelines for both us and upstream, and I would prefer to avoid intentionally (re-)introducing a policy violation that had been explicitly fixed in the past for a marginal improvement in ease of cherry-picking, especially being relatively close to upstream's HEAD in this particular file; which, given a quick glance, seems to be one of, if not the only file that has this styling discrepancy. |
|
Upstream PR addressing the indentation issue can be found at bitcoin#22562 |
|
yeah, it's the only file with text indentation discrepancies. I made it because after, literally, more than two hundred commits back ported and adapted to our sources, it was becoming a pain in the ass to struggle against the same text indentation issue over and over, (with literally no benefits as it's just text indentation that can be re-added all at once at the end). Conflicting most of the file most of the times. Saying that, now that the road to Tor path is clear (In |
furszy
force-pushed
the
2021_ASN_IP_based_bucketing
branch
from
a7138cf
to
011d033
Compare
The scripts for creating a compact IP->ASN mapping are here: https://github.com/sipa/asmap Co-authored-by: Pieter Wuille <pieter.wuille@gmail.com>
Instead of using /16 netgroups to bucket nodes in Addrman for connection diversification, ASN, which better represents an actor in terms of network-layer infrastructure, is used. For testing, asmap.raw is used. It represents a minimal asmap needed for testing purposes. Adaptation of btc@ec45646de9e62b3d42c85716bfeb06d8f2b507dc with the latest version of the asmap.raw file.
-BEGIN VERIFY SCRIPT- sed --in-place'' --expression='s/NET_TOR/NET_ONION/g' $(git grep -I --files-with-matches 'NET_TOR') -END VERIFY SCRIPT- The --in-place'' hack is required for sed on macOS to edit files in-place without passing a backup extension.
If ASN bucketing is used, return a corresponding AS used in bucketing for a given peer.
to verify node behaviour and debug log when launching bitcoind in these cases: 1. `bitcoind` with no -asmap arg, using /16 prefix for IP bucketing 2. `bitcoind -asmap=<relative path>`, using the unit test skeleton asmap 3. `bitcoind -asmap/-asmap=` with no file specified, using the default asmap 4. `bitcoind -asmap` with no file specified, and a missing default asmap file The tests are order-independent. The slowest test (missing default asmap file) is placed last.
- allow passing an absolute file path to the -asmap config arg - update the -asmap config help - add a functional test in feature_asmap.py
and update the tests.
This is now testable after separating the asmap finding and parsing checks in the previous commit.
- move asmap #includes to sorted positions in addrman and init (move-only) - remove redundant quotes in asmap InitError, update test - remove full stops from asmap logging to be consistent with debug logging, update tests
Remove redundant public declaration and mention if mapped_as is only available if the asmap config flag is set.
and update feature_asmap.py and test_runner.py This commit moves the asmap init.cpp code from the end of "Step 12: start node" to "Step 6: network initialization" to provide feedback on passing an -asmap config arg much more quickly. This change speeds up the feature_asmap.py functional test file from 60 to 5 seconds by accelerating the 2 tests that use `assert_start_raises_init_error`. Credit to Wladimir J. van der Laan for the suggestion.
furszy
force-pushed
the
2021_ASN_IP_based_bucketing
branch
from
011d033
to
16791f2
Compare
|
rebased again, conflict solved. Ready to go. |
random-zebra
removed
the
Needs Release Notes
Decoupled from #2411, built on top of #2479. Probably the last decouple from the "road to Tor" work.
Focused on porting the ASN nodes bucketing functionality. The hearth of this work is bitcoin#16702.
Providing an asmap file that contains the IP->ASN mapping, nodes will be bucketed by AS they belong to, in order to make impossible for a node to connect to several nodes hosted in a single AS.
This is done in response to Erebus attack, but also to generally diversify the connections every node creates, especially useful when a large fraction of nodes operate under a couple of cloud providers.
List of PRs:
PRs for a follow up PR: