Skip to content

Commit

Permalink
Merge #19988: Overhaul transaction request logic
Browse files Browse the repository at this point in the history
fd9a006 Report and verify expirations (Pieter Wuille)
86f50ed Delete limitedmap as it is unused now (Pieter Wuille)
cc16fff Make txid delay penalty also apply to fetches of orphan's parents (Pieter Wuille)
173a1d2 Expedite removal of tx requests that are no longer needed (Pieter Wuille)
de11b0a Reduce MAX_PEER_TX_ANNOUNCEMENTS for non-PF_RELAY peers (Pieter Wuille)
242d164 Change transaction request logic to use txrequest (Pieter Wuille)
5b03121 Add txrequest fuzz tests (Pieter Wuille)
3c7fe0e Add txrequest unit tests (Pieter Wuille)
da3b8fd Add txrequest module (Pieter Wuille)

Pull request description:

  This replaces the transaction request logic with an encapsulated class that maintains all the state surrounding it. By keeping it stand alone, it can be easily tested (using included unit tests and fuzz tests).

  The major changes are:

  * Announcements from outbound (and whitelisted) peers are now always preferred over those from inbound peers. This used to be the case for the first request (by delaying the first request from inbound peers), and a bias afters. The 2s delay for requests from inbound peers still exists, but after that, if viable outbound peers remain for any given transaction, they will always be tried first.
  * No more hard cap of 100 in flight transactions per peer, as there is less need for it (memory usage is linear in the number of announcements, but independent from the number in flight, and CPU usage isn't affected by it). Furthermore, if only one peer announces a transaction, and it has over 100 in flight already, we still want to request it from them. The cap is replaced with a rule that announcements from such overloaded peers get an additional 2s delay (possibly combined with the existing 2s delays for inbound connections, and for txid peers when wtxid peers are available).
  * The limit of 100000 tracked announcements is reduced to 5000; this was excessive. This can be bypassed using the PF_RELAY permission (to accommodate locally dumping a batch of many transactions).

  This replaces #19184, rebased on #18044 and with many small changes.

ACKs for top commit:
  ariard:
    Code Review ACK fd9a006. I've reviewed the new TxRequestTracker, its integration in net_processing, unit/functional/fuzzing test coverage. I looked more for soundness of new specification rather than functional consistency with old transaction request logic.
  MarcoFalke:
    Approach ACK fd9a006 🏹
  naumenkogs:
    Code Review ACK fd9a006. I've reviewed everything, mostly to see how this stuff works at the lower level (less documentation-wise, more implementation-wise), and to try breaking it with unexpected sequences of events.
  jnewbery:
    utACK fd9a006
  jonatack:
    WIP light ACK fd9a006 have read the code, verified that each commit is hygienic, e.g. debug build clean and tests green, and have been running a node on and off with this branch and grepping the net debug log. Am still unpacking the discussion hidden by GitHub by fetching it via the API and connecting the dots, storing notes and suggestions in a local branch; at this point none are blockers.
  ryanofsky:
    Light code review ACK fd9a006, looking at txrequest implementation, unit test implementation, and net_processing integration, just trying to understand how it works and looking for anything potentially confusing in the implementation. Didn't look at functional tests or catch up on review discussion. Just a sanity check review focused on:

Tree-SHA512: ea7b52710371498b59d9c9cfb5230dd544fe9c6cb699e69178dea641646104f38a0b5ec7f5f0dbf1eb579b7ec25a31ea420593eff3b7556433daf92d4b0f0dd7
  • Loading branch information
laanwj committed Oct 14, 2020
2 parents 99a1d57 + fd9a006 commit c2c4dba
Show file tree
Hide file tree
Showing 18 changed files with 2,297 additions and 447 deletions.
9 changes: 9 additions & 0 deletions doc/release-notes-19988.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
P2P changes
-----------

The size of the set of transactions that peers have announced and we consider
for requests has been reduced from 100000 to 5000 (per peer), and further
announcements will be ignored when that limit is reached. If you need to
dump (very) large batches of transactions, exceptions can be made for trusted
peers using the "relay" network permission. For localhost for example it can
be enabled using the command line option `-whitelist=relay@127.0.0.1`.
3 changes: 2 additions & 1 deletion src/Makefile.am
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,6 @@ BITCOIN_CORE_H = \
interfaces/wallet.h \
key.h \
key_io.h \
limitedmap.h \
logging.h \
logging/timer.h \
memusage.h \
Expand Down Expand Up @@ -215,6 +214,7 @@ BITCOIN_CORE_H = \
timedata.h \
torcontrol.h \
txdb.h \
txrequest.h \
txmempool.h \
undo.h \
util/asmap.h \
Expand Down Expand Up @@ -327,6 +327,7 @@ libbitcoin_server_a_SOURCES = \
timedata.cpp \
torcontrol.cpp \
txdb.cpp \
txrequest.cpp \
txmempool.cpp \
validation.cpp \
validationinterface.cpp \
Expand Down
9 changes: 8 additions & 1 deletion src/Makefile.test.include
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,7 @@ FUZZ_TARGETS = \
test/fuzz/tx_in_deserialize \
test/fuzz/tx_out \
test/fuzz/txoutcompressor_deserialize \
test/fuzz/txrequest \
test/fuzz/txundo_deserialize \
test/fuzz/uint160_deserialize \
test/fuzz/uint256_deserialize
Expand Down Expand Up @@ -235,7 +236,6 @@ BITCOIN_TESTS =\
test/interfaces_tests.cpp \
test/key_io_tests.cpp \
test/key_tests.cpp \
test/limitedmap_tests.cpp \
test/logging_tests.cpp \
test/dbwrapper_tests.cpp \
test/validation_tests.cpp \
Expand Down Expand Up @@ -275,6 +275,7 @@ BITCOIN_TESTS =\
test/torcontrol_tests.cpp \
test/transaction_tests.cpp \
test/txindex_tests.cpp \
test/txrequest_tests.cpp \
test/txvalidation_tests.cpp \
test/txvalidationcache_tests.cpp \
test/uint256_tests.cpp \
Expand Down Expand Up @@ -1214,6 +1215,12 @@ test_fuzz_txoutcompressor_deserialize_LDADD = $(FUZZ_SUITE_LD_COMMON)
test_fuzz_txoutcompressor_deserialize_LDFLAGS = $(FUZZ_SUITE_LDFLAGS_COMMON)
test_fuzz_txoutcompressor_deserialize_SOURCES = test/fuzz/deserialize.cpp

test_fuzz_txrequest_CPPFLAGS = $(AM_CPPFLAGS) $(BITCOIN_INCLUDES)
test_fuzz_txrequest_CXXFLAGS = $(AM_CXXFLAGS) $(PIE_FLAGS)
test_fuzz_txrequest_LDADD = $(FUZZ_SUITE_LD_COMMON)
test_fuzz_txrequest_LDFLAGS = $(FUZZ_SUITE_LDFLAGS_COMMON)
test_fuzz_txrequest_SOURCES = test/fuzz/txrequest.cpp

test_fuzz_txundo_deserialize_CPPFLAGS = $(AM_CPPFLAGS) $(BITCOIN_INCLUDES) -DTXUNDO_DESERIALIZE=1
test_fuzz_txundo_deserialize_CXXFLAGS = $(AM_CXXFLAGS) $(PIE_FLAGS)
test_fuzz_txundo_deserialize_LDADD = $(FUZZ_SUITE_LD_COMMON)
Expand Down
100 changes: 0 additions & 100 deletions src/limitedmap.h

This file was deleted.

1 change: 0 additions & 1 deletion src/net.h
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@
#include <compat.h>
#include <crypto/siphash.h>
#include <hash.h>
#include <limitedmap.h>
#include <net_permissions.h>
#include <netaddress.h>
#include <optional.h>
Expand Down
2 changes: 1 addition & 1 deletion src/net_permissions.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ const std::vector<std::string> NET_PERMISSIONS_DOC{
"bloomfilter (allow requesting BIP37 filtered blocks and transactions)",
"noban (do not ban for misbehavior; implies download)",
"forcerelay (relay transactions that are already in the mempool; implies relay)",
"relay (relay even in -blocksonly mode)",
"relay (relay even in -blocksonly mode, and unlimited transaction announcements)",
"mempool (allow requesting BIP35 mempool contents)",
"download (allow getheaders during IBD, no disconnect after maxuploadtarget limit)",
"addr (responses to GETADDR avoid hitting the cache and contain random records with the most up-to-date info)"
Expand Down
1 change: 1 addition & 0 deletions src/net_permissions.h
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ enum NetPermissionFlags {
// Can query bloomfilter even if -peerbloomfilters is false
PF_BLOOMFILTER = (1U << 1),
// Relay and accept transactions from this peer, even if -blocksonly is true
// This peer is also not subject to limits on how many transaction INVs are tracked
PF_RELAY = (1U << 3),
// Always relay transactions from this peer, even if already in mempool
// Keep parameter interaction: forcerelay implies relay
Expand Down
Loading

0 comments on commit c2c4dba

Please sign in to comment.