Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Split block announce processing into two parts #6958

Merged
merged 126 commits into from
Oct 2, 2020
Merged
Changes from 1 commit
Commits
Show all changes
126 commits
Select commit Hold shift + click to select a range
b336104
Split block announce processing into two parts
bkchr Aug 25, 2020
95c1420
Apply suggestions from code review
bkchr Sep 7, 2020
0416a0b
Update client/network/src/protocol/sync.rs
bkchr Sep 15, 2020
f974fe7
client/authority-discovery: Append PeerId to Multiaddr at most once (…
mxinden Aug 24, 2020
7afdc67
expose Deposit (#6943)
xlc Aug 24, 2020
67d811b
Add a `LightSyncState` field to the chain spec (#6894)
expenses Aug 24, 2020
e714641
Dynamically generate CHT roots on a full client (#6944)
expenses Aug 24, 2020
508dfcd
Enable verification logic when executing benchmarks (#6929)
shawntabrizi Aug 24, 2020
43cd1d0
grandpa: always create and send justification if there are any subscr…
andresilva Aug 24, 2020
0de5451
.maintain/monitoring/alerting-rules: Add fd alert (#6946)
mxinden Aug 24, 2020
20d4ca4
Fix benchmark read/write key tracker for keys in child storages. (#6905)
gui1117 Aug 24, 2020
4b3197c
client/authority-discovery: Limit number of addresses per authority (…
mxinden Aug 24, 2020
9f590df
⛓ ✨Add ShiftNrg Network SS58 address type (#6942)
mswezey23 Aug 24, 2020
761ec7c
update tracing attribute (#6950)
gui1117 Aug 24, 2020
f7791d0
Fix unwraps and other issues with benchmarks (#6957)
shawntabrizi Aug 25, 2020
127673f
Remove implementation of `Randomness for ()` (#6959)
bkchr Aug 26, 2020
144192c
Fix staking fuzzer. (#6954)
kianenigma Aug 26, 2020
1f33b6e
Enforce that ProtocolId is a string (#6953)
tomaka Aug 26, 2020
ebd6d5c
Support Staking Payout to Any Account (#6832)
shawntabrizi Aug 26, 2020
ce89cc9
Better prime election. (#6939)
kianenigma Aug 26, 2020
117bb45
babe: fix report_equivocation weight (#6936)
andresilva Aug 26, 2020
4c0a477
fix bench db wipe (#6965)
gui1117 Aug 26, 2020
3bddf6c
Implement request-responses protocols (#6634)
tomaka Aug 27, 2020
e4178cc
add generated weight info for pallet-collective (#6789)
apopiak Aug 28, 2020
8f28655
client/*: Treat protocol name as str and not [u8] (#6967)
mxinden Aug 28, 2020
6b2f7ea
update kvdb-rocksdb to 0.9.1 and rocksdb to 6.11.4 (#6963)
ordian Aug 29, 2020
7d53c94
Use AsyncReadExt::read_exact, not just read (#6977)
tomaka Aug 31, 2020
539f4d2
client/cli/src/config: Warn on low file descriptor limit (#6956)
mxinden Aug 31, 2020
8182870
Update substrate bip39 version. (#6955)
cheme Aug 31, 2020
b622015
Inverting events set and changed in nicks pallet (#6989)
g2udevelopment Aug 31, 2020
dbbaef1
Silence the error about non-registered protocols (#6987)
tomaka Aug 31, 2020
7972673
Change browser-demo build.sh to use python 3 again (#6992)
expenses Aug 31, 2020
9231166
fix pallet-evm features (#6995)
xlc Sep 1, 2020
9007d28
Move subcommands from sc-cli to nodes (#6948)
expenses Sep 1, 2020
8502300
ci: deploy alerting rules: fix run on changes (#6998)
gabreal Sep 1, 2020
f723458
*: Update to Prometheus v0.10.0 (#6964)
mxinden Sep 1, 2020
0f302b9
Ensure that handshake is sent back even in case of back-pressure (#6979)
tomaka Sep 2, 2020
3541fa8
frame/authority-discovery: Have authorities() return both current and…
mxinden Sep 2, 2020
f8189fc
Stop sending messages on legacy substream altogether (#6975)
tomaka Sep 2, 2020
658c389
manual seal is now consensus agnostic (#7010)
seunlanlege Sep 3, 2020
b309241
grandpa: report metrics on prevotes and precommits cast (#6970)
andresilva Sep 4, 2020
130b7d4
Fix compact npos solution edge count calculation (#7021)
kianenigma Sep 4, 2020
29c993a
Refactor & detach network metrics. (#6986)
romanb Sep 6, 2020
4dee242
Node template complete import pipeline (#7014)
JoshOrndorff Sep 7, 2020
68bdfec
client/authority-discovery: Throttle DHT requests (#7018)
mxinden Sep 7, 2020
8c92258
Update Nicks docs to clarify that it is not production-ready (#6990)
danforbes Sep 7, 2020
6deea30
Ignore wasm_gc for debug build. (#6962)
cheme Sep 8, 2020
8f4db26
Make `--file` optional for `generate-node-key` (#7043)
bkchr Sep 8, 2020
2bc6943
Downgrade wabt = 0.9.1 (#7042)
gnunicorn Sep 8, 2020
52ae515
Add metadata shadows to multisig pallet (#7029)
Sep 8, 2020
27aaf0a
Fix broken link to democracy pallet. (#7026)
humb1t Sep 8, 2020
49191fd
Revert "Fix broken link to democracy pallet. (#7026)" (#7047)
gavofyork Sep 8, 2020
79cc67d
Update the service tasks Grafana dashboard (#7038)
tomaka Sep 8, 2020
a22edf0
babe, grandpa: waive fees on valid equivocation report (#6981)
andresilva Sep 8, 2020
b4fbdda
Clarify Nicks docs (#7049)
danforbes Sep 8, 2020
67c778b
Improves EVM gas price check (#7051)
crystalin Sep 8, 2020
39d8f4d
Change wabt to wat (#7050)
pepyakin Sep 8, 2020
5fa219d
Add Dock network id for address generation (#6714)
lovesh Sep 8, 2020
0a850ae
Partial fix for transaction priority (#7034)
kianenigma Sep 9, 2020
29b42c3
What happens if we remove wat? (#7056)
pepyakin Sep 9, 2020
384f29f
Make SlashingSpans Public (#6961)
kianenigma Sep 9, 2020
d123792
client/authority-discovery/src/service: Improve docs (#7059)
mxinden Sep 9, 2020
d35044c
Decrease poll interval (#7063)
s3krit Sep 9, 2020
de59f04
Remove unused code (#7027)
Sep 9, 2020
f96646a
Disambiguate `BlockNumber` type in `decl_module` (#7061)
shawntabrizi Sep 9, 2020
89e6d66
Implement `FromStr` for `Ss58AddressFormat` (#7068)
bkchr Sep 10, 2020
bd44423
Set reserved nodes with offchain worker. (#6996)
kaichaosun Sep 10, 2020
0f61352
Rename `TRIGGER_WASM_BUILD` to `FORCE_WASM_BUILD` (#7080)
bkchr Sep 10, 2020
0fd0d95
Make decoding of `compact<perthing>` saturating instead of invalid (#…
gui1117 Sep 11, 2020
006f3f0
state_machine no_std witness externalities (#6934)
cheme Sep 11, 2020
cc3040d
Support hex encoded secret key for `--node-key` (#7052)
bkchr Sep 11, 2020
6db5fe3
Add a `build-sync-spec` subcommand and remove the CHT roots from the …
expenses Sep 11, 2020
bb7052d
Fix build sync spec (#7086)
expenses Sep 11, 2020
a9a3be3
Fail docs on warnings (#5923)
TriplEight Sep 11, 2020
4b471dd
Fix `storage::read` (#7084)
bkchr Sep 12, 2020
6c89d07
Add fuzzer for the compact custom codec implementation from PR #6720 …
viniul Sep 13, 2020
8b82790
add instantiable support for treasury pallet (#7058)
pfcoder Sep 14, 2020
a5977f5
grandpa-rpc don't share subscription manager, only executor (#7039)
octol Sep 14, 2020
caab538
pallet-collective: allow customized default vote (#6984)
sorpaas Sep 14, 2020
5f6987a
Upgrade to libp2p-0.28. (#7077)
romanb Sep 14, 2020
c071d06
pow: support uniform tie breaking in fork choice (#7073)
sorpaas Sep 14, 2020
72ea91f
Allow remotes to not open a legacy substream (#7075)
tomaka Sep 14, 2020
f5e0c63
Use diener for Polkadot companion prs (#7102)
bkchr Sep 14, 2020
b2b0db5
Update ui tests for rust 1.46.0 (#7106)
bkchr Sep 15, 2020
9d9cf8a
client/network: Expose number of entries per Kademlia bucket (#7104)
mxinden Sep 15, 2020
c2ef92d
Improve error output of wasm-builder when wasm ins't installed (#7105)
bkchr Sep 15, 2020
b805faf
Add ss58 address for Dark network (#6982)
Sep 15, 2020
c867bc2
Frame-support storage: make iterations and translate consistent (#5470)
gui1117 Sep 15, 2020
111a110
fix js dependancy alert, bumping bl version (#7110)
HarryHong Sep 15, 2020
f406f49
Make `transactional` attribute less scope dependent (#7112)
bkchr Sep 15, 2020
d2c7c1f
Add SS58 Registry (#7020)
joepetrowski Sep 16, 2020
0ddcd66
Move Staking Weights to T::WeightInfo (#7007)
kianenigma Sep 16, 2020
4e4c321
Send import notification always for re-orgs (#7118)
bkchr Sep 16, 2020
9167fe0
WeightInfo for Vesting Pallet (#7103)
shawntabrizi Sep 16, 2020
59b4668
Add benchmarking pipeline to node-template (#7122)
shawntabrizi Sep 17, 2020
2c65036
Use tracing-based subscriber logging (#6825)
sorpaas Sep 17, 2020
86026fc
Typo in error text (#7126)
Swader Sep 17, 2020
da02c0b
WeightInfo for ImOnline (#7128)
shawntabrizi Sep 17, 2020
f1380be
fix the new staking weight in substrate-node (#7131)
kianenigma Sep 17, 2020
585629a
Remove warning about deprecated PeerIds (#7132)
tomaka Sep 17, 2020
9bf2f2f
Fix db initialization for light client (#7130)
arkpar Sep 17, 2020
fa2fbe7
WeightInfo for Identity Pallet (#7107)
shawntabrizi Sep 17, 2020
fbf617e
Make sure we update the `Cargo.lock` in the polkadot companion (#7135)
bkchr Sep 17, 2020
5a2c400
Tracing for wasm with bridging to native (#6916)
gnunicorn Sep 18, 2020
b7a3b2b
Pallet Indices (#7137)
shawntabrizi Sep 18, 2020
f8ee0e1
pow: replace the thread-base mining loop with a future-based mining w…
sorpaas Sep 18, 2020
ecaf240
Bounties (#5715)
xlc Sep 18, 2020
6b1d600
Update SS58 configuration for Bifrost (#7142)
Dengjianping Sep 18, 2020
7cc397f
Prometheus metrics for RPC calls (#7088)
grbIzl Sep 18, 2020
98a6a8c
WeightInfo for Scheduler (#7138)
shawntabrizi Sep 18, 2020
f75c540
grandpa-rpc: use FinalityProofProvider to check finality for rpc (#6215)
octol Sep 18, 2020
3a0cb2a
Make it compile
bkchr Sep 18, 2020
a837031
Rework the implementation and decrease the peer reputation on invalid
bkchr Sep 18, 2020
a0f3789
Implement limits for block announce validation
bkchr Sep 24, 2020
20c00ea
Merge remote-tracking branch 'origin/master' into bkchr-async-sync
bkchr Sep 24, 2020
3f977ab
Remove accidentally added file
bkchr Sep 25, 2020
e352e41
Merge remote-tracking branch 'origin/master' into bkchr-async-sync
bkchr Sep 25, 2020
f7f0873
Apply suggestions from code review
bkchr Sep 29, 2020
4f252be
Rename `BlockAnnounceResult` to `PollBlockAnnounceValidation`
bkchr Sep 29, 2020
e08cbcf
Always return result using the internal future
bkchr Sep 29, 2020
41aa638
Merge remote-tracking branch 'origin/master' into bkchr-async-sync
bkchr Sep 30, 2020
2b18234
Move the polling and make sure all validation futures are registered
bkchr Sep 30, 2020
047f80c
Ignore the future in the legacy stuff
bkchr Sep 30, 2020
9cbfc19
Merge remote-tracking branch 'origin/master' into bkchr-async-sync
bkchr Sep 30, 2020
8ea6c88
Remove leftover stuff
bkchr Sep 30, 2020
8e5a00a
Update client/network/src/protocol/sync.rs
bkchr Sep 30, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 31 additions & 15 deletions client/network/src/protocol.rs
Original file line number Diff line number Diff line change
Expand Up @@ -572,10 +572,11 @@ impl<B: BlockT, H: ExHashT> Protocol<B, H> {
self.context_data.peers.iter().map(|(id, peer)| (id, &peer.info))
}

pub fn on_custom_message(
fn on_custom_message(
&mut self,
who: PeerId,
data: BytesMut,
cx: &mut std::task::Context,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Passing a context to an otherwise synchronous function gives the impression that this function is similar to a poll function that is repeatedly called. But on_custom_message is only called ... well on custom messages. I would much rather keep polling logic like poll_block_announce_validation within functions that adhere to the above convention.

Copy link
Member Author

@bkchr bkchr Sep 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But that doesn't work, I would need to call poll_block_announce_validation as well in on_custom_message. So, either I revert my latest commit and do it as it was done before, or we keep this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to poll after on_custom_message or we don't register the future in the task.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I am suggesting in #6958 (comment) is to ignore the fact that the future is not registered with the task for block announcements from the legacy substream for the following two reasons:

  1. Protocol:poll is polled regularly and includes a poll_block_announce_validation(cx) at the very beginning.

  2. This concerns block announcements send via the legacy substream only which (a) is not used much and (b) is going away soon.

As far as I can see this is a reasonable trade-off. What do you think @bkchr?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

) -> CustomMessageOutcome<B> {
let message = match <Message<B> as Decode>::decode(&mut &data[..]) {
Ok(message) => message,
Expand All @@ -600,7 +601,7 @@ impl<B: BlockT, H: ExHashT> Protocol<B, H> {
GenericMessage::Status(_) =>
debug!(target: "sub-libp2p", "Received unexpected Status"),
GenericMessage::BlockAnnounce(announce) =>
self.push_block_announce_validation(who.clone(), announce),
return self.push_block_announce_validation(who.clone(), announce, cx),
GenericMessage::Transactions(m) =>
self.on_transactions(who, m),
GenericMessage::BlockResponse(_) =>
Expand Down Expand Up @@ -1168,11 +1169,18 @@ impl<B: BlockT, H: ExHashT> Protocol<B, H> {
/// called later to check for finished validations. The result of the validation
/// needs to be passed to [`Protocol::process_block_announce_validation_result`]
/// to finish the processing.
///
/// This function ensures that the block announce validation future is polled
/// at least once so that the task is woken up when the future is ready.
///
/// Returns the message outcome from polling and processing a potential finished
/// block announce validation.
fn push_block_announce_validation(
&mut self,
who: PeerId,
announce: BlockAnnounce<B::Header>,
) {
cx: &mut std::task::Context,
) -> CustomMessageOutcome<B> {
let hash = announce.header.hash();

if let Some(ref mut peer) = self.context_data.peers.get_mut(&who) {
Expand All @@ -1185,6 +1193,14 @@ impl<B: BlockT, H: ExHashT> Protocol<B, H> {
};

self.sync.push_block_announce_validation(who, hash, announce, is_best);

// Make sure that the newly added block announce validation future was
// polled once to be registered in the task.
if let Poll::Ready(res) = self.sync.poll_block_announce_validation(cx) {
self.process_block_announce_validation_result(res)
} else {
CustomMessageOutcome::None
}
}

/// Process the result of the block announce validation.
Expand Down Expand Up @@ -1581,6 +1597,15 @@ impl<B: BlockT, H: ExHashT> NetworkBehaviour for Protocol<B, H> {
warn!(target: "sub-libp2p", "Inconsistent state, no peers for pending transaction!");
}
}

// Check if there is any block announcement validation finished.
while let Poll::Ready(result) = self.sync.poll_block_announce_validation(cx) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

match self.process_block_announce_validation_result(result) {
CustomMessageOutcome::None => {},
outcome => self.pending_messages.push_back(outcome),
}
}

if let Some(message) = self.pending_messages.pop_front() {
return Poll::Ready(NetworkBehaviourAction::GenerateEvent(message));
}
Expand Down Expand Up @@ -1656,7 +1681,7 @@ impl<B: BlockT, H: ExHashT> NetworkBehaviour for Protocol<B, H> {
self.on_peer_disconnected(peer_id)
},
GenericProtoOut::LegacyMessage { peer_id, message } =>
self.on_custom_message(peer_id, message),
self.on_custom_message(peer_id, message, cx),
GenericProtoOut::Notification { peer_id, protocol_name, message } =>
match self.legacy_equiv_by_name.get(&protocol_name) {
Some(Fallback::Consensus(engine_id)) => {
Expand All @@ -1675,12 +1700,11 @@ impl<B: BlockT, H: ExHashT> NetworkBehaviour for Protocol<B, H> {
}
Some(Fallback::BlockAnnounce) => {
if let Ok(announce) = message::BlockAnnounce::decode(&mut message.as_ref()) {
self.push_block_announce_validation(peer_id, announce);
self.push_block_announce_validation(peer_id, announce, cx)
} else {
warn!(target: "sub-libp2p", "Failed to decode block announce");
CustomMessageOutcome::None
}

CustomMessageOutcome::None
}
None => {
debug!(target: "sub-libp2p", "Received notification from unknown protocol {:?}", protocol_name);
Expand All @@ -1690,14 +1714,6 @@ impl<B: BlockT, H: ExHashT> NetworkBehaviour for Protocol<B, H> {
};

if let CustomMessageOutcome::None = outcome {
// Check if there is any block announcement validation finished.
while let Poll::Ready(result) = self.sync.poll_block_announce_validation(cx) {
match self.process_block_announce_validation_result(result) {
CustomMessageOutcome::None => continue,
outcome => return Poll::Ready(NetworkBehaviourAction::GenerateEvent(outcome))
}
}

Poll::Pending
} else {
Poll::Ready(NetworkBehaviourAction::GenerateEvent(outcome))
Expand Down