Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: Shadow block recovery mechanism #5296

Merged
merged 104 commits into from
Nov 22, 2024
Merged
Changes from 1 commit
Commits
Show all changes
104 commits
Select commit Hold shift + click to select a range
4f2c74f
fix/5274: add test coverage for microforks
jcnelson Oct 7, 2024
62d2629
chore: separate load_tenure_info() into inner_load_tenure_info(), so …
jcnelson Oct 10, 2024
e274770
feat: enable shadow blocks -- blocks that are added to the chainstate…
jcnelson Oct 10, 2024
a81f6f4
chore: update staging DB to add code to insert and query shadow block…
jcnelson Oct 10, 2024
a01b36c
chore: accomodate shadow blocks when validating a tenure against a so…
jcnelson Oct 10, 2024
f6e2b8d
chore: static transaction validation test coverage for data in shadow…
jcnelson Oct 10, 2024
9f9ff79
chore: add functionality to mine shadow block tenures and build atop …
jcnelson Oct 10, 2024
f07aca5
chore: don't divide by zero when processing a shadow tenure coinbase …
jcnelson Oct 10, 2024
de26a08
chore: try to load blocks from chainstate if we have them, so we don'…
jcnelson Oct 10, 2024
345f63a
chore: when getting tenure start/end blocks, see if we have them loca…
jcnelson Oct 10, 2024
136fd0f
chore: drive downloaders from chainstate data
jcnelson Oct 10, 2024
1cc53b7
chore: load tenure data from chainstate if we have it
jcnelson Oct 10, 2024
2c954dc
chore: log the neighbor address of the remote peer when we get back a…
jcnelson Oct 10, 2024
13d7212
chore: test that shadow tenures are reported as 1's in a nakamoto inv…
jcnelson Oct 10, 2024
5cf2946
Merge branch 'develop' into fix/5285
jcnelson Oct 11, 2024
6531b2c
Merge branch 'fix/5274' into fix/5285
jcnelson Oct 11, 2024
5154771
feat: epoch 3.0 supports shadow blocks
jcnelson Oct 18, 2024
e700ffa
feat: mock coinbaes in test burnchain blocks, since block-commits tha…
jcnelson Oct 18, 2024
81699d6
feat: a block-commit mined in epoch 3.x whose parent_vtxindex is 0 is…
jcnelson Oct 18, 2024
8b44fe2
feat: count shadow tenures as prepare-phase sortitions even if they'r…
jcnelson Oct 18, 2024
9d7f894
chore: make some tenure-mining code fallible so we can test resuming …
jcnelson Oct 18, 2024
5fe06ad
chore: more debug
jcnelson Oct 18, 2024
81b17ec
feat: fix process_next_nakamoto_block() to treat shadow blocks as una…
jcnelson Oct 18, 2024
fdbec99
feat: getters for shadow blocks and shadow tenures, and also, require…
jcnelson Oct 18, 2024
4f30e51
chore: make some mining code fallible so we can handle null-miner win…
jcnelson Oct 18, 2024
18d11c9
chore: API sync
jcnelson Oct 18, 2024
0f96ead
fix: pass the tenure IDs for the tenure start/end blocks to tenure do…
jcnelson Oct 18, 2024
aba7ca9
chore: with_dbs() helper
jcnelson Oct 18, 2024
4ff1a39
chore: drop "relayed" shadow blocks, since they get passed from the d…
jcnelson Oct 18, 2024
0b0f712
chore: test coverage for downloading tenures that start/stop with a s…
jcnelson Oct 18, 2024
4511362
chore: test that shadow blocks are part of the block inventory
jcnelson Oct 18, 2024
c077cf7
chore: some minnig functions are fallible now
jcnelson Oct 18, 2024
3e575e0
chore: always use affirmation maps from now on
jcnelson Oct 18, 2024
6101b9f
Merge branch 'develop' into fix/5285
jcnelson Oct 18, 2024
1035b89
chore: fix compiler warnings
jcnelson Oct 18, 2024
acd8732
chore: cargo fmt
jcnelson Oct 18, 2024
3a46282
fix: always use affirmation maps now that mainline testnet uses bitco…
jcnelson Oct 18, 2024
bdd5ea3
chore: address @kantai's PR feedback by refactoring the shadow block …
jcnelson Oct 21, 2024
47761f2
Merge branch 'develop' into fix/5285
jcnelson Oct 21, 2024
9ad17f1
chore: print out assert failure
jcnelson Oct 21, 2024
c478e29
chore: move staging blocks' shadow code to shadow.rs
jcnelson Oct 21, 2024
b7d5d4f
chore: cargo fmt
jcnelson Oct 21, 2024
fb54e4c
chore: add unit test coverage for shadow DB methods and validation logic
jcnelson Oct 22, 2024
4dd6630
Merge branch 'develop' into fix/5285
jcnelson Oct 22, 2024
8a5e6bd
Merge branch 'develop' into fix/5285
jcnelson Oct 22, 2024
81920f1
chore: fix unit test due to vtxindex=0 not being allowed by default i…
jcnelson Oct 22, 2024
8939fd3
Merge branch 'fix/5285' of https://github.com/stacks-network/stacks-b…
jcnelson Oct 22, 2024
19044ef
Merge branch 'develop' into fix/5285
jcnelson Oct 23, 2024
1597209
chore: add nakamoto testnet pox settings
jcnelson Oct 23, 2024
fbec882
chore: expose NakamotoBlockBuilder::get_account()
jcnelson Oct 23, 2024
7e26c01
feat: add `get-account`, `get-nakamoto-tip`, `make-shadow-block`, and…
jcnelson Oct 23, 2024
dcab2b6
chore: typo
jcnelson Oct 24, 2024
8e8981e
chore: typo
jcnelson Oct 24, 2024
0865787
Merge branch 'develop' into fix/5285
jcnelson Oct 24, 2024
9bcee64
fix: use correct pox params
jcnelson Oct 24, 2024
850d00b
chore: move shadow block creation and replay into separate functions …
jcnelson Oct 24, 2024
fb391fc
chore: make function public
jcnelson Oct 24, 2024
0f29c68
chore: shadow-chainstate-repair and shadow-chainstate-patch
jcnelson Oct 24, 2024
e8da151
fix: the parent can be a shadow block and thus not have a commit
jcnelson Oct 24, 2024
4851c3c
chore: integration test
jcnelson Oct 24, 2024
8fff6bd
chore: run integration test in CI
jcnelson Oct 24, 2024
e72a266
fix: use correct pox params
jcnelson Oct 24, 2024
05ac928
Merge branch 'fix/5285' into feat/shadow-block-tooling
jcnelson Oct 24, 2024
05a99fa
chore: typo
jcnelson Oct 24, 2024
bfbfbf7
chore: cargo fmt
jcnelson Oct 24, 2024
f7a06d4
Merge branch 'develop' into fix/5285
jcnelson Oct 25, 2024
8743b0b
chore: fix compile issue
jcnelson Oct 25, 2024
84c982c
Adding stacks-signer binary to image
wileyj Oct 25, 2024
c75ed39
chore: read/write
jcnelson Oct 25, 2024
b98239d
fix: load blocks json from file
jcnelson Oct 25, 2024
4153cec
Merge branch 'feat/shadow-block-tooling' of https://github.com/stacks…
jcnelson Oct 25, 2024
ca17ede
chore: don't use docstring
jcnelson Oct 25, 2024
439abdb
fix: accomodate shadow blocks which have no sortition but do have a s…
jcnelson Oct 25, 2024
69eb7ac
fix: use a real signer to verify that we can resume mining atop shado…
jcnelson Oct 25, 2024
3430a81
fix: advance cursor before depth check
jcnelson Oct 28, 2024
1578014
chore: handle shadow block case
jcnelson Oct 28, 2024
0ace927
chore: log HTTP RPC error values at debug level
jcnelson Oct 28, 2024
7bd1ae8
chore: info when post-shadow tenures happen
jcnelson Oct 28, 2024
7341a4b
chore: cargo fmt
jcnelson Oct 28, 2024
b30c0b4
chore: halt block-processing while we process shadow blocks
jcnelson Oct 28, 2024
ab605d9
Merge branch 'develop' into fix/5285
jcnelson Nov 8, 2024
5970a57
Merge branch 'fix/5285' into feat/shadow-block-tooling
jcnelson Nov 8, 2024
5f83847
chore: search depth only increases with sortitions, not shadow blocks
jcnelson Nov 9, 2024
4a105e4
Merge branch 'develop' into fix/5285
jcnelson Nov 11, 2024
6eaf1ed
Merge branch 'develop' into fix/5285
jcnelson Nov 11, 2024
ad01a76
Merge branch 'fix/5285' into feat/shadow-block-tooling
jcnelson Nov 11, 2024
62f5130
Merge branch 'develop' into fix/5285
jcnelson Nov 12, 2024
28e1be4
chore: fix failing integration test and unstage accidentally-added file
jcnelson Nov 12, 2024
f43ede5
Merge branch 'develop' into fix/5285
jcnelson Nov 15, 2024
08365d6
chore: fix slow unit test
jcnelson Nov 15, 2024
75e409d
chore: fix test
jcnelson Nov 16, 2024
c60b569
Merge branch 'fix/5285' of https://github.com/stacks-network/stacks-b…
jcnelson Nov 16, 2024
34d658a
Merge branch 'develop' into fix/5285
jcnelson Nov 19, 2024
a99a6d1
Merge branch 'fix/5285' into feat/shadow-block-tooling
jcnelson Nov 19, 2024
eebf10d
move net::api::tests::postblock_proposal::test_try_make_response to p…
wileyj Nov 19, 2024
a7e89bc
Merge branch 'develop' into fix/5285
jcnelson Nov 19, 2024
a3cdab6
Merge branch 'fix/5285' into feat/shadow-block-tooling
jcnelson Nov 19, 2024
82c36d7
move net::api::tests::postblock_proposal::test_try_make_response to p…
wileyj Nov 19, 2024
d038752
Merge pull request #5362 from stacks-network/feat/shadow-block-tooling
wileyj Nov 20, 2024
edc8b67
Merge branch 'develop' into fix/5285
jcnelson Nov 20, 2024
212914b
Merge branch 'develop' into fix/5285
jcnelson Nov 21, 2024
6ddfd6c
Merge branch 'develop' into fix/5285
jcnelson Nov 21, 2024
a3640b4
chore: PR feedback
jcnelson Nov 21, 2024
ca5ab06
chore: fix compile error
jcnelson Nov 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
chore: when getting tenure start/end blocks, see if we have them loca…
…lly first and advance the tenure downloader state machine from that
  • Loading branch information
jcnelson committed Oct 10, 2024
commit 345f63abdf1e79bb70075ae2cf932971fa84b4cb
53 changes: 53 additions & 0 deletions stackslib/src/net/download/nakamoto/tenure_downloader.rs
Original file line number Diff line number Diff line change
Expand Up @@ -541,6 +541,59 @@ impl NakamotoTenureDownloader {
Ok(Some(request))
}

/// Advance the state of the downloader from chainstate, if possible.
/// For example, a tenure-start or tenure-end block may have been pushed to us already (or they
/// may be shadow blocks)
pub fn try_advance_from_chainstate(
&mut self,
chainstate: &StacksChainState,
) -> Result<(), NetError> {
loop {
match self.state {
NakamotoTenureDownloadState::GetTenureStartBlock(
start_block_id,
_start_request_time,
) => {
// if we have this, then load it up
let Some((tenure_start_block, _sz)) = chainstate
.nakamoto_blocks_db()
.get_nakamoto_block(&start_block_id)?
else {
break;
};
self.try_accept_tenure_start_block(tenure_start_block)?;
if let NakamotoTenureDownloadState::GetTenureStartBlock(..) = &self.state {
break;
}
}
NakamotoTenureDownloadState::GetTenureEndBlock(
end_block_id,
_start_request_time,
) => {
// if we have this, then load it up
let Some((tenure_end_block, _sz)) = chainstate
.nakamoto_blocks_db()
.get_nakamoto_block(&end_block_id)?
else {
break;
};
self.try_accept_tenure_end_block(&tenure_end_block)?;
if let NakamotoTenureDownloadState::GetTenureEndBlock(..) = &self.state {
break;
}
}
NakamotoTenureDownloadState::GetTenureBlocks(..) => {
// TODO: look at the chainstate and find out what we don't have to download
break;
}
NakamotoTenureDownloadState::Done => {
break;
}
}
}
Ok(())
}

/// Begin the next download request for this state machine. The request will be sent to the
/// data URL corresponding to self.naddr.
/// Returns Ok(true) if we sent the request, or there's already an in-flight request. The
Expand Down