change(state): Add note subtree indexes for new and existing blocks #7437

teor2345 · 2023-08-31T04:14:19Z

Motivation

This is needed to support "spend before sync" in lightwalletd.

Closes #6953.

Specifications

See https://github.com/zcash/zcash/pull/6677/files#diff-decae4be02fb8a47ab4557fe74a9cb853bdfa3ec0fa1b515c0a1e5de91f4ad0bR1458-R1475

The specs for subtrees aren't in a Zcash specification yet.

Complex Code or Requirements

The upgrade runs concurrently with writing new blocks and subtrees, but the subtree indexes they write are different.

This code runs in separate threads, so we're careful to handle cancel, drop, and shutdowns.

Rebuilding subtrees could be CPU-intensive and take a long time.

Solution

Write subtrees to the finalized state for new blocks
Upgrade existing finalized states to calculate and add subtrees
- Assert that subtree indexes are correct
- Log when each subtree is added during an upgrade, because upgrades are slow
Bump the database format version to 25.2.0
Check the subtree upgrade is valid after it has completed, and on every Zebra launch

Bug fixes:

Prevent a concurrency bug by using the correct API (it's probably impossible anyway)

Refactors:

Simplify subtree APIs
Simplify subtree upgrade code
Clarify some comments

Testing

This PR has a state validity test. There are also asserts that subtrees are added in order and continuously.

Review

This PR is needed to test the subtree RPC method in PR #7436.

Reviewer Checklist

Will the PR name make sense to users?
- Does it need extra CHANGELOG info? (new features, breaking changes, large changes)
Are the PR labels correct?
Does the code do what the ticket and PR says?
- Does it change concurrent code, unsafe code, or consensus rules?
How do you know it works? Does it have tests?

Related Work

This is a copy of PR #7350 with some fixes to make it work with the latest main branch.

zebra-state/src/service/finalized_state/zebra_db/shielded.rs

teor2345 · 2023-08-31T04:35:41Z

On testnet this upgrade takes less than 1 second:

2023-08-31T04:31:39.100268Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: marked database format as upgraded running_version=Version { major: 25, minor: 2, patch: 0 } format_upgrade_version=Version { major: 25, minor: 1, patch: 1 } disk_version=Version { major: 25, minor: 1, patch: 0 }
2023-08-31T04:31:39.296963Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: marked database format as upgraded running_version=Version { major: 25, minor: 2, patch: 0 } format_upgrade_version=Version { major: 25, minor: 2, patch: 0 } disk_version=Version { major: 25, minor: 1, patch: 1 }
2023-08-31T04:31:39.296892Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: Zebra automatically upgraded the database format to: newer_running_version=Version { major: 25, minor: 2, patch: 0 }

Edit: This is probably because there are no subtrees. On mainnet the same upgrade panics because it's missing a Sapling subtree.

zebra-chain/src/orchard/tree.rs

zebra-state/src/service/finalized_state/disk_format/upgrade/add_subtrees.rs

teor2345 · 2023-08-31T22:12:03Z

On mainnet the upgrade takes about 1 second per subtree, I'm running it overnight locally so I can test the RPC.

Here is the performance of the mainnet upgrade on my machine:

~1100 sapling subtrees: 15 minutes
~700 orchard subtrees: 5 minutes

So the total upgrade time is 20 minutes.

Here are the logs from the upgrade task:

2023-08-31T06:40:07.609922Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: trying to open older database format: launching upgrade task running_version=Version { major: 25, minor: 2, patch: 0 } disk_version=Version { major: 25, minor: 1, patch: 1 }
2023-08-31T06:40:07.610006Z  INFO zebra_state::service::finalized_state::disk_db: the open file limit is high enough for Zebra current_limit=1024 min_limit=512 ideal_limit=1024
2023-08-31T06:40:07.846572Z  INFO zebra_state::service::finalized_state::disk_db: Opened Zebra state cache at /home/dev/.cache/zebra-custom/state/v25/mainnet
2023-08-31T06:40:07.848013Z  INFO zebra_state::service::finalized_state: loaded Zebra state cache tip=Some((Height(2210287), block::Hash("0000000000ec22974f7a26b974dcbaf095615e6e8a2d5b508b09d9cc3dd3c80c")))
2023-08-31T06:40:08.016508Z  INFO zebra_state::service::finalized_state::disk_format::upgrade::add_subtrees: calculated and added sapling subtree height=Height(558822) index=0
2023-08-31T06:41:10.854368Z  INFO zebra_state::service::finalized_state::disk_format::upgrade::add_subtrees: calculated and added sapling subtree height=Height(1733780) index=100
2023-08-31T06:42:54.680556Z  INFO zebra_state::service::finalized_state::disk_format::upgrade::add_subtrees: calculated and added sapling subtree height=Height(1741676) index=200
2023-08-31T06:44:29.688948Z  INFO zebra_state::service::finalized_state::disk_format::upgrade::add_subtrees: calculated and added sapling subtree height=Height(1753551) index=300
2023-08-31T06:45:44.096756Z  INFO zebra_state::service::finalized_state::disk_format::upgrade::add_subtrees: calculated and added sapling subtree height=Height(1762369) index=400
2023-08-31T06:47:10.256812Z  INFO zebra_state::service::finalized_state::disk_format::upgrade::add_subtrees: calculated and added sapling subtree height=Height(1771447) index=500
2023-08-31T06:48:27.590899Z  INFO zebra_state::service::finalized_state::disk_format::upgrade::add_subtrees: calculated and added sapling subtree height=Height(1783656) index=600
2023-08-31T06:49:50.870770Z  INFO zebra_state::service::finalized_state::disk_format::upgrade::add_subtrees: calculated and added sapling subtree height=Height(1791579) index=700
2023-08-31T06:51:26.138087Z  INFO zebra_state::service::finalized_state::disk_format::upgrade::add_subtrees: calculated and added sapling subtree height=Height(1800361) index=800
2023-08-31T06:53:06.716403Z  INFO zebra_state::service::finalized_state::disk_format::upgrade::add_subtrees: calculated and added sapling subtree height=Height(1809496) index=900
2023-08-31T06:54:29.509025Z  INFO zebra_state::service::finalized_state::disk_format::upgrade::add_subtrees: calculated and added sapling subtree height=Height(1819275) index=1000
2023-08-31T06:55:23.494550Z  INFO zebra_state::service::finalized_state::disk_format::upgrade::add_subtrees: calculated and added sapling subtree height=Height(2118375) index=1100
2023-08-31T06:55:24.233642Z  INFO zebra_state::service::finalized_state::disk_format::upgrade::add_subtrees: calculated and added orchard subtree height=Height(1707429) index=0
2023-08-31T06:55:54.634851Z  INFO zebra_state::service::finalized_state::disk_format::upgrade::add_subtrees: calculated and added orchard subtree height=Height(1836742) index=100
2023-08-31T06:56:35.204731Z  INFO zebra_state::service::finalized_state::disk_format::upgrade::add_subtrees: calculated and added orchard subtree height=Height(1857929) index=200
2023-08-31T06:57:21.018305Z  INFO zebra_state::service::finalized_state::disk_format::upgrade::add_subtrees: calculated and added orchard subtree height=Height(1881127) index=300
2023-08-31T06:58:03.194716Z  INFO zebra_state::service::finalized_state::disk_format::upgrade::add_subtrees: calculated and added orchard subtree height=Height(1925537) index=400
2023-08-31T06:58:38.744790Z  INFO zebra_state::service::finalized_state::disk_format::upgrade::add_subtrees: calculated and added orchard subtree height=Height(1957755) index=500
2023-08-31T06:59:12.502288Z  INFO zebra_state::service::finalized_state::disk_format::upgrade::add_subtrees: calculated and added orchard subtree height=Height(2011784) index=600
2023-08-31T06:59:41.355723Z  INFO zebra_state::service::finalized_state::disk_format::upgrade::add_subtrees: calculated and added orchard subtree height=Height(2084076) index=700
2023-08-31T06:59:45.984703Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: marked database format as upgraded running_version=Version { major: 25, minor: 2, patch: 0 } format_upgrade_version=Version { major: 25, minor: 2, patch: 0 } disk_version=Version { major: 25, minor: 1, patch: 1 }
2023-08-31T06:59:45.984716Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: Zebra automatically upgraded the database format to: newer_running_version=Version { major: 25, minor: 2, patch: 0 }

zebra-chain/src/orchard/tree.rs

zebra-state/src/service/finalized_state/disk_format/upgrade/add_subtrees.rs

teor2345 · 2023-09-05T03:22:36Z

GitHub actions seems to be having some issues, I've bumped this PR up to critical priority so it merges first.

teor2345 · 2023-09-05T03:22:47Z

@Mergifyio refresh

mergify · 2023-09-05T03:22:55Z

refresh

✅ Pull request refreshed

teor2345 · 2023-09-05T03:29:35Z

@Mergifyio refresh

mergify · 2023-09-05T03:29:42Z

refresh

✅ Pull request refreshed

teor2345 · 2023-09-05T05:21:35Z

@Mergifyio refresh

mergify · 2023-09-05T05:21:41Z

refresh

✅ Pull request refreshed

teor2345 · 2023-09-05T05:22:04Z

GitHub said there was an API request issue which got resolved:
https://www.githubstatus.com/history

upbqdn · 2023-09-05T14:38:28Z

@Mergifyio refresh

mergify · 2023-09-05T14:39:01Z

refresh

✅ Pull request refreshed

…7437) * Copy the add_subtrees upgrade from the original branch * Copy the database write changes in shielded.rs from the original branch * Copy the tree API changes from the original branch * Simplify subtree APIs to avoid exposing frontiers * Fix a dead code warning by re-using existing methods * Use mpsc::Receiver<CancelFormatChange> in the subtree upgrade * Run the subtree upgrade on startup * Bump the database format version to 25.2.0 * Fix a confusing 'upgrade complete' log * Clarify some comments and error messages * Simplify prev_tree unwrap to avoid an (impossible?) concurrency bug * Use separate subtree writing functions * Use common note commitment list code * Fix subtree completion condition and add asserts * Simplify subtree API and avoid exposing Address * Fix API compatibility when Arcs are removed * Log when each subtree is added * If a format change is cancelled, don't mark the database as upgraded or do format checks * Log subtree progress about once every two minutes * Adds a state validity check for subtrees upgrade * Orchard is faster, decrease log interval * Clarify subtree index docs * Move a log to the correct location * Refactor subtree upgrade to remove duplicate inverted loop conditions * updates subtree state validity check * Add a subtree format check when there is no upgrade * Fix an off-by-one error with the final subtree check * Use error-level logs for database format checks * Skip format checks in tests that create invalid formats * fix state validity test * Add a concurrency comment to subtree by height methods * Add individual subtree state methods: reverts removing these methods in an earlier PR * fastmod "subtrees_by_index" "subtree_list_by_index_for_rpc" --------- Co-authored-by: arya2 <aryasolhi@gmail.com>

teor2345 added P-Medium ⚡ A-state Area: State / database changes C-feature Category: New features A-concurrency Area: Async code, needs extra work to make it work properly. A-compatibility Area: Compatibility with other nodes or wallets, or standard rules labels Aug 31, 2023

teor2345 requested a review from a team as a code owner August 31, 2023 04:14

teor2345 assigned arya2 Aug 31, 2023

teor2345 requested a review from a team as a code owner August 31, 2023 04:14

teor2345 requested review from upbqdn and removed request for a team August 31, 2023 04:14

teor2345 self-assigned this Aug 31, 2023

teor2345 marked this pull request as draft August 31, 2023 04:14

teor2345 mentioned this pull request Aug 31, 2023

OUTDATED: change(state): Add note subtree indexes to the database, and calculate subtrees for existing blocks #7350

Closed

6 tasks

teor2345 commented Aug 31, 2023

View reviewed changes

zebra-state/src/service/finalized_state/zebra_db/shielded.rs Outdated Show resolved Hide resolved

teor2345 mentioned this pull request Aug 31, 2023

change(state): Add state requests and support code for the z_getsubtreesbyindex RPC #7408

Merged

6 tasks

This comment was marked as outdated.

Sign in to view

teor2345 marked this pull request as ready for review August 31, 2023 06:48

oxarbitrage added the do-not-merge Tells Mergify not to merge this PR label Aug 31, 2023

arya2 force-pushed the subtree-upgrade-data-only branch from 48a31fd to fa0c50e Compare August 31, 2023 20:53

teor2345 commented Aug 31, 2023

View reviewed changes

zebra-chain/src/orchard/tree.rs Show resolved Hide resolved

teor2345 commented Aug 31, 2023

View reviewed changes

zebra-state/src/service/finalized_state/disk_format/upgrade/add_subtrees.rs Outdated Show resolved Hide resolved

teor2345 commented Aug 31, 2023

View reviewed changes

zebra-state/src/service/finalized_state/disk_format/upgrade/add_subtrees.rs Show resolved Hide resolved

teor2345 commented Aug 31, 2023

View reviewed changes

zebra-state/src/service/finalized_state/disk_format/upgrade/add_subtrees.rs Outdated Show resolved Hide resolved

arya2 previously approved these changes Aug 31, 2023

View reviewed changes

upbqdn previously approved these changes Aug 31, 2023

View reviewed changes

zebra-chain/src/orchard/tree.rs Show resolved Hide resolved

zebra-state/src/service/finalized_state/disk_format/upgrade/add_subtrees.rs Show resolved Hide resolved

arya2 dismissed stale reviews from upbqdn and themself via 43923b4 September 1, 2023 01:22

Copy the add_subtrees upgrade from the original branch

ec684a4

teor2345 added P-Critical 🚑 and removed P-Medium ⚡ labels Sep 5, 2023

mergify bot added a commit that referenced this pull request Sep 5, 2023

Merge of #7437

160bbb6

mergify bot mentioned this pull request Sep 5, 2023

merge queue: embarking main (ecd4e1f) and #7437 together #7475

Closed

mergify bot added a commit that referenced this pull request Sep 5, 2023

Merge of #7437

66fc9ec

mergify bot mentioned this pull request Sep 5, 2023

merge queue: embarking main (ecd4e1f) and [#7466 + #7437] together #7476

Closed

mergify bot added a commit that referenced this pull request Sep 5, 2023

Merge of #7437

ef8de62

mergify bot mentioned this pull request Sep 5, 2023

merge queue: embarking main (ecd4e1f) and #7437 together #7478

Closed

mergify bot added a commit that referenced this pull request Sep 5, 2023

Merge of #7437

40cb97c

mergify bot mentioned this pull request Sep 5, 2023

merge queue: embarking main (ecd4e1f) and [#7437 + #7466] together #7482

Closed

mergify bot added a commit that referenced this pull request Sep 5, 2023

Merge of #7437

6f0d54b

mergify bot mentioned this pull request Sep 5, 2023

merge queue: embarking main (ecd4e1f) and #7437 together #7483

Closed

mergify bot mentioned this pull request Sep 5, 2023

merge queue: embarking main (97b43fb) and #7455 together #7487

Closed

mergify bot added a commit that referenced this pull request Sep 5, 2023

Merge of #7437

5c64d3e

mergify bot mentioned this pull request Sep 5, 2023

merge queue: embarking main (97b43fb) and #7437 together #7488

Closed

mergify bot merged commit cc61bd5 into main Sep 5, 2023
289 checks passed

mergify bot deleted the subtree-upgrade-data-only branch September 5, 2023 16:52

arya2 mentioned this pull request Sep 5, 2023

change(docs): Add code comments to state validity test for subtree format upgrade #7496

Merged

6 tasks

teor2345 mentioned this pull request Sep 29, 2023

Zebra full sync is taking at least 50% longer over the last 3 weeks #7618

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

change(state): Add note subtree indexes for new and existing blocks #7437

change(state): Add note subtree indexes for new and existing blocks #7437

teor2345 commented Aug 31, 2023 •

edited

Loading

teor2345 commented Aug 31, 2023 •

edited

Loading

This comment was marked as outdated.

teor2345 commented Aug 31, 2023 •

edited

Loading

teor2345 commented Sep 5, 2023

teor2345 commented Sep 5, 2023

mergify bot commented Sep 5, 2023

teor2345 commented Sep 5, 2023

mergify bot commented Sep 5, 2023

teor2345 commented Sep 5, 2023

mergify bot commented Sep 5, 2023

teor2345 commented Sep 5, 2023

upbqdn commented Sep 5, 2023

mergify bot commented Sep 5, 2023

change(state): Add note subtree indexes for new and existing blocks #7437

change(state): Add note subtree indexes for new and existing blocks #7437

Conversation

teor2345 commented Aug 31, 2023 • edited Loading

Motivation

Specifications

Complex Code or Requirements

Solution

Testing

Review

Reviewer Checklist

Related Work

teor2345 commented Aug 31, 2023 • edited Loading

This comment was marked as outdated.

teor2345 commented Aug 31, 2023 • edited Loading

teor2345 commented Sep 5, 2023

teor2345 commented Sep 5, 2023

mergify bot commented Sep 5, 2023

✅ Pull request refreshed

teor2345 commented Sep 5, 2023

mergify bot commented Sep 5, 2023

✅ Pull request refreshed

teor2345 commented Sep 5, 2023

mergify bot commented Sep 5, 2023

✅ Pull request refreshed

teor2345 commented Sep 5, 2023

upbqdn commented Sep 5, 2023

mergify bot commented Sep 5, 2023

✅ Pull request refreshed

teor2345 commented Aug 31, 2023 •

edited

Loading

teor2345 commented Aug 31, 2023 •

edited

Loading

teor2345 commented Aug 31, 2023 •

edited

Loading