Improve shielded sync (backport #4785)

[mergify]

Describe your changes

• Build commitment tree and witness map locally
  • Uses leanbridgetree crate for this purpose
  • If the number of blocks to sync exceeds a certain threshold, we race with the local approach, by fetching the commitment tree, witness map and notes index in the background
    • Whatever finishes first is the data we store in the shielded wallet (which should have been identical, regardless of being built locally or fetched from the MASP indexer)
• Fix block index not being properly used
  • This is the major source of the speed-up in this PR
• Optimize the layout of the shielded wallet
  • Removed unnecessary fields
• Only fetch block index after a threshold of $N$ blocks (where it makes sense latency wise to wait for that network query vs. fetching $N$ potentially empty blocks)

Checklist before merging

• If this PR has some consensus breaking changes, I added the corresponding breaking:: labels
  • This will require 2 reviewers to approve the changes
• If this PR requires changes to the docs or specs, a corresponding PR is opened in the namada-docs repo
  • Relevant PR if applies:
• If this PR affects services such as namada-indexer or namada-masp-indexer, a corresponding PR is opened in that repo
  • Relevant PR if applies:

---

This is an automatic backport of pull request #4785 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of f7ef28f has failed:

On branch mergify/bp/maint-201.0/pr-4785
Your branch is ahead of 'origin/maint-201.0' by 2 commits.
  (use "git push" to publish your local commits)

You are currently cherry-picking commit f7ef28fd4.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   crates/shielded_token/src/masp.rs

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   crates/shielded_token/src/masp/shielded_sync/dispatcher.rs
	both modified:   crates/shielded_token/src/masp/shielded_wallet.rs
	both modified:   crates/shielded_token/src/masp/wallet_migrations.rs

Cherry-pick of a3f44a9 has failed:

On branch mergify/bp/maint-201.0/pr-4785
Your branch is ahead of 'origin/maint-201.0' by 3 commits.
  (use "git push" to publish your local commits)

You are currently cherry-picking commit a3f44a9c0.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   crates/shielded_token/src/masp.rs

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   crates/shielded_token/src/masp/shielded_wallet.rs

Cherry-pick of 5adae71 has failed:

On branch mergify/bp/maint-201.0/pr-4785
Your branch is ahead of 'origin/maint-201.0' by 8 commits.
  (use "git push" to publish your local commits)

You are currently cherry-picking commit 5adae714b.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   crates/shielded_token/src/masp/shielded_sync/dispatcher.rs

no changes added to commit (use "git add" and/or "git commit -a")

Cherry-pick of 7c7c745 has failed:

On branch mergify/bp/maint-201.0/pr-4785
Your branch is ahead of 'origin/maint-201.0' by 11 commits.
  (use "git push" to publish your local commits)

You are currently cherry-picking commit 7c7c74501.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   crates/shielded_token/src/masp/wallet_migrations.rs

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   crates/shielded_token/src/masp/shielded_wallet.rs

Cherry-pick of b69a0ae has failed:

On branch mergify/bp/maint-201.0/pr-4785
Your branch is ahead of 'origin/maint-201.0' by 12 commits.
  (use "git push" to publish your local commits)

You are currently cherry-picking commit b69a0ae8e.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   crates/shielded_token/src/masp.rs
	modified:   crates/shielded_token/src/masp/shielded_sync/dispatcher.rs
	modified:   crates/shielded_token/src/masp/test_utils.rs
	modified:   crates/shielded_token/src/masp/wallet_migrations.rs

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   crates/shielded_token/src/masp/shielded_wallet.rs

Cherry-pick of 8348a34 has failed:

On branch mergify/bp/maint-201.0/pr-4785
Your branch is ahead of 'origin/maint-201.0' by 14 commits.
  (use "git push" to publish your local commits)

You are currently cherry-picking commit 8348a34d2.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   crates/shielded_token/src/masp/shielded_wallet.rs

no changes added to commit (use "git add" and/or "git commit -a")

Cherry-pick of 8ce76f7 has failed:

On branch mergify/bp/maint-201.0/pr-4785
Your branch is ahead of 'origin/maint-201.0' by 19 commits.
  (use "git push" to publish your local commits)

You are currently cherry-picking commit 8ce76f7fc.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   Cargo.lock
	modified:   Cargo.toml
	modified:   crates/shielded_token/Cargo.toml
	modified:   crates/shielded_token/src/masp/test_utils.rs
	modified:   crates/shielded_token/src/masp/wallet_migrations.rs
	modified:   wasm/Cargo.lock
	modified:   wasm_for_tests/Cargo.lock

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   crates/shielded_token/src/masp/shielded_wallet.rs

Cherry-pick of d99fa4c has failed:

On branch mergify/bp/maint-201.0/pr-4785
Your branch is ahead of 'origin/maint-201.0' by 22 commits.
  (use "git push" to publish your local commits)

You are currently cherry-picking commit d99fa4cb9.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   crates/shielded_token/src/masp/shielded_sync/utils.rs
	modified:   crates/shielded_token/src/masp/shielded_wallet.rs
	modified:   crates/shielded_token/src/masp/wallet_migrations.rs

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   crates/shielded_token/src/masp/shielded_sync/dispatcher.rs

Cherry-pick of 4147507 has failed:

On branch mergify/bp/maint-201.0/pr-4785
Your branch is ahead of 'origin/maint-201.0' by 32 commits.
  (use "git push" to publish your local commits)

You are currently cherry-picking commit 4147507c9.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   crates/shielded_token/src/masp/shielded_wallet.rs

no changes added to commit (use "git add" and/or "git commit -a")

Cherry-pick of 01970ac has failed:

On branch mergify/bp/maint-201.0/pr-4785
Your branch is ahead of 'origin/maint-201.0' by 34 commits.
  (use "git push" to publish your local commits)

You are currently cherry-picking commit 01970ac6c.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   crates/sdk/src/masp/utilities.rs
	modified:   crates/shielded_token/src/masp/shielded_sync/utils.rs
	modified:   crates/shielded_token/src/masp/shielded_wallet.rs
	modified:   crates/shielded_token/src/masp/test_utils.rs

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   crates/shielded_token/src/masp/shielded_sync/dispatcher.rs

Cherry-pick of 43e2640 has failed:

On branch mergify/bp/maint-201.0/pr-4785
Your branch is ahead of 'origin/maint-201.0' by 40 commits.
  (use "git push" to publish your local commits)

You are currently cherry-picking commit 43e264071.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   crates/shielded_token/src/masp/shielded_sync/dispatcher.rs
	both modified:   crates/shielded_token/src/masp/shielded_wallet.rs

no changes added to commit (use "git add" and/or "git commit -a")

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[mergify]

🧪 CI Insights

Here's what we observed from your CI run for bed4fc6.

🟢 All jobs passed!

But CI Insights is watching 👀