Add public key deduplication migration

[fxamacker]

Updates #7573

This PR adds public key deduplication by migration (to run during spork).

The migration will deduplicate public keys to reduce the number of registers (aka payloads).

Reducing the number of payloads (and mtrie nodes that pointed to them) has benefits beyond EN memory reduction. For example, other machines and components that handle payloads can startup faster, use less memory, etc. such as databases, caches, indexes, etc.

Results using a July 2025 mainnet snapshot show:

• number of registers (aka payloads) was reduced by over 86 million registers
• EN memory reduction is TBD (this can sometimes be ~3x state size reduction)
• state size was reduced by ~21 GB
• duration of public key deduplication step is only 4.5 minutes (using m1 vm)
• max RSS of util program running migration is 890 GB RAM

Also, roughly 2 nodes from mtrie that are 96 bytes each are removed for each register removed.

Details

Account public key deduplication migration deduplicates account public keys and related data:

• Add optional public key metadata to account status register
• Store unique public keys in batch
• Store sequence number separately from public key
• etc.

This migration and these changes will reduce number of payloads from mtrie without preventing concurrent execution that uses multiple proposal keys from the same account.

Runtime Deduplication As Keys are Added (Post-Spork)

A separate PR will provide post-spork runtime deduplication (automatic deduplication as public keys are added) and support of account status format v4.

Tradeoffs

The deduplication data format balances speed, storage used, number of registers (aka payloads), etc.

As an analogy, 3rd-party programs like gzip, zstd, etc. don't use their max compression ratio by default because the tradeoff in speed won't be acceptable for most users.

NOTE: deduplication doesn't use gzip or zstd (they are good examples of tradeoffs).

Notes about running migration

When using nworkers=64, the util program running the migration had max RSS of 890 GB RAM (using a July 2025 mainnet snapshot) and the deduplication migration step took ~4.5 mins. To avoid OOM crashes and slowdowns, the OS should have extra RAM available for caching large files and running other processes, etc.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 63.41969% with 353 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...er/migrations/account_key_deduplication_encoder.go	73.78%	79 Missing and 34 partials ⚠️
.../migrations/account_key_deduplication_migration.go	59.13%	81 Missing and 13 partials ⚠️
...tions/account_key_deduplication_migration_utils.go	59.09%	30 Missing and 15 partials ⚠️
...ons/account_key_deduplication_migration_results.go	0.00%	43 Missing ⚠️
cmd/util/cmd/execution-state-extract/cmd.go	0.00%	27 Missing ⚠️
cmd/util/ledger/util/registers/registers.go	0.00%	11 Missing ⚠️
model/flow/account_encoder.go	78.57%	6 Missing and 3 partials ⚠️
...d/util/ledger/migrations/storage_used_migration.go	81.81%	5 Missing and 1 partial ⚠️
fvm/environment/accounts_status.go	91.66%	3 Missing ⚠️
utils/unittest/execution_state.go	0.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[bluesign]

quick question, does this deduplicate by account ? or is it global ? ( just curious about the technique )

[fxamacker]

quick question, does this deduplicate by account ? or is it global ? ( just curious about the technique )

Hi @bluesign, the deduplication is by account.

EDIT: I added more code comments about deduplication.

[fxamacker]

@janezpodhostnik thanks for meeting on short notice and sharing your thoughts about storage used computation (e.g., related to sequence numbers and individual sequence number registers needed to avoid blocking concurrent execution).

Commit df0775a incorporates some ideas from our meeting into the storage used computation. PTAL 🙏