Skip to content

Optimize memory usage of signer for Cardano transactions #1904

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 13 commits into from
Aug 29, 2024
Merged

Optimize memory usage of signer for Cardano transactions #1904

merged 13 commits into from
Aug 29, 2024

Conversation

jpraynaud
Copy link
Member

@jpraynaud jpraynaud commented Aug 27, 2024 •
edited
Loading

Content

This PR includes the optimization of the memory usage of the signer when signing Cardano transactions:

  • The capability of the Merkle tree MKTree to support multiple MKTreeStorer implementations for data storage.
  • The implementation of a signing specific Merkle store which is dedicated for the signer and with optimized memory usage.
  • The benchmarks of the new signer Merkle store.
  • Also some enhancements of the benchmarks of the Merkle tree.

Signer footprint on the mainnet

The signer memory usage (Resident memory) is ~-67% less.

Screenshot from 2024-08-27 18-52-00

Benchmark

The Merkle tree creation takes ~+50% longer than with the original in memory store.

  • Merkle tree creation for 100,000 leaves
    pdf
  • Merkle tree creation for 1,000,000 leaves
    pdf
  • Merkle tree creation for 10,000,000 leaves
    pdf

Pre-submit checklist

  • Branch
    • Tests are provided (if possible)
    • Crates versions are updated (if relevant)
    • CHANGELOG file is updated (if relevant)
    • Commit sequence broadly makes sense
    • Key commits have useful messages
  • PR
    • No clippy warnings in the CI
    • Self-reviewed the diff
    • Useful pull request description
    • Reviewer requested

Issue(s)

Closes #1903

@jpraynaud jpraynaud self-assigned this Aug 27, 2024
@jpraynaud jpraynaud force-pushed the jpraynaud/1903-handle-multiple-merkle-tree-storage-backend branch from 98a4f17 to d0da026 Compare August 27, 2024 17:16
@jpraynaud jpraynaud temporarily deployed to testing-sanchonet August 27, 2024 17:28 — with GitHub Actions Inactive
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 27, 2024 •
edited
Loading

Test Results

    4 files  ±0     53 suites  ±0   9m 34s ⏱️ +12s
1 244 tests +1  1 244 ✅ +1  0 💤 ±0  0 ❌ ±0 
1 455 runs  +1  1 455 ✅ +1  0 💤 ±0  0 ❌ ±0 

Results for commit a897e21. ± Comparison against base commit 3d75d3f.

♻️ This comment has been updated with latest results.

@jpraynaud jpraynaud temporarily deployed to testing-sanchonet August 27, 2024 18:11 — with GitHub Actions Inactive
@jpraynaud jpraynaud requested review from Alenar and sfauvel August 28, 2024 09:28
sfauvel
sfauvel approved these changes Aug 28, 2024
View reviewed changes
Copy link
Collaborator

@sfauvel sfauvel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
I just see a design problem but without an obvious solution.

@jpraynaud
perf: enhance Merkle tree benchmarks
d24be7c 
Enhance their meaning by benchmarking the proof creation/verification:
- without measuring the time needed to create the Merkle tree itself
- by benchmarking appropriate number of leaves to prove (instead of all the leaves).
@jpraynaud
perf: add benchmark for MKTreeStoreSqlite in signer
29c9c72
Alenar
Alenar reviewed Aug 28, 2024
View reviewed changes
@jpraynaud jpraynaud force-pushed the jpraynaud/1903-handle-multiple-merkle-tree-storage-backend branch from d0da026 to 6922ac6 Compare August 28, 2024 14:15
@jpraynaud jpraynaud temporarily deployed to testing-sanchonet August 28, 2024 14:24 — with GitHub Actions Inactive
@jpraynaud
chore: bump crates versions
a897e21 
- 'mithril-persistence' from '0.2.23' to '0.2.24'
- 'mithril-aggregator' from '0.5.55' to '0.5.56'
- 'mithril-common' from '0.4.45' to '0.4.46'
- 'mithril-signer' from '0.2.173' to '0.2.174'.
@jpraynaud jpraynaud marked this pull request as ready for review August 28, 2024 16:58
@jpraynaud jpraynaud temporarily deployed to testing-sanchonet August 28, 2024 17:11 — with GitHub Actions Inactive
@jpraynaud jpraynaud merged commit 8c3a9b0 into main Aug 29, 2024
40 checks passed
@jpraynaud jpraynaud deleted the jpraynaud/1903-handle-multiple-merkle-tree-storage-backend branch August 29, 2024 07:25
@jpraynaud jpraynaud mentioned this pull request Aug 29, 2024
Merged
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Alenar Alenar Alenar approved these changes

@sfauvel sfauvel sfauvel approved these changes

Assignees

@jpraynaud jpraynaud

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Optimize memory usage of signer for Cardano transactions
3 participants
@jpraynaud @Alenar @sfauvel