Incremental storage of Cardano transactions #1619

Alenar · 2024-04-08T09:58:21Z

Content

This PR optimize the retrieval of cardano transactions. Before each time a Cardano Transaction signed entity type was signed the whole cardano chain was parsed. Now we only parse the immutables that was not parsed before, with a parameter that allow to re-parse previous immutables in case of a previous failure.

There's several internal mithril-common api changes to handle that:

Remove TransactionStore trait: persistence is now defined by the nodes themselves.
Add TransactionsImporter trait to handle both parsing and (possibly if the node need it) persistence.
Add from_immutable parameter to TransactionParser trait: an optional lower bound, if None it tell the parser to parse everything from the start.

Pre-submit checklist

Branch
- Tests are provided (if possible)
- Crates versions are updated (if relevant)
- CHANGELOG file is updated (if relevant)
- Commit sequence broadly makes sense
- Key commits have useful messages
PR
- No clippy warnings in the CI
- Self-reviewed the diff
- Useful pull request description
- Reviewer requested
Documentation
- Update README file (if relevant)
- Update documentation website (if relevant)
- Add dev blog post (if relevant)

Comments

We simplified some signatures that were using a CardanoDbBeacon but only needed it's immutable_file_number part.

Issue(s)

Relates to #1591

…gner

As they will diverge in the future.

If data are stored in db returns it and don't parse/store anything

When data are partially stored in the database.

Before it would aggregate result from the not update db and the parser.

This allow to rescan immutables in case of a previous partial scan likely when this scan failed.

The beacon parameter wasn't clearly expressing that it was an upper bound.

Since multiple traits can define it

Since it's a sqlite limitation, it's the repository job to handle that.

github-actions · 2024-04-08T10:04:28Z

Test Results

3 files ± 0 42 suites ±0 8m 34s ⏱️ -48s
962 tests +16 962 ✅ +16 0 💤 ±0 0 ❌ ±0
1 056 runs +16 1 056 ✅ +16 0 💤 ±0 0 ❌ ±0

Results for commit ed4d46d. ± Comparison against base commit 336c070.

♻️ This comment has been updated with latest results.

mithril-aggregator/src/database/provider/cardano_transaction.rs

mithril-aggregator/src/dependency_injection/builder.rs

mithril-aggregator/src/services/cardano_transactions_importer.rs

mithril-common/src/cardano_transaction_parser.rs

mithril-aggregator/src/services/cardano_transactions_importer.rs

- Fix comments typo - Rename function name - Change `from` type to `ImmutableFileNumber` in `parse_and_store_transactions_not_imported_yet` signature

…igner nodes

jpraynaud

LGTM 👍

Alenar and others added 21 commits April 4, 2024 12:26

Extract importer code from the signable builder into a method

5c6af1e

Define TransactionImporter trait and do a first naive impl

2682947

Implement and wire a TransactionsImporter in both aggregator and si…

063f722

…gner

Move TransactionStore trait from common to signer & aggregator

ab75f90

As they will diverge in the future.

Test actual behaviour in signer TransactionStore

d13db99

Implement naive cache

4207d0c

If data are stored in db returns it and don't parse/store anything

Transaction importer now handle partial import

3f0278c

When data are partially stored in the database.

Add a test to check if database order and parsed order are consistent

cced3ac

Split get from db highest beacon & transactions in two

80adb16

Refactor import: retrieve all data from db after updating it

015ff69

Before it would aggregate result from the not update db and the parser.

Implement get_up_to to signer cardano transactions repository

a56e42b

Implement get_highest_beacon to signer cardano transactions repository

fa449e0

Add rescan offset to signer TransactionsImporter

1fd9475

This allow to rescan immutables in case of a previous partial scan likely when this scan failed.

Simplify rescan offset computation & testing

1ceb5b0

Clarify TransactionImporter::import signature

329b3ad

The beacon parameter wasn't clearly expressing that it was an upper bound.

Don't use ServiceBuilder to build the sqliteconnection in unit tests

b646831

Test the repository 'get tx up to' method instead of the trait implem

580b38b

Since multiple traits can define it

Synchronise TransactionsImporter change from signer to aggregator

fd345f4

Add log to TransactionsImporter when the database is up to date

eb5c95f

Make transaction repository handle the store in chunk

bf7d1dd

Since it's a sqlite limitation, it's the repository job to handle that.

Handle lower bound in Transactions parser

396ed9e

Alenar temporarily deployed to testing-preview April 8, 2024 10:14 — with GitHub Actions Inactive

Alenar temporarily deployed to testing-sanchonet April 8, 2024 10:14 — with GitHub Actions Inactive

Alenar requested review from jpraynaud, sfauvel and dlachaume April 8, 2024 10:21

Alenar self-assigned this Apr 8, 2024

jpraynaud reviewed Apr 8, 2024

View reviewed changes

Minor changes after review

baf3e08

- Fix comments typo - Rename function name - Change `from` type to `ImmutableFileNumber` in `parse_and_store_transactions_not_imported_yet` signature

dlachaume added 2 commits April 8, 2024 18:14

Add migrations to create index for block_number on aggregator and s…

8931a0a

…igner nodes

Reajust rescan offset from 2 to 1 immutable

2902dbf

dlachaume temporarily deployed to testing-preview April 8, 2024 16:22 — with GitHub Actions Inactive

dlachaume temporarily deployed to testing-sanchonet April 8, 2024 16:22 — with GitHub Actions Inactive

sfauvel approved these changes Apr 8, 2024

View reviewed changes

dlachaume added 2 commits April 8, 2024 18:25

Update UNRELEASED section of the CHANGELOG

5da2c23

Update crates versions

ed4d46d

jpraynaud approved these changes Apr 8, 2024

View reviewed changes

dlachaume temporarily deployed to testing-preview April 8, 2024 16:37 — with GitHub Actions Inactive

dlachaume temporarily deployed to testing-sanchonet April 8, 2024 16:37 — with GitHub Actions Inactive

Alenar merged commit b9e810f into main Apr 9, 2024
41 checks passed

Alenar deleted the ensemble/1591/incremental-storage-of-cardano-tx branch April 9, 2024 07:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Incremental storage of Cardano transactions #1619

Incremental storage of Cardano transactions #1619

Alenar commented Apr 8, 2024 •

edited by dlachaume

Loading

github-actions bot commented Apr 8, 2024 •

edited

Loading

jpraynaud left a comment

Incremental storage of Cardano transactions #1619

Incremental storage of Cardano transactions #1619

Conversation

Alenar commented Apr 8, 2024 • edited by dlachaume Loading

Content

Pre-submit checklist

Comments

Issue(s)

github-actions bot commented Apr 8, 2024 • edited Loading

Test Results

jpraynaud left a comment

Choose a reason for hiding this comment

Alenar commented Apr 8, 2024 •

edited by dlachaume

Loading

github-actions bot commented Apr 8, 2024 •

edited

Loading