Skip to content

Remove connection from database provider #1715

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 21 commits into from
May 28, 2024

Conversation

Alenar
Copy link
Collaborator

@Alenar Alenar commented May 27, 2024

Content

This PR includes a refactor of the database API used to query entities from the database.

As of today we used the Provider trait to encapsulate a entity query, this trait come with a major issue: it needs to borrow a connection to the database, making it difficult to return a iterator to a query result.
This PR replace this trait with a new one: Query. Basically a query is what was a provider but instead of owning a reference to a database connection it own a WhereCondition that will be applied when this query is run.

This means that a query implementation can't run itself like the provider could, instead you use an extension method directly on the sql connection: fetch. This method take a query as a parameter and nothing more since it own the where condition to apply

When we doesn't need the iterator returned by fetch there's two new other extension method on the db connection that are available:

  • fetch_one : return the first result of the iterator if any
  • fetch_and_collect : that collect the iterator into the desired container (like a iterator.collect do).

Query design philosophy

All implementation have been made using the following model

pub struct GetStakePoolQuery {
    condition: WhereCondition,
}

impl Query for GetStakePoolQuery {
    type Entity = StakePool;

    fn filters(&self) -> WhereCondition {
        self.condition.clone()
    }

    fn get_definition(&self, condition: &str) -> String { [...] }
}

This is at creation, and by the query itself, that the condition is defined:

impl GetStakePoolQuery {
    pub fn by_epoch(epoch: Epoch) -> StdResult<Self> {
        let condition = WhereCondition::new("epoch = ?*", vec![Value::Integer(epoch.try_into()?)]);

        Ok(Self { condition })
    }
}

Caveat: This system make it more difficult to chain conditions outside of the query (like in the repository). This is only visible in the changes to the OpenMessageRepository and its associated 'Get' queries but it may be cumbersome if we want to multiply such uses.

Usage

Before:

let provider = GetCardanoTransactionProvider::new(&self.connection);
let filters = provider.get_transaction_between_blocks_condition(range);
let transactions = provider.find(filters)?;

After:

let transactions = self
        .connection
        .fetch(GetCardanoTransactionQuery::between_blocks(range))?;

Pre-submit checklist

  • Branch
    • Tests are provided (if possible)
    • Crates versions are updated (if relevant)
    • CHANGELOG file is updated (if relevant)
    • Commit sequence broadly makes sense
    • Key commits have useful messages
  • PR
    • No clippy warnings in the CI
    • Self-reviewed the diff
    • Useful pull request description
    • Reviewer requested

Comments

Caveat:

  • Before we could define helpers methods on the providers to execute the query and pre-process the result, this is no longer possible (that why some panic are pulled up from those, now suppressed, methods to the repositories).
  • The Query::get_definition method could lose its condition parameter and pull it up directly fromQuery::filter.
    But this would mean cloning the stored definition twice (one to get the filter sql expression needed and one to get the condition values in fetch).
    We may have to rethink the WhereCondition if we want to address that issue.

Issue(s)

Closes #1711

@Alenar Alenar requested review from sfauvel, jpraynaud and dlachaume May 27, 2024 13:22
Copy link

github-actions bot commented May 27, 2024

Test Results

    3 files  ±0     43 suites  ±0   8m 35s ⏱️ +10s
  992 tests +1    992 ✅ +1  0 💤 ±0  0 ❌ ±0 
1 090 runs  +1  1 090 ✅ +1  0 💤 ±0  0 ❌ ±0 

Results for commit c3a60e6. ± Comparison against base commit 8b66e45.

This pull request removes 28 and adds 29 tests. Note that renamed tests count towards both.
mithril-aggregator ‑ database::provider::certificate::get_certificate::tests::test_get_certificate_records
mithril-aggregator ‑ database::provider::certificate::insert_certificate::tests::test_insert_certificate_record
mithril-aggregator ‑ database::provider::certificate::insert_certificate::tests::test_insert_many_certificates_records
mithril-aggregator ‑ database::provider::epoch_setting::delete_epoch_setting::tests::test_delete
mithril-aggregator ‑ database::provider::epoch_setting::delete_epoch_setting::tests::test_prune
mithril-aggregator ‑ database::provider::epoch_setting::get_epoch_setting::tests::test_get_epoch_settings
mithril-aggregator ‑ database::provider::epoch_setting::update_epoch_setting::tests::test_update_epoch_setting
mithril-aggregator ‑ database::provider::signed_entity::get_signed_entity::tests::test_get_signed_entity_records
mithril-aggregator ‑ database::provider::signed_entity::insert_signed_entity::tests::test_insert_signed_entity_record
mithril-aggregator ‑ database::provider::signer::get_signer::tests::test_get_signer_records
…
mithril-aggregator ‑ database::query::certificate::get_certificate::tests::test_get_all_certificate_records
mithril-aggregator ‑ database::query::certificate::get_certificate::tests::test_get_certificate_records_by_epoch
mithril-aggregator ‑ database::query::certificate::insert_certificate::tests::test_insert_certificate_record
mithril-aggregator ‑ database::query::certificate::insert_certificate::tests::test_insert_many_certificates_records
mithril-aggregator ‑ database::query::epoch_setting::delete_epoch_setting::tests::test_delete_below_threshold
mithril-aggregator ‑ database::query::epoch_setting::delete_epoch_setting::tests::test_delete_by_epoch
mithril-aggregator ‑ database::query::epoch_setting::get_epoch_setting::tests::test_get_epoch_settings
mithril-aggregator ‑ database::query::epoch_setting::update_epoch_setting::tests::test_update_epoch_setting
mithril-aggregator ‑ database::query::signed_entity::get_signed_entity::tests::test_get_signed_entity_records
mithril-aggregator ‑ database::query::signed_entity::insert_signed_entity::tests::test_insert_signed_entity_record
…

♻️ This comment has been updated with latest results.

@Alenar Alenar temporarily deployed to testing-preview May 27, 2024 13:29 — with GitHub Actions Inactive
@Alenar Alenar temporarily deployed to testing-sanchonet May 27, 2024 13:29 — with GitHub Actions Inactive
@Alenar Alenar force-pushed the djo/1711/remove-connexion-from-provider branch 2 times, most recently from 0dbe949 to 750ef3a Compare May 27, 2024 15:45
@Alenar Alenar temporarily deployed to testing-preview May 27, 2024 15:52 — with GitHub Actions Inactive
@Alenar Alenar temporarily deployed to testing-sanchonet May 27, 2024 15:52 — with GitHub Actions Inactive
Copy link
Member

@jpraynaud jpraynaud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀

I just left few suggestions and comments below.

fn fetch<Q: Query>(&self, query: Q) -> StdResult<EntityCursor<Q::Entity>>;

/// Fetch the first entity from the database returned using the given query.
fn fetch_one<Q: Query>(&self, query: Q) -> StdResult<Option<Q::Entity>> {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another naming could be: fetch_first / fetch_all

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree for fetch_first.
For fetch_all it would introduce confusion with a query that is parametrized to fetch all row in a table (ie: connection.fetch_all(GetTransaction::all())), instead I just simplified from fetch_and_collect to just fetch_collect.

Alenar added 15 commits May 28, 2024 14:37
Basically a `Query` a revamp provider without a reference to a
SQLite connection.
Instead of asking to implement a method that returns a connection it ask
to implement a method that return the `WhereCondition` to apply when
running the query.

The actual execution of the query is done by the `fetch` extension method
directly on the connection, allowing the following syntax:
```
let cursor =
connection.fetch(GetCardanoTransactionQuery::by_transaction_hash("hash"))?;
```

Also define two more extension methods to help when running sql queries:
* `fetch_one`: run `fetch` and call next one time, returning the result.
  Usefull when you only need one entity or when running CRUD queries
  (that are executed only after next is call on the cursor at least
  once).
* `fetch_and_collect`: run `fetch` and collect it's result in the
  desired containers as a classic collect does. Usefull when you want
  directly the result and don't care about the iterator.
And rename the `provider` module to `query`
Alenar added 5 commits May 28, 2024 14:37
As all code have been adapted to the new 'query' system.
* Rename `UpdateDatabaseVersionQuery::save` to `one` following the
  existing naming convention.
* Simplify `DatabaseVersionChecker::create_table_if_not_exists` by using
  `connection.query_single_cell`.
* `fetch_one` to `fetch_first`
* `fetch_and_collect` to `fetch_collect`
@Alenar Alenar force-pushed the djo/1711/remove-connexion-from-provider branch from 8d7772c to 0af77af Compare May 28, 2024 12:37
* Mithril-aggregator from `0.5.10` to `0.5.11`
* Mithril-signer from `0.2.136` to `0.2.138`
* Mithril-persistence from `0.1.13` to `0.2.0`
@Alenar Alenar temporarily deployed to testing-preview May 28, 2024 12:52 — with GitHub Actions Inactive
@Alenar Alenar temporarily deployed to testing-sanchonet May 28, 2024 12:52 — with GitHub Actions Inactive
@Alenar Alenar merged commit 635ba54 into main May 28, 2024
40 of 41 checks passed
@Alenar Alenar deleted the djo/1711/remove-connexion-from-provider branch May 28, 2024 13:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Remove connections coupling with providers in database
4 participants