feat: sqlite backend #108 #158
Conversation
1. Define a `NodeStorage` trait (or type alias) that bundles `RawOperationRepo + IndexedOperationRepo + IndexerStateRepo + DltCursorRepo + Clone + Send + Sync`.
2. Move the existing `PostgresDb` implementation into `lib/node-storage/src/backend/postgres.rs`; re-export via `pub use backend::{PostgresDb, SqliteDb, NodeStorageBackend};`.
3. Introduce `SqliteDb` built on `sqlx::SqlitePool` and `lazybe::db::sqlite::SqliteDbCtx`; ensure every trait impl mirrors the Postgres logic.
4. Add a backend selector enum plus factory (`NodeStorageBackend::connect(DbBackendConfig) -> Result<NodeStorage>`) to hide engine-specific wiring.
Signed-off-by: Yurii Shynbuiev <yurii.shynbuiev@iohk.io>
1. Split migrations into `lib/node-storage/migrations/postgres` (existing SQL) and `lib/node-storage/migrations/sqlite` (new scripts with equivalent schema, foreign keys, and views).
2. Replace `sqlx::migrate!("./migrations")` with backend-specific paths.
3. Update `sqlfluff` and any formatting/tooling hooks to lint only Postgres SQL; add a lightweight formatter (or document manual steps) for SQLite files.
4. Add CI scripts/tests that run `sqlx migrate run` for both backends and compare normalized schemas to detect divergence early.
Signed-off-by: Yurii Shynbuiev <yurii.shynbuiev@iohk.io>
1. Enable the `sqlite` feature for `sqlx`, `sea-query`, and `lazybe` in the workspace; gate it behind a new Cargo feature (e.g., `sqlite-backend`) if binary size becomes a concern. 2. Extend the dev shell (`nix/devShells/development.nix`) with `libsqlite3`/`sqlite` and ensure `sqlx-cli` can talk to SQLite files. 3. Document/update `cargo sqlx prepare` steps so developers can regenerate offline metadata for both engines if needed. Signed-off-by: Yurii Shynbuiev <yurii.shynbuiev@iohk.io>
1. Extend `DbArgs` with `#[arg(long, env = "NPRISM_DB_BACKEND", value_enum)] pub backend: DbBackend`, defaulting to `Postgres`. 2. Derive the SQLite default URL when `backend == Sqlite` (per-platform app-data path that includes the network name) but still allow overriding via `--db-url`. 3. Update `init_database`, workers, and services to accept the abstracted storage type instead of `PostgresDb` directly. 4. Ensure multi-component setups (DLT sync/index workers, resolver service) clone the backend safely; verify Send/Sync bounds. Signed-off-by: Yurii Shynbuiev <yurii.shynbuiev@iohk.io>
1. Add `just` targets for SQLite usage (`just db-init-sqlite`, etc.) and clarify that `db-up`/`db-dump` remain Postgres-specific. 2. Extend README + docs with: - instructions for choosing a backend - default SQLite file locations & permissions (0700 parent dir) - limitations (single-writer, WAL requirement, performance expectations) 3. Update Docker/Dhall configs to keep using Postgres but mention the embedded option for local/dev scenarios. Signed-off-by: Yurii Shynbuiev <yurii.shynbuiev@iohk.io>
1. Implement backend-agnostic backup/export APIs (serialize DID operations + cursor state), plus matching import logic; wire them into new `just db-backup` / `db-restore` recipes. 2. Add integration tests that: - run migrations + smoke tests against temporary Postgres (test container) and SQLite (temp file) - verify that exported data from one backend can be imported into the other. 3. Ensure `just test` (or a new CI lane) exercises both backends, at least for the critical repository tests. Signed-off-by: Yurii Shynbuiev <yurii.shynbuiev@iohk.io>
Signed-off-by: Yurii Shynbuiev <yurii.shynbuiev@iohk.io>
Signed-off-by: Yurii Shynbuiev <yurii.shynbuiev@iohk.io>
Signed-off-by: Yurii Shynbuiev <yurii.shynbuiev@iohk.io>
…test`, and `just e2e::run` are green) Signed-off-by: Yurii Shynbuiev <yurii.shynbuiev@iohk.io>
…heck all important staff Signed-off-by: Yurii Shynbuiev <yurii.shynbuiev@iohk.io>
Signed-off-by: Yurii Shynbuiev <yurii.shynbuiev@iohk.io>
Signed-off-by: Yurii Shynbuiev <yurii.shynbuiev@iohk.io>
… run script Signed-off-by: Yurii Shynbuiev <yurii.shynbuiev@iohk.io>
Signed-off-by: Yurii Shynbuiev <yurii.shynbuiev@iohk.io>
Signed-off-by: Yurii Shynbuiev <yurii.shynbuiev@iohk.io>
Signed-off-by: Yurii Shynbuiev <yurii.shynbuiev@iohk.io>
Signed-off-by: Yurii Shynbuiev <yurii.shynbuiev@iohk.io>
patextreme
left a comment
patextreme
left a comment
There was a problem hiding this comment.
Thank you @yshyn-iohk , approved with minor suggestions. mostly around cleanup and database CLI options. :)
There was a problem hiding this comment.
This has recently been added to .gitignore as this is generated from tailwind cli. We can safely remove it.
| /// Database backend to use. | ||
| #[arg(long, env = "NPRISM_DB_BACKEND", default_value = "postgres")] | ||
| pub db_backend: DbBackend, |
There was a problem hiding this comment.
Would it make more sense for user to just use db_url? It should already contain the prefix postgres://... and sqlite://...
There was a problem hiding this comment.
I thought about this as well.
Theoretically, it should work with PostgreSQL and SQLite backends, but if something else is used, we might have some difficulties defining it.
Also, separating the definition of the backend from its configuration provides more clarity and less room for error.
| @@ -0,0 +1,46 @@ | |||
| # SQLite Backend Support — Implementation Plan | |||
There was a problem hiding this comment.
Better to avoid commiting AI temporary plan
There was a problem hiding this comment.
I want to keep a directory of implementation plans to track which decisions were made.
I will align it with the actual implementation plan to reduce confusion.
There was a problem hiding this comment.
Ok, I would propose to archive or turn it into live docs with paper trail. Keep environment clean for future AI 😄. Raw feature implementation plan is a stale doc.
For inspiration https://github.com/Fission-AI/OpenSpec/tree/main/openspec/changes/archive
| /// Database URL (e.g. postgres://user:pass@host:5432/db) | ||
| #[arg(long, env = "NPRISM_DB_URL")] | ||
| pub db_url: String, | ||
| pub db_url: Option<String>, |
There was a problem hiding this comment.
I think it would be more transaparent for user to have db_url required, but sensible default value available (like sqlite::memory: for example) . This will also be visible in the CLI help message. Now we have logic to conditionally generate sqlite location based on network. It may be surprising for some users.
There was a problem hiding this comment.
Let me think about it. Probably, you are right.
Contributes to #108
Summary
Storage backend architecture
NodeStorageBackendso binaries can select Postgres or SQLite via--db-backend.backend/postgres.rs,backend/sqlite.rs) with equal trait coverage.lib/node-storage/migrations/{postgres,sqlite}) and wiredsqlx migrateaccordingly.just db-init-sqlite/db-clean-sqliteplus detailed docs on SQLite defaults, file locations, and limitations.Tooling & e2e
sqlx-cli,libsqlite3, and guidance for regeneratingsqlx-data.jsonper backend.dev,dev-sqlite,ci,ci-sqlite);just e2e::runiterates across all of them.just e2e::docker-publish-local) now enable the SQLite feature by default.full-check.sh: runscargo clean,just format,cargo build --all-features,just build-config,just test, docker builds, fulljust e2e::run, and a SQLite-specific smoke test.Documentation & cleanup
full-check.sh.data/directory (SQLite DB files) is now ignored (.gitignore) and no longer committed.Testing
nix develop -c ./full-check.sh(covers formatting, build, unit tests, docker build/load, and all e2e stacks)