Persist ntx state

[Mirko-von-Leipzig]

I think we can simplify our ntx component by adding a Sqlite database to store the current state. This is desired in the long term to reduce store IO in any case, but I actually think this will simplify our state management dramatically.

Our data use already follows a relational pattern which will map naturally to sql database queries.

Proposal

The database will hold the latest committed network account states and all unconsumed network notes. In addition, the database will also contain all uncommitted network account deltas and notes/nullifiers aka it will also track inflight network stuff.

Whether committed and inflight state should be in separate tables is tbd, and depends largely on how complex queries/joins become.

We should also store the latest chain tip header.

Start up

On startup the ntx syncs its local database from the canonical state in the store. This would comprise of:

• Purging all local inflight state left-over from the last run.
• Sync from local chain tip to store chain tip.
• We can now start the mempool subscription (and ensure our chain tip matches the mempool one).

Run time

The database would have a single writer only - the main co-ordinator thread. Each mempool event received it writes the changes to the database.

• Transaction received: add new inflight state (account delta, note nullified, note created)
• Transaction reverted: revert the delta
• Block committed: mark data as committed. Nullified notes can be pruned (or kept if we want to have a bit of history), update chain tip

Instead of passing the mempool event on, we now just need to notify actor accounts if they should wake up (if they're asleep).

Actors

Actors have a read-only database connection. They can query for available notes and decide on behaviour based on that.

When a transaction is emitted successfully, the actor should sleep until the transaction event was received (actor will be notified by writer and can check the database for the transaction/account update).

Schema

This will require a bit of design. We have committed notes & account state. And then inflight transactions linked to notes and account deltas.

[Mirko-von-Leipzig]

cc @bobbinth @sergerad @SantiagoPittella for feedback.

I think our ntx state management needs some clean-up (@SantiagoPittella has already started on this), but I think we may as well jump straight to the end-game? I also think it simplifies matters ito code weirdly enough. Because sql/relational DB models our current multiple hashmap looksups much better.

[SantiagoPittella]

Sounds good, sounds like it would help debuggability. Have this questions:
On restart, would inflight transactions that were submitted but not confirmed be lost?
If multiple NTX builders run, should they share the DB? Or is single-instance assumed? Would they need also an instance of the store? This one goes in line with the discussion (or issue?) about letting others spin up their ntxs

[Mirko-von-Leipzig]

On restart, would inflight transactions that were submitted but not confirmed be lost?

Correct; these (and their associated data) need to be deleted on start-up since we can't know what state they're in or what happened since our last run. This is what I meant with

• Purging all local inflight state left-over from the last run.

If multiple NTX builders run, should they share the DB? Or is single-instance assumed? Would they need also an instance of the store? This one goes in line with the discussion (or issue?) about letting others spin up their ntxs

We haven't really discussed this in any detail; but my thinking is:

• lets assume single instance for now
• if we want to do multiple, then I think its sensible to shard them so that they don't overlap
  • e.g. shard by account ID, or
  • have specialized instances for particular types of accounts
  • but ideally each account would be handled exclusively by one instance
• this means they still have an isolated database each

[bobbinth]

I agree that this is the end-state design and overall, it has a number of benefits:

• It simplifies querying the data during transaction execution (e.g., we could have a single "long-lived" DataStore per actor - or maybe even a single DataStore for the whole NTB).
• It reduces the load on the store.
• It makes it relatively easy to shard NTBs by accounts, in case we run into scalability issues sooner than expected.

The main downside, in my mind, was additional complexity. But if we think that additional complexity is not that big (or especially if it is lower than the alternative), then we can go directly to this end-state design.

Whether committed and inflight state should be in separate tables is tbd, and depends largely on how complex queries/joins become.

My intuition is that they may need to be in separate tables, or may that the database even just holds the committed states, keeping uncommitted states in memory.

[Mirko-von-Leipzig]

My intuition is that they may need to be in separate tables, or may that the database even just holds the committed states, keeping uncommitted states in memory.

Mixing database + in-memory is painful because actions can't be atomic. I'd much prefer having it all in database then the database can handle atomicity & proper transactions.

[SantiagoPittella]

• Do we prefer to have separate tables like committed_accounts + inflight_acc_deltas, committed_notes + inflight_notes with simpler queries, or fewer tables with a bit more complex queries?
• Should we make the local database of the component configurable via NtxBuilderConfig? Default to a path relative to the node's data directory?

Regarding an implementation plan, we can split this in tasks like:

1. First we migrate the schema and models. This changes are easy to check since everything shouldl still run as expected.
2. Then we can change the NTX builder component management migration.
3. And end up refactoring the component's DataStore implementation (NtxDataStore) to read committed account/note data from local DB

[Mirko-von-Leipzig]

Do we prefer to have separate tables like committed_accounts + inflight_acc_deltas, committed_notes + inflight_notes with simpler queries, or fewer tables with a bit more complex queries?

Simpler until we can demonstrate a performance problem. In particular, in this case we know this will always be a local/embedded database I think we should worry much less about "round trips" to the database.

Should we make the local database of the component configurable via NtxBuilderConfig? Default to a path relative to the node's data directory?

Yes, similar to the validator database in #1614

Regarding an implementation plan, we can split this in tasks like:

I think that works, though maybe a rough sketch of the database tables versus the data store requirements upfront would be a good idea. The final step I agree should be the actor actually switching to the database.

[bobbinth]

There is one potential complication that I didn't consider before: the NTB state would also need to keep track of all relevant Merkle authentication paths for its accounts (these include storage map and account vault paths). In the store we have the InnerForest component that keeps track of these. Currently, these live in memory, but in the near future we'll transition to using SmtForest struct that can be persisted (most likely using RocksDB backend).

So, to transfer account storage to NTB, we'd also need have equivalent structure there - i.e., right now we could use the InnerForest (and make sure we update it accordingly), and later on, we'd also migrate it to a persistent data structure for Merkle paths as well.