Decouple ntx builder from block-producer startup #1487
Description
Summary
Decouple the ntx builder by moving account loading from start-up to a separate concurrent task.
History
The network tx builder currently has to start-up in lockstep with the block-producer. This is enforced by having the block-producer start the ntx builder and synchronicity is done via a Barrier such that both components can proceed together.
This was required initially to ensure the ntx builder and mempool have the same latest chain-tip state:
- ntx builder loads account states from the store until it reaches the chain tip
- block-producer starts
- ntx builder connects to mempool subscription
We have since refactored the ntx builder to no longer pre-load all account state from the store. Instead each account's actor loads its state separately as it starts up. This has its own issues, however it now means we can decouple the startup of these components.
What to do
Spawn a separate task in the ntx builder which loads the network accounts from the store, page-by-page. Each account is then communicated back to the main event loop via a channel. There is no need to load all accounts - accounts can be submitted as they arrive. Something to keep in mind is that if you end this task the channel receiver will close which may cause issues depending on the implementation. It may be best to simply pending() the task forever at this stage.
The event loop will now need a select! to differentiate between the above channel, or the mempool subscription. The idea with the new channel is that this notifies the event loop that this account exists and this should follow the same logic as then current event handle. This logic isn't perfect, but we'll tackle that in other work.
Subscribing to the mempool currently requires the block-height, but this requirement can be removed so long as the above task is spawned after the subscription is created.
And of course remove the barrier.
Follow-ups
#1478 can be considered as well after this. The other thing to note is that this implementation doesn't care about being lined up with block edges; only that we at some point get all committed network accounts. So we could paginate by a constant via rowid (so long as we don't delete accounts from the db) which can simplify the store impl and api.
We can also actually make the ntx builder a proper separate component after this is done. i.e. with its own CLI and so on.