Stream `GetNetworkAccountIds` response

[SantiagoPittella]

As Mirko suggested in #1452 , we can try to stream the response of the mentioned method.

I would also be interested in an experiment where we change the proto to return a stream of AccountId instead. Internally, the store would manage the stream by paginating internally (which no longer needs to be block bound, but can use rowid or any other internally consistent row token). e.g. request a page from db, stream the page, request next page, stream it etc.

Client can recover from a broken stream by initializing stream with Option<AccountId>, the last account ID received. Though we could also just restart from scratch since its an internal only API.

Originally posted by @Mirko-von-Leipzig in #1452 (review)

[bobbinth]

I'm assuming that the store keeps the lock on the DB while streaming, and that's OK because this endpoint is only used during NTB startup, right?

[Mirko-von-Leipzig]

I'm assuming that the store keeps the lock on the DB while streaming, and that's OK because this endpoint is only used during NTB startup, right?

No; store would load pages from the store with a new connection for each page and then stream those out:

1. Load page
2. Stream out page
3. Load next page and repeat

We could just stick to pages; I just want to see what the stream version would look like because I think it could simplify both ends.

---

Tangentially, I also want to remove the coupling between the components at startup so that we can properly split out the ntx builder as a component.

Currently the block-producer startup is blocked until the ntx builder has fetched all network accounts from the store. This ensures that no network account is missed.

Instead, the ntx builder spawns a separate task which streams the network accounts from the store. The accounts are then fed back to the main ntx builder task which then uses it to spawn actors etc as it wishes. This means committed accounts from the store will arrive concurrently with new mempool events, and there will be some overlap - but this doesn't matter.

[Mirko-von-Leipzig]

@SantiagoPittella maybe the tangential part I mentioned above would be a good next issue? If so, I can split it out and we can talk through any questions, with some input from @sergerad?

[SantiagoPittella]

@SantiagoPittella maybe the tangential part I mentioned above would be a good next issue?

LGTM 👍🏼 , I can tackle it.