Batch sync state

[hackaugusto]

Currently a client needs to perform sync calls on a loop, roughly like this:

chain_tip = sync()
while local_block != chain_tip:
    chain_tip = sync()

On the pseudo code above, each call to sync is a gRPC request, that fetchs the data of a single updating block ¹. The loop may reutilize a connection, but it still has to perform many round trips (the number of roundtrips depends on how many updating blocks exist). This can be very inefficient for some use cases, for example, an off-chain app, that uses notes as an event mechanism.

For the scenario above, it make sense to have an api that instead of requiring a loop, return a batch of updates:

chain_tip = batch_sync()  # returns multiple updates, instead of a single one
while local_block != chain_tip:
    chain_tip = batch_sync()

We probably still need the loop, because we should have an upper bound on the amount of work the server does per request, lets say produce a response with up to 5MB of data. But this interface should allow for much more efficient syncing on some use cases.

Footnotes

1. By "updating block", I mean a block that contains a change to an account/nullifier of interest, or a note that matches the user's tags. So a block that matches the search criteria and is returned in the response ↩

[hackaugusto]

A straightforward implementation of the above would be to just collect existing sync updates into a list, the server would do something like this:

reply = empty()
while start != chain_tip && has_space(reply):
    let (data, start) = next_block(start)
    reply.append(data)
return reply

Another approach, which would result in a more efficient encoding format is to have a separate response for this request type. Here we can:

• Return a single list of nullifier updates. If inclusion proofs are part of the request, then those can be done w.r.t. to a single block.
• Notes can be collected into a single list.
• Account updates could be collapsed, meaning that if the same account transitions from s1 -> s2 -> s3, it is possible to return only the state s3. IMO this should be configurable, since a user may want to see when the other transitions happened (edit: the current behavior is the overwrite, without configuration). Account updates are collapsed because we only have the latest hash saved in the DB, it is not possible to return intermediary values.
• Mmr updates can be merged. For example, if we have two consecutive blocks in the reply, block n1 should include the path, but for block n2 we can reutilize the blocks of n1

[bobbinth]

With the naive approach we could probably just redefine the current sync response to be something like:

message SyncStateResponse {
    repeat SyncStatePage pages = 1;
}

message SyncStatePage {
    uint32 chain_tip = 1;
    block_header.BlockHeader block_header = 2;
    mmr.MmrDelta mmr_delta = 3;
    repeated AccountHashUpdate accounts = 5;
    repeated note.NoteSyncRecord notes = 6;
    repeated NullifierUpdate nullifiers = 7;
}

The changes to the both the client and the node would be minimal.

As you say, we could compress things more with a specialized response format - but I wonder if it is worth it. I'd expect that the bulk of the savings would come from having a single MMR delta, but even that shouldn't be that significant.

To make this work, we will need a way to estimate the size of each "page" - but I'm assuming that shouldn't be too difficult to do.

A separate, but related point is that we should also limit the size of each "page". Currently, if the client doesn't have any notes of interest and hasn't synced up for a long time, a single page may span a long period of time (e.g., a year) and thus could result in many MB of data. We could limit the span of each page to be at most 2 weeks or something like that - but not sure if there is a better way to handle it.

[hackaugusto]

To make this work, we will need a way to estimate the size of each "page" - but I'm assuming that shouldn't be too difficult to do.

Actually, we could make the whole API streamable, instead of changing the response type, we can change the rpc definition to:

rpc SyncState(requests.SyncStateRequest) returns (stream responses.SyncStateResponse) {}

That would remove the need for estimation, remove the delay from the roundtrip (because data is pushed and not pulled), and simplify the client code a bit (no need for nested loops), and if we are not going with the Mmr compression, seems like the best alternative.