Guardrails: Transaction Re-execution in the RPC

[Mirko-von-Leipzig]

We want to (short term) loosen miden's privacy guarantees while we are still actively developing in a centralized fashion. Effectively, miden would have privacy only in the network sense and not from the node operator.

Private transactions will include the transaction data when submitting to the node operator. This data will not appear on-chain (same as it is now), but would be kept separately by the node.

This would act as guardrails, and the intention is to drop this sometime before decentralization.

There are some benefits:

• less regulatory issues
• (private) account recovery
• better debugging and support
• trivial full chain history and therefore recovery from some critical failure

and some downsides:

• more work for the node - tx data must be verified, probably by re-execution, negating (some) of the benefits of private transactions
• less complete privacy

ito implementation, it makes sense to store this data completely separately from the current database. This would minimize any actual node IO with the data beyond writing it.

This will of course require changes to RPC and the client.

[bobbinth]

One of the other main benefits is that we get to "double-check" the proofs and so any potential soundness bugs in the proving system would not cause catastrophic failures.

[Mirko-von-Leipzig]

@igamigo wdyt of this as a starting issue? It touches RPC, storage and will require some integration with the client.

Might be a bit broad, but there is also no rush on this as users are unlikely to care :)

Could also be implemented in stages e.g. (a) update RPC protos & update client, (b) verify via re-execution, (c) figure out long-term storage.

[igamigo]

@igamigo wdyt of this as a starting issue? It touches RPC, storage and will require some integration with the client.

Might be a bit broad, but there is also no rush on this as users are unlikely to care :)

Could also be implemented in stages e.g. (a) update RPC protos & update client, (b) verify via re-execution, (c) figure out long-term storage.

I agree that this sounds quite broad but looks like good one. After a brief chat with Bobbin we settled on tackling #673 first (while he also moves forward with #174).
Maybe we could split this one up in more explicit tasks in the meantime to tackle next (although the stages you mention sound good to me). Depending on milestone priorities, of course.

[bobbinth]

I think from the client/RPC standpoint the main thing that needs to change is that all transactions submitted via SubmitProvenTransaction endpoint would look like "public transactions" (i.e., include the same data for private account/notes as if they were public accounts/notes). So, the two main modifications will probably be:

1. Update ProvenTransaction struct in miden-base to track all relevant data for private account/notes.
2. Update the storage system here to somehow keep this data separate from the public data, yet still make sure it is accessible accessible.

[sergerad]

What data does this refer to exactly?

Private transactions will include the transaction data when submitting to the node operator.

Are there endpoints / structs that already contain these for the purposes of understanding what is missing in the private route (RPC and storage)?

e.g.:

pub struct TransactionSummary {
    pub account_id: AccountId,
    pub block_num: BlockNumber,
    pub transaction_id: TransactionId,
}

My best guess so far is that the following would be stored with proven transactions:

1. Transaction ID
2. Account ID
3. Nullifiers
4. Output notes summary
5. State commitments

[sergerad]

Is the idea that the client will supply all the additional metadata via the RPC calls to SubmitProvenTransaction for example? Or would the node be responsible for somehow performing a mapping of proven transactions to the rest of the metadata?

My intuition is that SubmitProvenTransaction would involve some additional metadata but the rest would be mapped by the node before storing it. But maybe we want to limit the amount of client-side code we impact with this.

[bobbinth]

Is the idea that the client will supply all the additional metadata via the RPC calls to SubmitProvenTransaction for example? Or would the node be responsible for somehow performing a mapping of proven transactions to the rest of the metadata?

The additional data would have to come though the SubmitProvenTransaction endpoint (the node would not have enough info to map a proven transaction to all required data to re-execute it).

One option is to update ProvenTransaction message to look something like:

message ProvenTransaction {
    bytes transaction = 1;
    optional bytes transaction_replay = 2;
}

Where transaction_replay would basically be serialized ExecutedTransaction for now. This should allow us to re-execute the transaction to verify the result in the RPC. In the future (e.g., after the next release of the VM) we'd be able to replace ExecutedTransaction with more compact data that would enable the same functionality.

Also, I marked this field as optional not to break the client. I think the sequence of events could be:

1. First add this field and the associated functionality to re-run the transaction.
2. Then update the client to send the executed transaction to the node.
3. Then make the field required. Though, maybe we can keep it optional until we remove it entirely (though, this will be quite a bit further down the road).