Public note creation during `ntx` execution

[partylikeits1983]

Background

Currently, it is difficult to create public notes as outputs during the execution of a network transaction. This is because, during the execution of a network transaction, the network transaction builder has no knowledge of the public notes that may be generated as a result of the transaction. For a public note to be created, its public details must exist in the AdviceMap.

Problem

The primary problem I want to address is how a note-script-based AMM would operate on Miden. Consider an AMM account that is a Network account. Users create network SWAP notes (for simplicity, assume these SWAP notes can only be consumed by the AMM network account). When the network transaction builder processes these SWAP notes, it executes them against the AMM network account.

In this example, the AMM account consumes the network SWAP note and asserts that the x * y = k formula (or whichever underlying formula is used) holds after fulfilling the SWAP note by creating an output P2ID note.

Currently, this process works only if the output P2ID note is private. The transaction fails when trying to create a public P2ID note because the network transaction builder does not add the public note details to the AdviceMap. This means that either the network note or the network account must populate the AdviceMap with the required data for creating the P2ID note.

Given the storage limitations of notes, it is likely that the network account will need to store the public note data on-chain in order to create public output notes.

Short-Term Solution

A short-term solution would be for network accounts wishing to achieve such functionality would need to write custom logic which would populate the AdviceMap in MASM with the output note it intends to create. Ideally, we should add a helper function for this to the miden namespace.

As Bobbin noted, this issue might alternatively be solved by creating an on-chain NoteScript registry. However, building a NoteScript registry may be overkill in the short term. Although, with such a registry, network note scripts that want to create public output notes could use FPI to read the necessary data, such as the NoteScript, which would be stored in the registry as a vector of felts. This approach might introduce another complication, as it is unclear whether the network transaction builder can utilize FPI.

Concrete Short-Term Tasks

Tasks that would help enable public note creation during the execution of a network note transaction:

• Create a generalized procedure in the miden namespace that inserts public note details into the AdviceMap.

Later on:

• Once implemented, modify the current SWAP script to allow it to be filled as a network note.
• Generalize the standard SWAP note to be able to be filled as a Public, Private, or Network note.

---

Context from Slack thread:

Alex:
Just thinking about how a network note could output another note during its execution.

As far as I can tell to do so, the note needs to do the following:

Key	Value
output_note.recipient()	output_note.digest().format_for_advice()
output_note.inputs().commitment()	output_note.inputs().format_for_advice()
output_note.script().root()	output_note.script()

This way, when the transaction note calls exec.tx::create_note all the public note details are already in the AdviceMap

The question is though, where could the vectors of Felts be stored in the note, for the recipient, inputs, and script? Note inputs is only 128 Felt s. The Vec<Felt> for a P2ID note script is 362 felts.

I don't think it would be possible to write the note script as felts directly in the note as this would change the note script itself (chicken or the egg problem).

This is where public output notes are inserted into the AdviceMap: https://github.com/0xMiden/miden-base/blob/ba5bed3e31d205040a6a23735aadca8f608ddaec/crates/miden-objects/src/transaction/tx_args.rs#L175

---

Bobbin's reponse:

We could provide a helper procedure in the miden namespace - something like load_recipient. This procedure could take the following inputs:

• SERIAL_NUM
• inputs_ptr and num_inputs.
• SCRIPT_ROOT

This procedure then would need to insert all relevant entries into the advice map based on these.
The trickiest part here would be to figure out how to map the SCRIPT_ROOT to the actual script. A few thoughts on this:

• For well-known notes (e.g., P2ID) that should be straight-forward.
  If the script root is for the note that is currently being executed, that should be pretty simple as well.
• The main complexity would be to load the script for other non-standard notes. Here, we could have several options:
  • Encode the script into the auxiliary note data - something similar to what I described in `NoteTag` requirements and simplification #1449 (reply in thread)
  • Create a concept of on-chain note script registry.

The registry approach is probably the most robust. User would be able to execute transactions to add scripts to the onchain registry. And then these scripts would be similar to the well-known note scripts. I don't have a good idea yet of how this note script registry would work. A few questions to think through:

Can we store this in an account somehow or do we need to create a new "logical entity" in the chain for it.
How would users add entries into this registry?

What kind of validation would need to be done before an entry can be added to the registry? (e.g., we may want to make sure that the key is in-fact the MAST root of the underlying script - but doing this may not be trivial).

---

Somewhat related to #1049

cc-ing @Mirko-von-Leipzig re network transactions

[bobbinth]

Thank you for writing this up in such detail! Now that I think about this a bit more, there may be a relatively simple solution to this. Specifically, we may not need to build a separate note script registry because we could use an existing set of notes for such registry.

In the node, we already have a note_scripts table which is basically a map of note_script_root to serialized note scripts. So, we just need to make the data from this table accessible to the transaction executor. It is tricky to do now, but once we have lazy loading in the transaction host, this should be relatively straight forward. The way it could work would be as follows:

First, we'd need to add a new method to the DataStore interface. It could look something like this:

pub trait DataStore {
    ...

    fn get_note_script(
        &self,
        script_root: Digest
    ) -> impl FutureMaybeSend<Result<NoteScript, DataStoreError>>;
}

Then, in the on_note_after_created() event handler, the output note builder would use this method to retrieve note scripts from the DataStore rather than looking for them in the advice map. We could also add an optimization here that for well-known notes, we don't even need to go to the data store and can just get the script from the static variables.

Then, in the node, the implementations of the DataStore interface would need to read data from the note_scripts table (and maybe have some kind of LRU cache as well to make sure we don't go to the store for frequently used not scripts).

From the user's standpoint the process should be relatively simple:

• To add a note script to this "registry" they'd need to execute one transaction that created a public note with the script they care about (this needs to be done once per script and could be done by the app developer).
• From that point on, any network transaction would be able to create a public note with this script without any extra work.

---

The changes that are need to make this work:

1. We need to extend the DataStore interface with the get_note_script() method and update the transaction host/executor to work with it.
   a. We may need to add something like load_recipient procedure to the miden library let users load relevant recipient data into the advice provider.
2. On the node side, we'll need to update the DataStore implementation in the network transaction builder to return note scripts from the note_scripts table. cc @Mirko-von-Leipzig and @sergerad
   a. We probably also want to re-visit the logic of how the data gets inserted into this table. Specifically, we need to make sure we always insert the most "complete" version of a script since there are many ways to get the same script root by "hiding" parts of the script.
3. We probably will need to make adjustments on the client for how expected output notes are handled (e.g., we won't need to load note script data into the advice provider any more but may need to insert it into the client's local database so that it can be returned via the data store interface later). cc @igamigo

[bobbinth]

We need to extend the DataStore interface with the get_note_script() method and update the transaction host/executor to work with it.

This is now described in #1972.

We may need to add something like load_recipient procedure to the miden library let users load relevant recipient data into the advice provider.

This is the main remaining piece that we still need to figure out. The main issue is that output_note::create procedure takes RECIPIENT as a parameter, and TransactionHost::on_note_after_created() expects to find the data that hashes to RECIPIENT in the advice map. Basically, we are looking for the following entry:

RECIPIENT |-> [INPUTS_COMMITMENT, SCRIPT_ROOT, SERIAL_NUM]

Currently, we expect this data to be loaded into the advice provider before transaction execution starts (e.g., via the "expected output note recipients" included in transaction requests) - but this doesn't work in network transaction settings.

What we need, is to have an option to insert this entry programmatically (i.e., via MASM). Then, it would be up to note/tx scripts to provide this data.

I think there are broadly two ways to do it.

Approach 1

We could add something like note::load_recipient procedure, which could look something like this:

#! Inputs:  [inputs_ptr, num_inputs, SCRIPT_ROOT, SERIAL_NUM]
#! Outputs: [RECIPIENT]
export.load_recipient
    ...
end

This will build the recipient and insert the required data into the advice map.

We could also just modify the current note::build_recipient procedure (which has exactly the same signature) insert the recipient info into the advice map after it builds it. That's probably a better approach than introducing a new procedure.

Then, user code would first call note::build_recipient and then call output_note::create and the host would be able to reconstruct the note.

Approach 2

Another approach would be to have dedicated note creation procedures in the output_notes module. For example:

#! This is the same as the current `create` procedure
#!
#! Inputs:  [tag, aux, note_type, execution_hint, RECIPIENT]
#! Outputs: [note_idx]
export.create_partial_note
    ...
end

#! Inputs:  [tag, aux, note_type, execution_hint, inputs_ptr, num_inputs, SCRIPT_ROOT, SERIAL_NUM]
#! Outputs: [note_idx]
export.create_full_note
    ...
end

Then, user code can decide which procedure to call.

---

I probably favor the first approach because it is a bit more flexible. Specifically, the code that creates the note doesn't need to know how the data gets into the advice provider - so, we could still manually load the advice provider with required data (as we do now) or we could populate the data programatically.

[mmagician]

From the user's standpoint the process should be relatively simple:

• To add a note script to this "registry" they'd need to execute one transaction that created a public note with the script they care about (this needs to be done once per script and could be done by the app developer).

Continuing the excellent thread by @partylikeits1983 on Slack:
I think this could lead to a particularly obscure DevEx if not abstracted well. Specifically, if Note A directed at a ntx account were to create a previously-unseen note, the transaction that creates Note A would succeed from the users' point of view. However, it would later fail at the ntx transaction builder, with the details (logs) only available to us on the node, making it near impossible to debug for the users/developers.

A good abstraction on the client that handles the scenario of ntx transactions creating public notes would look like:

1. Client detects that a note created by the user is for a ntx account (presumably by the NoteTag prefix?)
2. Client dry-runs this transaction, and sees that it would create a public output note.*
3. Client queries the node to see if this note's script is available in the registry.

If it is available, proceed as normal.
If it is not available, the client effectively needs to do something equivalent to a POST /add_note_script_to_registry request to the node. However, due to spam concerns, this needs gatekeeping and is likely best achieved at the protocol level. Namely, as mentioned above, the client would have to "execute one transaction that created a public note with the script they care about".
One optimization here: I don't think this needs to be a separate transaction: we can extend the transaction script to output the desired note (TransactionRequestBuilder::own_output_notes) and just create one transaction.

I'm not a huge fan of creating an output note just for the purpose of registering its script. For one, it's an anti-pattern for using notes in Miden. Then there might be unforeseen security implications we might need to think about: can such an "empty" note be exploited?
While there are mechanisms that let users add note scripts to the node's registry (e.g. API-key - based), atm I can't convince myself of a better solution than what was originally suggested^, but I wanted to at least mention that there might be alternatives in case others have better ideas for populating the note script registry outside the protocol.

---

*How the client dry-runs a transaction where the network account consumes the note is not implemented AFAIK. It's still unclear to me how this would be done, but likely along the lines of running the ntx transaction builder locally.

[bobbinth]

I think there are 3 separate issues here:

1. How can a client detect which not scripts would be needed for successful execution of future network transactions.
2. How can we ensure that clients don't submit transactions which could create notes that can't be consumed in future network transactions.
3. How should the client add note scripts to the "script registry".

I think the first point is the concern of the client and there could be multiple strategies there. In some cases, the required note scripts could be known statically, in other cases we may need to determine them dynamically. We may also be able to use TransactionRequest struct to communicate this info. For example, we could either check the existing "expected future notes" field or maybe add "expected future scripts" field so that the creator of the transaction request could specify which note scripts need to be on-chain to successfully execute this and future transactions.

The second point should probably be handled by the client as well. Another option is to delegate this to the node by including something like "required public scripts" field in ProvenTransaction - but I think handling this on the client is cleaner.

The third point is the protocol concern. Here, I still think that it is better to use the existing note architecture to keep track of the public note script registry (rather than creating a separate structure for this). We could consider creating a special note to make this more explicit - e.g., something like "script registration" note. This would have some benefits (e.g., the node would know that only these types of notes should be considered when adding scripts to the public registry) - but it also require more thinking to make sure we handle all the edge cases correctly. So, it is not clear if such special note types are worth it.

[igamigo]

3. We probably will need to make adjustments on the client for how expected output notes are handled (e.g., we won't need to load note script data into the advice provider any more but may need to insert it into the client's local database so that it can be returned via the data store interface later). cc @igamigo

I think we currently preload scripts as part of TransactionArgs::extend_output_note_recipients(). If we drop that call, we'd have to change the API here as well unless I'm misunderstanding something.

[bobbinth]

I think we currently preload scripts as part of TransactionArgs::extend_output_note_recipients(). If we drop that call, we'd have to change the API here as well unless I'm misunderstanding something.

With the current implementation we are using DataStore::get_note_script() as a "backup" - i.e., we first check if the script data is in the advice provider, and if not, try to get it from the DataStore. So, I believe this should work with the client as is right now, but does create opportunities for refactoring in the future.