Faucet redesign

[Mirko-von-Leipzig]

The public faucet appears to become unresponsive to mint requests.

Symptoms

While under load, the faucet

• webpage loads correctly but
• returns 504 after a while when requesting tokens

Logs indicate that proving commences, but no note is ever produced. Its unclear what is going wrong as there are no error logs being produced.

Some potential candidates

• proving is local so instance is overloaded
• errors aren't being logged
• state is desyncing from the chain state

One potential desync way is that the tx reaches the node, but is dropped, or the faucet request is canceled before the faucet state is updated.

We can't really tell without improved logs ala #811.

A potential quickfix is to avoid local state entirely. However this restricts the throughput dramatically.

Logs

Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.298279Z  INFO miden-faucet: bin/faucet/src/handlers.rs:85: Proving and submitting transaction.
Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.299412Z  INFO miden-faucet: bin/faucet/src/handlers.rs:49: Received a request, account_id: 0xcaefc58cafbbdf100000262e2218d1, is_private_note: false, asset_amount: 100
Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.300626Z  INFO prove_program: miden_prover: /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/miden-prover-0.13.0/src/lib.rs:57: new
Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.300677Z  INFO prove_program:execute_program: miden_processor: /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/miden-processor-0.13.0/src/lib.rs:125: new
Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.473044Z  INFO prove_program:execute_program: miden_processor: /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/miden-processor-0.13.0/src/lib.rs:125: close, time.busy: 172ms, time.idle: 25.4µs
Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.473095Z  INFO prove_program: miden_prover: /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/miden-prover-0.13.0/src/lib.rs:71: Generated execution trace of 71 columns and 131072 steps (49% padded) in 172 ms
Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.473168Z  INFO prove_program:build_domain: winter_prover: /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/winter-prover-0.12.2/src/lib.rs:312: new trace_length: 131072, lde_domain_size: 1048576
Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.477672Z  INFO prove_program:build_domain: winter_prover: /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/winter-prover-0.12.2/src/lib.rs:312: close, time.busy: 4.48ms, time.idle: 25.9µs trace_length: 131072, lde_domain_size: 1048576
Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.477727Z  INFO prove_program:commit_to_main_trace_segment: winter_prover: /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/winter-prover-0.12.2/src/lib.rs:497: new
Apr 29 16:02:30 ip-32-0-133-125 miden-faucet[3242459]: 2025-04-29T16:02:30.477757Z  INFO prove_program:commit_to_main_trace_segment:extend_execution_trace: winter_prover::trace::trace_lde::default: /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/winter-prover-0.12.2/src/trace/trace_lde/default/mod.rs:258: new num_cols: 71, blowup: 8

[igamigo]

Likely not desyncing from the chain because there are transactions going through and updating the faucet account (block committed ~1 minute ago). Notes seem to be created as well (example).
Probably the request queue is backed up and requests time out before getting a hold of the executor and prover.

[Mirko-von-Leipzig]

Likely not desyncing from the chain because there are transactions going through and updating the faucet account (block committed ~1 minute ago). Notes seem to be created as well (example). Probably the request queue is backed up and requests time out before getting a hold of the executor and prover.

Its because we're force restarting the faucet every 5 minutes :D

Without this it just stalls forever.

The execution and proving is also done locally afaik :/ And network is limited to 8 txs per block so txs/s are max 1.3s. Lots of potential issues.

[igamigo]

I see. I was thinking since the sync only happens at startup and transactions are going through, it must've not desynced and so the bottleneck would just be the local proving aspect.

[Mirko-von-Leipzig]

I think a redesign is required to fully fix this. There are several options, e.g. using network transactions etc.

A fundamental problem currently is that the account state update is performed as part of the request handler. So if the handler fails, the account may or may not be in sync with the onchain state. Similarly, its possible for a tx to expire and be reverted which would also cause the local and onchain state to differ.

All of this is difficult to handle because its occurring with the request handler. I propose handling this in a more central fashion with a separate thread/task dedicated to running the account. Requests will communicate with it using channels.

The account task would effectively have a queue of requests, and would process these one-by-one. This can then be trivially upgraded to aggregating multiple requests into one transaction once we have this capability.

Similarly, any transaction failure due to account state desync can be detected and a fresh resync can be done before continuing with request processing.

[bobbinth]

If I'm understanding correctly, the improvements here could be:

• Have a separate thread that receives requests, creates transactions against the faucet, and sends these transactions to the node.
• As an improvement, batch requests so that a single transaction creates multiple notes (when there is demand).

I think this will address most of the issues, but a more robust way would also be to wait until the transaction is included in a block. This way, we'd know for sure that the state was updated. This would require a bit better UI though - to show different stages (e.g., transaction submitted vs. transaction committed). And we can even add some retry logic in the future (i.e., if a transaction was submitted but didn't commit, we can retry with a new transaction).

We can build this functionality incrementally though.

[Mirko-von-Leipzig]

I'll submit a fairly large refactor PR on Monday, but to summarize my idea:

1. Monday PR
   • Refactor to a central faucet thread.
   • Its optimistic and assumes all txs land.
   • It detects desync via a tx's gRPC error code, and then
   • rolls back to a matching prior state (kept in-memory).
   • Tx queue size is capped, overflow requests are immedietely rejected, indicating faucet is too busy.
2. Batch requests
   • Easy to implement since we already have a queue. Just grab the next M requests at once.
   • Requires changing the tx script but I'm sure that's easy (for someone else).
3. Improve UI responsiveness
   • Quite a few options here but I like Server Side Events.
   • aka server pushes updates to the client via the connection.
   • Essentially a one-sided channel.
   • Server can update request status e.g.
     • Position in queue
     • Building TX, executing, proving, submitting..