Network Transaction Builder: Account-level Parallelization

[sergerad]

Context

We currently store state in a builder-wide, shared State struct. All actively processed accounts are shared:

    /// Tracks all network accounts with inflight state.
    ///
    /// This is network account deltas, network notes and their nullifiers.
    accounts: HashMap<NetworkAccountPrefix, AccountState>,

The AccountState instances hold account updates and inflight notes:

/// Tracks the state of a network account and its notes.
pub struct AccountState {
    /// The committed account state, if any.
    ///
    /// Its possible this is `None` if the account creation transaction is still inflight.
    committed: Option<Account>,

    /// Inflight account updates in chronological order.
    inflight: VecDeque<Account>,

    /// Unconsumed notes of this account.
    available_notes: HashMap<Nullifier, InflightNetworkNote>,

    /// Notes which have been consumed by transactions that are still inflight.
    nullified_notes: HashMap<Nullifier, InflightNetworkNote>,
}

We have the option to split up network account workloads because network accounts are independent of each other. The main benefit of this is to allow for specialized per-account logic. But we expect to also have wins in terms of performance, complexity, and further functionality as we continue to develop the NTX builder.

High Level Design

Core Components

1. Account Actor (AccountActor)

Each network account prefix gets a dedicated long-running async task that:

• Maintains isolated account state (inflight notes, nullifiers)
• Processes transactions sequentially (no internal concurrency)
• Handles any account-specific configuration and logic
• Captures panics in order to inform coordinator of impending shutdown

struct AccountActor {
    account_prefix: NetworkAccountPrefix,
    state: AccountState,
    coordinator_rx: mpsc::UnboundedReceiver<CoordinatorMessage>,
    coordinator_tx: mpsc::UnboundedSender<ActorMessage>,
    ntx_context: NtxContext,
    rate_limiter: Arc<Semaphore>,
    config: AccountConfig,
}

pub struct ActorHandle {
    account_prefix: NetworkAccountPrefix,
    coordinator_tx: mpsc::UnboundedSender<CoordinatorMessage>,
    join_handle: tokio::task::JoinHandle<()>,
}

2. Central Coordinator (NtxCoordinator or just existing NetworkTransactionBuilder)

A single coordinator task that:

• Subscribes to mempool events from the block producer
• Routes events to appropriate account actors via channels
• Manages actor lifecycle (spawn, monitor, restart on failure)
• Implements global rate limiting
• Maintains a registry of active account actors
• Handles account discovery and cleanup

This could either be embedded directly into the current NetworkTransactionBuilder::serve_new() loop, or made as a separate struct that is integrated a bit more loosely.

// Within NetworkTransactionBuilder::serve_new()
let mut actor_registry = HashMap::<NetworkAccountPrefix, ActorHandle>::new();
let rate_limiter = Arc::new(Semaphore::new(MAX_IN_PROGRESS_TXS));

loop {
    tokio::select! {
        event = mempool_events.try_next() => {
            // Route to appropriate actors or spawn new ones
        },
        actor_msg = actor_message_rx.recv() => {
            // Handle actor completion/failure messages
        },
        _tick = interval.tick() => {
            // Trigger periodic processing across all actors
        },
    }
}

3. Coordinator-Actor Communication

• Mempool Events: Coordinator fans out events to relevant actors via bounded channels
• Control Messages: Coordinator sends lifecycle and configuration messages to actors
• Status Reporting: Actors report health and completion status back to coordinator
• Global Coordination: Semaphore-based permits for global rate limiting

enum CoordinatorMessage {
    MempoolEvent(MempoolEvent),
    ConfigUpdate(AccountConfig),
    Shutdown,
}

enum ActorMessage {
    TransactionCompleted { 
        account: NetworkAccountPrefix,
        tx_id: TransactionId, 
        result: Result<()>, NtxError> 
    },
    ActorFailed { 
        account: NetworkAccountPrefix,
        error: NtxError 
    },
    ActorReady { 
        account: NetworkAccountPrefix 
    },
}

Key Flows

Actor Lifecycle Management

Actor management is handled by the NtxCoordinator.

New Transaction → New Network Account → Spawn Actor
Actor Crash → Capture and Send Message to Coordinator → Respawn Actor
Account Deleted → Shutdown Actor

Event Flow

Typical transaction processing would occur as follows:

Block Producer → Coordinator → Account Actor → Coordinator

The second actor message would indicate success or failure.

[Mirko-von-Leipzig]

Thanks! This looks like it should work!

I think we can simplify the messaging by using the channel and task results to communicate failure and shutdowns.

CoordinatorMessage

I think this can become only a MempoolEvent. Shutdown can be communicated by ending the channel, and we currently don't have any config we want to update.

Actor feedback channel

I think we can simplify this by removing the actor feedback channel.

Actor failure is already covered by the actor task itself ending. And the coordinator will already need to handle actor panics so joining the task is required regardless. To that end I'd suggest splitting out ActorHandle because you'll want all tasks in a JoinSet to have inside the select!. This would cover handling actor panicks and errors.

The tx completion message is also not something that is actionable for the coordinator - at most they can log it; but we can do that in the actor.

This results in ActorHandle becoming something like

tasks: JoinSet<Result<(), ActorError>>,
actor_tasks: HashMap<NetworkAccountPrefix, tokio::task::Id>,
actor_sender: HashMap<NetworkAccountPrefix, Sender<MempoolEvent>>,

Actor startup and resets

This is something I spent some time thinking on because it gets a bit complicated.

The simplest imo is something like this:

1. The coordinator spawns the task, including the current local chain tip N.
2. The coordinator sends all events that are still inflight aka uncommitted at N.
3. The task init
   1. Loads account state and notes as at N from the store.
   2. Processes all pending events
4. Task now enters work loop i.e. processes new events, creates txs etc.

A complication is step (2) because this implies the coordinator should retain all uncommitted events so that it can replay them when an actor is restarted. And so it can submit inflight notes for a new account created after the notes were. I've done something similar in the mempool event subscriber here. We can consider making this a follow-up issue.

Another problem is (3i) as its not yet possible to load account state at a specific block height, so there is a bit of a race condition, but technically the actor could wait until the next block committed event comes along to align things. This is being solved on the store side to allow getting the state at a specific recent block so I think we just proceed.

[SantiagoPittella]

Since we already have parallel actors, i'm not sure of what's the scope of this issue. We can simplify actor shutdown signaling by removing CancellationToken and using channel close instead. Is that the scope? 🤔

[sergerad]

I think we can close this one now and move the sub-issues elsewhere.