Improving actor design

[tindzk]

I’ve been exploring Miden’s node implementation over the past few days and like its architecture, especially that it is designed around actors.

Based on the code I have read so far, I would suggest a few changes to the current actor design. When working on other distributed systems, I found these constraints particularly effective:

• Give actors full data ownership and avoid shared references
• Using a non-async on_message function that mutates the internal state directly in response to events
• Assign each actor a unique identity (e.g. TypeId, metadata)
• Spawn actors through an actor system
• Route messages to destination actors via handles (e.g. ActorRef) through a bus

These constraints have various benefits: As actors operate only on local state, they can use &mut self and thus avoid locks. Actors with this design can be tested in isolation. Not using async enables Deterministic Simulation Testing which TigerBeetle used to find subtle bugs.

The Akka framework is worth looking at in this context. It has additional features such as network transparency and automated actor restarts, which might not be relevant for Miden at this stage.

I am currently exploring how these constraints could be applied after #1219 has been merged. Here is a rough sketch of what I have in mind: https://gist.github.com/tindzk/e53772c940ac0fb2e3e886b0431c21e3

If this is a direction you deem worth exploring, I’d be happy to spend more time on it.

[sergerad]

@bobbinth how many network accounts do we expect to have in the short term and long term?

[bobbinth]

@bobbinth how many network accounts do we expect to have in the short term and long term?

In the shorter term, I'd be surprised if we have more than 100K accounts (our current testnet has about 200K accounts total, this will increase in the future, but I think network accounts will be a minority).

In the longer term, we should expect tens of millions (though, probably not more until the very long term).

[sergerad]

Given those numbers of network accounts, it will be worth refactoring the coordinator to at least:

1. Run account actor tasks for individual events (on_event described in this issue) rather than have long-running actor tasks; and
2. Manage which actors receive each event rather than broadcasting all events to actors.

We should start that after #1317 which might involve refactor number 2 anyway.

[Mirko-von-Leipzig]

Some thoughts on how we can reduce the number of active account actors. This is largely orthogonal to whether we use an actor framework or just keep it simple with our current approach.

Some of these thoughts might be 💩

Self shut down

Let's allow accounts to shut themselves down. They can do this by simply exiting with return Ok(()). The coordinator then interprets this as a deliberate decision and does not restart the actor automatically.

Accounts could shut themselves down if they have no useful notes and haven't received an event in e.g. 5 minutes.

This technically introduces a race condition, where the actor shut downs and then an event arrives which should have kept it alive. In order to detect this, we could have the actor return the event receiver on exist i.e. return Ok(rx_events). The coordinator can then check if the receiver is empty and if it isn't, restart the actor which would mean it would reprocess things.

Blacklisting accounts

The co-ordinator can keep a hashmap of accounts that crash or return an error. Once the crash count reaches some limit they get added to a blacklist and are no longer considered. This should be fairly straight-forward to add.

Standardizing account startup

I think at the moment we have separate flows for starting an account from an event, or starting an account from the store. I think we can connect these by always first getting the account state from the store. If there is no account, then the actor waits until it receives a new account event. We can similarly use the timeout strategy here so that the actor shuts down if the new account event doesn't arrive within a few minutes after start up.

This means we have the same flow for all actors.

[SantiagoPittella]

@Mirko-von-Leipzig question here: was the unassignment an error? The GH UI is ...flaky sometimes and I'm not sure if this was an error

[Mirko-von-Leipzig]

Closing in favor of #1694