As the project matures we will want to validate that Substrate works "at scale", but what does that mean beyond "create/maintain a lot of things"?
We should define primary use cases and try to define the abstract behaviors that they exhibit to enable continuously validating performance improvements over time.
Example classes of workloads:
- Burst-type workloads where an agent spins up a lot of short-lived sub-agents that do their work and die
- Polling-type workloads (actor executes a long-running operation and needs to either wake up to check status periodically or receive a message that the task is complete and process the result)
- Server-type workloads where an actor should be present to handle incoming requests
- Sandbox-type workloads where an environment is provided with a persistent, isolated state for things like developing/executing code, working on scratch files, etc.
Abstract behaviors:
-
Burst-type workloads could consume network capacity, either in a one-to-many pattern where there is a single orchestrator, a many-to-many mesh-type communication model, or reach out to external systems. The work each worker is doing may be more or less resource-intensive (RAM-CPU-disk), worker may or may not have periods of idleness during its lifespan. Total population of actors is generally expected not to grow unbounded over time. Ideal suspend time likely known by either orchestrator or the worker itself.
-
Polling-type workloads could be a specific kind of burst-type workload, or they could be an example of a user requesting a long-running operation (example: "notify me every time my favorite band is in my area"). The total population of actors may be expected to increase unbounded over time due to no maximum lifespan for any individual actor. Unclear how frequently each actor needs to be awakened, but the type of model (polling vs. listening) likely affects this. May not be latency-sensitive. Likely uses state persistence (RAM, disk, remote DB). Ideal suspend time likely known by the worker.
-
Server-type workloads respond to inbound traffic, likely resulting in an uneven distribution of activity across the entire set of actors. Latency-sensitive due to the need to respond to inbound requests. May have concerns around disk usage complexity for storing necessary files, may include serving state in RAM and open FDs for long-running connections. Ideal suspend time likely indeterminate from server's perspective. There may be scaling concerns (increasing backend replicas to serve spikes in traffic and/or increasing server's resource footprint). Actor growth depends on who/what is defining these servers and load distribution depends on the popularity of a server and the characteristics of its work/audience (ex: what time zone do most users live in?)
-
Sandbox-type workloads likely have disk activity that needs persisting. Suspend/resume behavior likely known by sandbox owner. There are no sleep/wake patterns or usage distributions inherent in this model, though the caller likely knows the ideal suspend time.
Are there other workload classes/abstract behaviors people can think of?
I think the proposal in #98 can create useful mocks of most of the above systems, but we still need to define reasonable at-scale behavior for different:
- Activity distributions across the actor population (is it a small subset of actors doing the work while most are idle?)
- Different duty cycles (suspend/resume) and different distributions of duty cycles across the actor population
- What levels of disk/network/etc. activity are associated with each model use case? With what pacing?
- Are there correlations/anti-correlations of different scale dimensions (e.g. use cases with massive numbers of actors tend to use little disk per-actor)?
- Are there different tolerances for different use cases (e.g. backend tasks are less latency-sensitive)?
As functionality grows we likely want to be able to stress other targeted behaviors like scheduling, scaling, handling hotspotting, etc.
This is a big, ambiguous issue, but we should probably at least:
- Come up with one or more story for each type of workloads that represents a plausible system design/ product requirements
- For each story, estimate how a system will behave at scale using the above "abstract behaviors" as a guide for things to look out for
- For each story, come up with a target scale and what success looks like in terms of acceptable latency (or other criteria)
Quick example of why the story is super-important here: if we create one actor per minute, it does one activity and then suspends we can effectively support an infinite number of actors so long as we have enough storage for the snapshots. If 100% of the actors are active 100% of the time, we are hard-limited by the resources of the system. If we have a 50% duty cycle for all actors, we have a theoretical maximum of twice the system capacity, intermediated by things like sleep/wake times, scheduling efficiency, how much slack exists between idle time and suspend time, etc.
As the project matures we will want to validate that Substrate works "at scale", but what does that mean beyond "create/maintain a lot of things"?
We should define primary use cases and try to define the abstract behaviors that they exhibit to enable continuously validating performance improvements over time.
Example classes of workloads:
Abstract behaviors:
Burst-type workloads could consume network capacity, either in a one-to-many pattern where there is a single orchestrator, a many-to-many mesh-type communication model, or reach out to external systems. The work each worker is doing may be more or less resource-intensive (RAM-CPU-disk), worker may or may not have periods of idleness during its lifespan. Total population of actors is generally expected not to grow unbounded over time. Ideal suspend time likely known by either orchestrator or the worker itself.
Polling-type workloads could be a specific kind of burst-type workload, or they could be an example of a user requesting a long-running operation (example: "notify me every time my favorite band is in my area"). The total population of actors may be expected to increase unbounded over time due to no maximum lifespan for any individual actor. Unclear how frequently each actor needs to be awakened, but the type of model (polling vs. listening) likely affects this. May not be latency-sensitive. Likely uses state persistence (RAM, disk, remote DB). Ideal suspend time likely known by the worker.
Server-type workloads respond to inbound traffic, likely resulting in an uneven distribution of activity across the entire set of actors. Latency-sensitive due to the need to respond to inbound requests. May have concerns around disk usage complexity for storing necessary files, may include serving state in RAM and open FDs for long-running connections. Ideal suspend time likely indeterminate from server's perspective. There may be scaling concerns (increasing backend replicas to serve spikes in traffic and/or increasing server's resource footprint). Actor growth depends on who/what is defining these servers and load distribution depends on the popularity of a server and the characteristics of its work/audience (ex: what time zone do most users live in?)
Sandbox-type workloads likely have disk activity that needs persisting. Suspend/resume behavior likely known by sandbox owner. There are no sleep/wake patterns or usage distributions inherent in this model, though the caller likely knows the ideal suspend time.
Are there other workload classes/abstract behaviors people can think of?
I think the proposal in #98 can create useful mocks of most of the above systems, but we still need to define reasonable at-scale behavior for different:
As functionality grows we likely want to be able to stress other targeted behaviors like scheduling, scaling, handling hotspotting, etc.
This is a big, ambiguous issue, but we should probably at least:
Quick example of why the story is super-important here: if we create one actor per minute, it does one activity and then suspends we can effectively support an infinite number of actors so long as we have enough storage for the snapshots. If 100% of the actors are active 100% of the time, we are hard-limited by the resources of the system. If we have a 50% duty cycle for all actors, we have a theoretical maximum of twice the system capacity, intermediated by things like sleep/wake times, scheduling efficiency, how much slack exists between idle time and suspend time, etc.