benchmarking/locust: record cluster hardware facts and density frontiers

[Nishanth Kotla (Nishanth29)]

Part of #1590

First of two PRs. This one covers Kubernetes-side hardware discovery; the follow-up adds the Prometheus harvest.

What this PR does

The locust runner records no information about the hardware it runs on, so trial results cannot be normalized across machine types or node counts. This adds cluster capacity discovery and derives actor density frontiers from it, persisting both the raw readings and the derived ratios to stats.jsonl.

Proposed Changes

Cluster hardware discovery

New module benchmarking/locust/cluster_facts.py, kept separate from runner.py. It reads node count, machine type, allocatable cores and allocatable RAM, and the worker pod count via the official Kubernetes Python client, which resolves in-cluster and local kubeconfig auth without a kubectl subprocess.

Node and pod listing is proportional to cluster size, so --no-cluster-facts disables discovery. Discovery is best effort in either case: an unreachable API server or missing RBAC leaves the facts null and does not fail the run.

No value is defaulted or inferred. Unmeasured fields are null; a measured zero is 0. The two remain distinguishable to any consumer.

Density frontiers

Added to the trial_summary row in stats.jsonl:

Field	Meaning
actors_per_node	peak actors over node count
actors_per_vcpu	peak actors over allocatable cores
actors_per_gb_ram	peak actors over allocatable RAM
actors_per_pod_p50 / _p90 / _p99	distribution of actors per worker pod
aggregate_failure_ratio	failures over requests, all RPCs
<operation>_failure_ratio	one per operation Locust reported, e.g. dur_dir_write_failure_ratio

Actors per pod is reported as a distribution across the run's time samples (p50 / p90 / p99) rather than one average, because ramp-up and custom load shapes have no single user count to call steady.

The raw readings (machine_type, node_count, allocatable_cores, allocatable_ram_gb, worker_pod_count) are written alongside the derived ratios so they can be re-derived without re-running the trial.

RBAC

Nodes are cluster scoped and require a ClusterRole with list on nodes. Pods require only a namespaced Role with list in benchmark-workloads. locust.yaml and runner-job.yaml.tmpl each define their own separately named pair.

Note that locust.yaml now binds a Role in benchmark-workloads, so that namespace must exist before the manifest is applied. benchmarking/workloads/deploy.sh creates it.

status.json is unchanged.

How this was tested

11 unit tests in test_cluster_facts.py covering percentile boundaries, capacity scoped to only the nodes running worker pods, zero worker pods treated as a valid reading, unreadable facts returning null, and flag behavior.

Verified against a live GKE cluster. Discovery returned c3d-standard-8, 1 node, 7.91 allocatable cores, 27.73 GB, 5 worker pods, all matching kubectl. The emitted frontiers were re-derived by hand from stats_history.csv and matched. --no-cluster-facts returns all nulls.

Both manifests pass kubectl apply --dry-run=client, with no cluster-scoped name collisions between them.

References

Agent Substrate: Actor Density Benchmark Specs

• Tests pass
• Appropriate changes to documentation are included in the PR

[Max Smythe (maxsmythe)]

Getting closer, a bit more feedback.

[Haowei Cai (Roy) (roycaihw)]

LGTM

[Max Smythe (maxsmythe)]

LGTM