Part of #550. First of the four phases in the design comment:
| Phase |
Scope |
Unblocks |
| 0 |
Stats method: proto, both reads, identity retention in ateom |
Independently testable, nothing user-facing |
Why the read lives in ateom
atelet is a node-level DaemonSet whose only host mount is /var/lib/ateom-gvisor — no /sys/fs/cgroup, no hostPID. The per-actor cgroup leaves are only addressable as /sys/fs/cgroup/<containerName> from inside the worker pod's own cgroup namespace, and the kata-agent vsock is only reachable from the ateom that booted the VMM. So the read has to originate in ateom.
Phase 0 delivers only the RPC that later phases call. atelet's reader, the aggregation, and the OTel emission are Phase 1+.
Scope
Add StatsWorkload to ateom.Ateom (internal/proto/ateompb/ateom.proto) — a pure read that does not change the ateom's available/executing state and is safe to call on a timer:
rpc StatsWorkload(StatsWorkloadRequest) returns (StatsWorkloadResponse) {}
message StatsWorkloadRequest {
// The actor the caller believes is running here. ateom rejects a mismatch
// with FAILED_PRECONDITION rather than reporting another actor's numbers.
string actor_uid = 1;
}
StatsWorkloadResponse as specified in the design comment: the five identity fields, sandbox_class, source, memory_current_bytes, memory_peak_bytes, memory_working_set_bytes, cpu_usage_usec, observed_at_unix_nano.
gVisor read — from /sys/fs/cgroup/pause: memory.current, memory.peak, memory.stat:inactive_file, cpu.stat:usage_usec. memory.peak must be optional (kernel 5.19+). pause/ and not main/ because setupCgroupForSubcontainer installs application-container leaves with empty resources and every application process actually runs inside the sentry, which is a single process in pause/.
Micro-VM read — via the kata guest agent's StatsContainer, not the host cgroup. Guest RAM is a fixed allocation (default 2048 MiB), so the host cgroup is roughly constant regardless of actor behaviour, and Shared: true backs it with a memfd — putting those pages on the file LRU, which makes the working-set subtraction cancel out idle guest RAM and understate the footprint. StatsContainerRequest/Response/CgroupStats are already in the vendored cmd/ateom-microvm/internal/third_party/kata/agentpb.
Identity retention — ateom must be able to answer with the identity from the original RunWorkloadRequest.
Two corrections to the design comment
- It says "
ateom-microvm already keeps them in running." It does not. runningActor (cmd/ateom-microvm/run.go:48-83) holds baseID, chCmd, vfsdCmd, durableVfsdCmd, apiSocket, restoreSourceDir, logAgent — no identity fields. The identity is already assembled as actorBootParams (run.go:224-231, exactly the five fields) and flows into coldBootActor / restoreFullScope, so retention is cheap — but it is real work on both runtimes, not just gVisor.
runningActor.logAgent is a live kata-agent ttrpc client held open for the actor's lifetime (closed only by teardownActor), so the micro-VM read needs no new connection — it reuses logAgent.
Planned PRs
- Proto, identity retention,
Unimplemented stub. ateom.proto + regen; workloadIdentity on the gVisor AteomService (set in RunWorkload/RestoreWorkload, cleared in CheckpointWorkload and on the retErr cleanup paths) and on runningActor for micro-VM. Retention covered by unit tests; not meaningfully testable in a live cluster until PR 2.
- gVisor cgroup read. New
cmd/ateom-gvisor/internal/cgroupstats with a caller-supplied root directory, so the parser is unit testable from a fixture tree without root.
- Micro-VM guest-agent read.
AgentClient.StatsContainer alongside the existing thin wrappers; one call per container keyed by overlayWorkloadID(name) (each workload has its own guest cgroup /ateomchv/<workloadID>), summed into the flat response. Note the unit conversion: the agent reports CPU in nanoseconds, the response field is microseconds.
Out of scope
Per-container breakdown for gVisor (needs runsc events / ContainerUsage from the sentry — Phase 3). ate.workerpool.name as a label: agreed in the thread to include it early, but ateletpb carries no workerpool field today, so plumbing it would widen this into atelet and its callers. Adding it to ateompb later is a backward-compatible proto change — deferring to Phase 1/2.
Part of #550. First of the four phases in the design comment:
Why the read lives in ateom
atelet is a node-level DaemonSet whose only host mount is
/var/lib/ateom-gvisor— no/sys/fs/cgroup, nohostPID. The per-actor cgroup leaves are only addressable as/sys/fs/cgroup/<containerName>from inside the worker pod's own cgroup namespace, and the kata-agent vsock is only reachable from the ateom that booted the VMM. So the read has to originate in ateom.Phase 0 delivers only the RPC that later phases call. atelet's reader, the aggregation, and the OTel emission are Phase 1+.
Scope
Add
StatsWorkloadtoateom.Ateom(internal/proto/ateompb/ateom.proto) — a pure read that does not change the ateom's available/executing state and is safe to call on a timer:rpc StatsWorkload(StatsWorkloadRequest) returns (StatsWorkloadResponse) {} message StatsWorkloadRequest { // The actor the caller believes is running here. ateom rejects a mismatch // with FAILED_PRECONDITION rather than reporting another actor's numbers. string actor_uid = 1; }StatsWorkloadResponseas specified in the design comment: the five identity fields,sandbox_class,source,memory_current_bytes,memory_peak_bytes,memory_working_set_bytes,cpu_usage_usec,observed_at_unix_nano.gVisor read — from
/sys/fs/cgroup/pause:memory.current,memory.peak,memory.stat:inactive_file,cpu.stat:usage_usec.memory.peakmust be optional (kernel 5.19+).pause/and notmain/becausesetupCgroupForSubcontainerinstalls application-container leaves with empty resources and every application process actually runs inside the sentry, which is a single process inpause/.Micro-VM read — via the kata guest agent's
StatsContainer, not the host cgroup. Guest RAM is a fixed allocation (default 2048 MiB), so the host cgroup is roughly constant regardless of actor behaviour, andShared: truebacks it with a memfd — putting those pages on the file LRU, which makes the working-set subtraction cancel out idle guest RAM and understate the footprint.StatsContainerRequest/Response/CgroupStatsare already in the vendoredcmd/ateom-microvm/internal/third_party/kata/agentpb.Identity retention — ateom must be able to answer with the identity from the original
RunWorkloadRequest.Two corrections to the design comment
ateom-microvmalready keeps them inrunning." It does not.runningActor(cmd/ateom-microvm/run.go:48-83) holdsbaseID,chCmd,vfsdCmd,durableVfsdCmd,apiSocket,restoreSourceDir,logAgent— no identity fields. The identity is already assembled asactorBootParams(run.go:224-231, exactly the five fields) and flows intocoldBootActor/restoreFullScope, so retention is cheap — but it is real work on both runtimes, not just gVisor.runningActor.logAgentis a live kata-agent ttrpc client held open for the actor's lifetime (closed only byteardownActor), so the micro-VM read needs no new connection — it reuseslogAgent.Planned PRs
Unimplementedstub.ateom.proto+ regen;workloadIdentityon the gVisorAteomService(set inRunWorkload/RestoreWorkload, cleared inCheckpointWorkloadand on theretErrcleanup paths) and onrunningActorfor micro-VM. Retention covered by unit tests; not meaningfully testable in a live cluster until PR 2.cmd/ateom-gvisor/internal/cgroupstatswith a caller-supplied root directory, so the parser is unit testable from a fixture tree without root.AgentClient.StatsContaineralongside the existing thin wrappers; one call per container keyed byoverlayWorkloadID(name)(each workload has its own guest cgroup/ateomchv/<workloadID>), summed into the flat response. Note the unit conversion: the agent reports CPU in nanoseconds, the response field is microseconds.Out of scope
Per-container breakdown for gVisor (needs
runsc events/ContainerUsagefrom the sentry — Phase 3).ate.workerpool.nameas a label: agreed in the thread to include it early, butateletpbcarries no workerpool field today, so plumbing it would widen this into atelet and its callers. Adding it toateompblater is a backward-compatible proto change — deferring to Phase 1/2.