Skip to content

Phase 0: StatsWorkload RPC on ateom (proto, both runtime reads, identity retention) #594

Description

@baizhenyu

Part of #550. First of the four phases in the design comment:

Phase Scope Unblocks
0 Stats method: proto, both reads, identity retention in ateom Independently testable, nothing user-facing

Why the read lives in ateom

atelet is a node-level DaemonSet whose only host mount is /var/lib/ateom-gvisor — no /sys/fs/cgroup, no hostPID. The per-actor cgroup leaves are only addressable as /sys/fs/cgroup/<containerName> from inside the worker pod's own cgroup namespace, and the kata-agent vsock is only reachable from the ateom that booted the VMM. So the read has to originate in ateom.

Phase 0 delivers only the RPC that later phases call. atelet's reader, the aggregation, and the OTel emission are Phase 1+.

Scope

Add StatsWorkload to ateom.Ateom (internal/proto/ateompb/ateom.proto) — a pure read that does not change the ateom's available/executing state and is safe to call on a timer:

rpc StatsWorkload(StatsWorkloadRequest) returns (StatsWorkloadResponse) {}

message StatsWorkloadRequest {
  // The actor the caller believes is running here. ateom rejects a mismatch
  // with FAILED_PRECONDITION rather than reporting another actor's numbers.
  string actor_uid = 1;
}

StatsWorkloadResponse as specified in the design comment: the five identity fields, sandbox_class, source, memory_current_bytes, memory_peak_bytes, memory_working_set_bytes, cpu_usage_usec, observed_at_unix_nano.

gVisor read — from /sys/fs/cgroup/pause: memory.current, memory.peak, memory.stat:inactive_file, cpu.stat:usage_usec. memory.peak must be optional (kernel 5.19+). pause/ and not main/ because setupCgroupForSubcontainer installs application-container leaves with empty resources and every application process actually runs inside the sentry, which is a single process in pause/.

Micro-VM read — via the kata guest agent's StatsContainer, not the host cgroup. Guest RAM is a fixed allocation (default 2048 MiB), so the host cgroup is roughly constant regardless of actor behaviour, and Shared: true backs it with a memfd — putting those pages on the file LRU, which makes the working-set subtraction cancel out idle guest RAM and understate the footprint. StatsContainerRequest/Response/CgroupStats are already in the vendored cmd/ateom-microvm/internal/third_party/kata/agentpb.

Identity retention — ateom must be able to answer with the identity from the original RunWorkloadRequest.

Two corrections to the design comment

  1. It says "ateom-microvm already keeps them in running." It does not. runningActor (cmd/ateom-microvm/run.go:48-83) holds baseID, chCmd, vfsdCmd, durableVfsdCmd, apiSocket, restoreSourceDir, logAgent — no identity fields. The identity is already assembled as actorBootParams (run.go:224-231, exactly the five fields) and flows into coldBootActor / restoreFullScope, so retention is cheap — but it is real work on both runtimes, not just gVisor.
  2. runningActor.logAgent is a live kata-agent ttrpc client held open for the actor's lifetime (closed only by teardownActor), so the micro-VM read needs no new connection — it reuses logAgent.

Planned PRs

  1. Proto, identity retention, Unimplemented stub. ateom.proto + regen; workloadIdentity on the gVisor AteomService (set in RunWorkload/RestoreWorkload, cleared in CheckpointWorkload and on the retErr cleanup paths) and on runningActor for micro-VM. Retention covered by unit tests; not meaningfully testable in a live cluster until PR 2.
  2. gVisor cgroup read. New cmd/ateom-gvisor/internal/cgroupstats with a caller-supplied root directory, so the parser is unit testable from a fixture tree without root.
  3. Micro-VM guest-agent read. AgentClient.StatsContainer alongside the existing thin wrappers; one call per container keyed by overlayWorkloadID(name) (each workload has its own guest cgroup /ateomchv/<workloadID>), summed into the flat response. Note the unit conversion: the agent reports CPU in nanoseconds, the response field is microseconds.

Out of scope

Per-container breakdown for gVisor (needs runsc events / ContainerUsage from the sentry — Phase 3). ate.workerpool.name as a label: agreed in the thread to include it early, but ateletpb carries no workerpool field today, so plumbing it would widen this into atelet and its callers. Adding it to ateompb later is a backward-compatible proto change — deferring to Phase 1/2.

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions