lima-guestagent sustains ~0.75 of a CPU core on an idle guest (2.1.1, colima/vz, aarch64)

[CodeArtificial]

Summary

lima-guestagent 2.1.1 sustains ~0.75 of a CPU core on a guest that is otherwise idle from a port-forwarding point of view. Over 47 ten-minute windows spanning seven hours it averaged 0.737 cores (sd 0.152, range 0.455–0.997) — about 6% of a 12-vCPU guest, permanently.

96.5% of it is system time (utime 1,620,785 jiffies vs stime 44,319,479). Sampling /proc/<pid>/task/*/syscall 400 times across 19 threads (7,600 samples): 82% futex, 8.6% epoll_pwait, 3.5% nanosleep, 5.3% userspace. 5.4% of thread samples are runnable, which independently reproduces ~1 core.

Environment

lima-guestagent	2.1.1
driver	colima on macOS (Apple silicon), vz
guest	12 vCPU, aarch64
workload	~38 containers, ~90 bind(2)/s steady state
portForwarder	ssh

What it is not — each ruled out by a controlled experiment

Not /proc/net/tcp polling. /proc/<pid>/io over 10 s shows 50 reads and 38 writes (~9/s) and read_bytes does not move. sendmsg/recvmsg appear zero times in the 7,600 syscall samples, so it is not a netlink sock_diag dump either.

Not the eBPF event rate. The agent holds a 13-instruction BPF_PROG_TYPE_TRACEPOINT, a 1 MiB ringbuf and a perf link on sys_exit_bind (PR #3067). Injecting 1,452,954 bind() calls in 5 s moved run_cnt by 1,453,438 — a 1:1 match, so the tracepoint counts what it looks like it counts. But a controlled 22× increase in the event rate (90/s → 2,016/s) moved CPU from 1.020 to 0.998 cores. The BPF program's own cost is 1,913 runs in 20 s = 25.9 ms, 0.13% of a core.

Note for anyone reproducing: run_cnt and run_time_ns in a BPF fdinfo read 0 unless kernel.bpf_stats_enabled=1. They would otherwise "prove" the program never runs.

Not memory. VmRSS is 40,380 kB, byte for byte, both freshly started and after millions of events.

Not the size of the port set. 2,295,702 binds to one fixed port cost the same as 2,155,640 binds across ~28,000 ephemeral ports.

Not fixable by restarting. Six 60-second windows after a restart: 0.965 mean, against 0.927 as-found — +4.1%, Welch t = 1.44. (A shorter earlier look suggested a restart bought ~0.35 cores; a controlled repeat says it buys nothing.)

Not GOMAXPROCS. GOMAXPROCS=1 (threads 18 → 8) measured 0.668 against 0.965 unset, which looks like −31% at t = −5.39. The reversal arm refutes it: reverting to the shipped configuration gave 0.756, not a climb back (t = 1.21), and a second GOMAXPROCS=1 arm gave 0.760. The reverted arm also differs from the identically-configured earlier arm by 0.209 cores at t = −4.16, and the last arm decays from 0.967 to 0.658 across six minutes on its own. The process's own drift is larger than any treatment we could apply, which is itself part of this report.

Questions

1. Is ~0.75 of a core expected for an idle guestagent? If not, a profile of where the futex traffic comes from would settle it — we cannot profile the Go runtime from outside the process.
2. Is there a documented way to disable the eBPF port detector and fall back to the 3-second polling it replaced? lima-guestagent daemon --help exposes only --tick, which is the polling interval rather than a gate on the eBPF path, and neither the docs nor Support for eBPF based port forwarding #3067 name a flag or environment variable.
3. Would a static portForwards: list bypass the detector? Untested here, because disabling port forwarding blind would take down services that depend on it.

Reproduction

A guest with a few dozen containers and a modest bind(2) rate; watch ps -eo pid,pcpu,etimes,comm --sort=-pcpu. Measure over hours, not minutes — see the drift note above. Short windows on this process produced three confident wrong answers in our own testing before a reversal arm caught them.