[BUG] docker check: findDockerNetworks reads routes via pids[0], selecting an ephemeral child PID after host PID wraparound

[mday-cat]

[BUG] docker check: findDockerNetworks reads routes via pids[0], which selects an ephemeral child PID after host PID wraparound

Agent Environment

• Agent 7.82.x (datadog/agent:7 container image), ECS EC2 (Amazon Linux), daemon task
• Host uptime > 47 days; kernel PID counter has wrapped (long-running processes hold PIDs ~2.28M while newly spawned processes are allocated PIDs ~1.5M)

Describe what happened

The docker check logs the following WARN on every 30s check cycle, always for the same (healthy) container, with a different PID each time:

CORE | WARN | (pkg/collector/corechecks/containers/docker/check_network.go:243 in findDockerNetworks) |
Cannot list routes for container id <id>: unable to read file at: /host/proc/<pid>/net/route,
err: open /host/proc/<pid>/net/route: no such file or directory, skipping

The container is a PHP job worker (supervisord + several laravel queue:listen masters) that forks a short-lived child process per job. docker inspect shows State.Status=running, RestartCount=0, up 47+ days. docker.net.* metrics for the container are dropped on affected cycles.

Root cause

findDockerNetworks reads the container's route table through the first PID of the container's PID list:

// pkg/collector/corechecks/containers/docker/check_network.go (7.82.x)
pids, err := collector.GetPIDs(container.Namespace, container.ID, cacheValidity)  // L99
...
destinations, err := getRoutesFunc(procPath, entry.pids[0])                       // L241

GetPIDs returns the cgroup's PID list in ascending numeric order, so pids[0] is implicitly assumed to be the oldest/main process. That assumption holds only until the kernel PID counter wraps. After wraparound, freshly forked (short-lived) children receive PIDs numerically lower than the long-lived main processes, so pids[0] selects the newest ephemeral child. Combined with cacheValidity on the PID list, the selected PID is reliably dead by the time /proc/<pid>/net/route is read — producing a persistent per-cycle WARN and dropped network metrics for any process-churning container on a long-uptime host.

Evidence from the affected host:

$ docker top <container> -eo pid,etime,cmd | head
PID       ELAPSED      CMD
2280542   47-19:45:13  bash /docker-entrypoint.sh /usr/bin/supervisord ...
2280856   47-19:45:12  /usr/bin/python3 /usr/bin/supervisord ...
2281016   47-19:45:10  php artisan queue:listen sqs --tries=5 --timeout=7200
...

Stable masters: PIDs ~2.28M (47 days old). WARN-referenced PIDs across consecutive cycles: 1510160, 1510473, 1511051, 1511384, 1511936, 1512219, 1512806, 1513098, 1513706 — all post-wraparound ephemeral children, each already reaped.

Steps to reproduce

1. Host with uptime long enough for PID wraparound (or lower kernel.pid_max to accelerate).
2. Run a container whose main process is long-lived and which forks short-lived children at a steady rate (e.g. a Laravel queue:listen worker).
3. Enable the docker check; observe Cannot list routes WARNs on every cycle once new PIDs allocate below the container's main PID.

Expected behavior

Route lookup uses a PID that is still alive — e.g. prefer the container's init PID (State.Pid from inspect / the PID workloadmeta already tracks), or on ENOENT retry with the next PID in the list rather than skipping the container's network metrics for the cycle.

Suggested fix

In findDockerNetworks, iterate entry.pids until a readable /proc/<pid>/net/route is found (bounded), or use the container's main PID rather than pids[0]. Either removes the wraparound sensitivity.