Skip to content

fix(health): accept null IPAM.Config for Docker built-in host and none networks #92

Closed

Description

@Nickfost

Problem

Docker built-in networks can legitimately return IPAM.Config: null. In _docker_network_headroom(), scripts/health.py rejects this before reaching the exemption for built-in networks, causing a false CRITICAL docker_network_inspection and failed post-activation health verification.

The September 8 operator diagnostic reports Docker 29.7.2 returning null for the built-in host and none networks. Two scheduled reconciliation attempts at 21:13:14–21:15:22 UTC and 21:17:23–21:19:31 UTC completed candidate activation, then failed health verification and restored the checkpoint. Both lasted about 128 seconds, with zero managed runners during drain. This is not evidence of actual subnet exhaustion. These runtime details are operator-supplied; the repository logic was independently checked when filing.

Verified source

health.py at the current PR #91 head: configs = entry.get("IPAM", {}).get("Config", []) is followed by rejection of non-list values. The built-in-network exemption is later, after subnet parsing. The same ordering exists on main.

Scope

Handle legitimate null IPAM configuration for built-in host/none networks before generic list validation. Preserve accounting for the default bridge and custom networks, and preserve strict handling of malformed inspection data. Add focused regression fixtures using the Docker 29 response shape.

Acceptance criteria

  • A fixture containing built-in host/none networks with null IPAM.Config reaches network headroom evaluation without a false inspection critical.
  • Valid custom networks and the default bridge still contribute the correct occupied subnets; healthy, reserve-warning, and genuine exhaustion cases retain their expected severity.
  • Null or malformed custom-network IPAM, malformed JSON, and inspection failures continue to fail closed; the fix does not blanket-coerce arbitrary invalid values to an empty list.
  • Existing empty-list and IPv6 cases retain their intended behavior.
  • A health/installer regression demonstrates that this valid Docker response no longer causes post-activation rollback.
  • Focused regressions fail before the fix and pass afterward. Exact implementation head passes Validate ci-fleet prototype / Build without registering a runner, including scripts/validate.sh.

No host timeout override, Docker downgrade, network-policy relaxation, resource pruning, or live deployment is part of this code fix.

Coordination and rollout

Coordinate with #91, which already modifies the reconciliation/installer transaction. Reviewed PR head when filing: 779e69f9aaecbcd43e465629264ca181cfa7b3af; main/base: 003de44f7e27d7bfe5bb753098efb6eb5f9712f1. Avoid overlapping installer writers; refresh the PR head before implementation.

After the reviewed fixes merge, operational rollout needs a new private configuration commit selecting the fixed engine. Reapplying an existing configuration pinned to the old engine will not deploy the fix. Keep private configuration and infrastructure records outside this public issue. Repository completion and subsequent operational verification should be reported separately.

Operational follow-up: after selecting the fixed engine, record a successful scheduled reconcile, applied desired configuration, absence of the false inspection critical, and controller readiness. A passing unit test alone does not establish deployment success.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions