Skip to content

Actor egress is activated only after readyz, so a workload that fetches remote content to become ready can never start #1543

Open

Description

Disclosure: this issue was written by an AI agent (Claude) while verifying kagent
documentation against a live cluster, and reviewed by a human before filing.

Summary

An Actor has no working egress until every readyz-enabled container reports ready, but a
workload that must fetch remote content in order to become ready therefore cannot start.
Actor networking is activated only after readyz.WaitAll returns, which creates a circular
dependency for any image that pulls from the network during startup.

This is the substrate half of kagent-dev/kagent#2604, which blocks Git, OCI, and S3 Agent
Plugin sources outright. Filing here because the ordering that has to change is in
ateom-gvisor.

The ordering

cmd/ateom-gvisor/main.go, in both the Run and the Restore paths:

Step Run Restore
deactivateActorNetworking :625 :905
containers created and started :679-700 —
readyz.WaitAll blocks for 200 from every readyz container :722 :1031
activateActorNetworking :725 :1034

Preparing the egress identity and installing the redirect is not the same as allowing
traffic. Until activateActorNetworking runs, the redirect has no active tunnel and the
downstream connection is closed as soon as it is made, which is why callers see a connection
that opens and immediately dies rather than a refusal or a timeout.

readiness requires remote content
    -> fetching remote content requires egress
        -> egress activation requires readiness

Observed

On a kind cluster (kagent b8d53d39, substrate 0.0.25), a golden Actor in the
ate-golden atespace failed both source kinds during startup:

Initialized empty Git repository in /plugins/standalone-0/.git/
fatal: unable to access 'https://github.com/...': Send failure: Broken pipe
materialize agent plugins: materialize skill "skill-creator": exit status 128
pull ghcr.io/.../...@sha256:d497...: Get "https://ghcr.io/v2/": EOF

Both end in failed to materialize Agent Plugins and the process exits, so the container
never reports ready and networking is never activated.

It is not a cluster-wide network problem. A normal, already-golden Actor reached its model
provider fine over the same path — atenet-egress logged exactly one request in two hours,
tls.sni=api.openai.com — and an in-cluster MCP server was reachable from an Actor. There
is no egress-gateway log line at all for github.com or ghcr.io, consistent with the traffic
never leaving the sandbox.

Impact

The failure is also hard to diagnose from the consumer side. The template reports
Ready=False, reason ActorTemplatePending, message waiting for the ActorTemplate golden snapshot — identical to a healthy template that is still booting, and it names neither the
source nor the network. The real error appears only in the worker pod logs.

Expected

A workload can reach its declared egress destinations during startup, before it is required
to report ready. Options that would resolve it:

  • Activate actor networking before waiting on readyz, so startup fetches can use the
    configured egress policy.
  • Or introduce an explicit initialization phase with egress enabled that runs before the
    readiness gate.
  • Or expose an ActorTemplate option for workloads that need egress during startup.

Whichever way it goes, a startup fetch failure should surface as a distinguishable condition
rather than presenting as "still waiting for the golden snapshot".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions