feat(skills): private Git skill sources via egress gateway credentials

[acherifi]

Problem

A v1alpha3 AgentTemplate cannot use a skill from a private Git repository: skills[].source.git has no credential, and the runtime runs a plain git fetch (#2815).

Adding a credential field does not fix this on its own. Every Git, OCI or S3 skill, public or private, fails today while the golden Actor starts (#2604), for two independent reasons in Substrate v0.4.0-alpha1:

1. ateom-gvisor RunWorkload activates the Actor's egress tunnel only after the wakeup probe passes (cmd/ateom-gvisor/main.go: wakeupprobe.WaitAll, then s.tunnel.Activate). Until then atunnel closes every outbound connection. The runtimes fetch skills before they serve /readyz.
2. The golden Actor runs in the reserved ate-golden atespace. Substrate's template reconciler creates it, and kagent installs per-Actor egress policies only on session Actors (ApplyActorTransition). atenet therefore denies the golden Actor's requests with "actor has no egress policy". Credential injection could not resolve there either: the credential provider authorizes Secrets per atespace (see feat(substrate): the compiled ActorTemplate carries the revision's egress policy as its default #2942).

The Substrate egress gateway is the right place for the credential: it keeps the token out of the runtime, the same way model keys and RemoteMCPServer.headersFrom are handled. The fetch just has to happen when the gateway is in the path.

Change

API (additive, optional). GitArtifact.authorizationFrom is a SecretKeyReference (name, key; no namespace) to a Secret in the AgentTemplate's namespace. The Secret value is the complete Authorization header, for example Basic base64(x-access-token:TOKEN) for GitHub, or Bearer TOKEN. This follows the existing headersFrom convention that the Secret holds the full value. CEL rules require an https URL, and reject the field on plugins[].source.git (see Limits). CRDs, deepcopy and the Helm CRD copies are regenerated.

Compiler. CompileCredentials binds each such key to the repository hostname as authorization (ate-secret://k8s.io/default/<template namespace>/<secret>/<key>). It walks Shared child templates as well. CompileSkillResources already adds the repository origin to the derived egress allow-list. The binding goes through egress.CanonicalCredentials, so two different Secrets for one Git host are rejected, and so is a conflict with a model or MCP credential on that host. URLs with user info are rejected. The runtime contract (agentplugin.GitSource) gains only gateway_authorization: true. It does not name the Secret.

Runtime.

• skillsinit.CloneGitCommit runs git -c http.extraHeader="Authorization: kagent-credential-injected" fetch … for these sources. Substrate's gateway replaces only headers a request already carries (applyEffects), so a placeholder is required. It is passed per command, so nothing is written to .git/config. All git invocations now set GIT_TERMINAL_PROMPT=0, so a missing credential fails at once instead of waiting for a username prompt.
• agentplugins.Materialize no longer fetches standalone skills. It validates them, still fetches plugins, and returns a Materialization whose EnsureSkills(ctx) fetches the standalone skills. That call is idempotent and safe for concurrent use. A failure resumes at the failed skill on the next call.
• The Go ADK (KAgentExecutor), Claude and Codex (ProcessDriver.Run) runtimes call EnsureSkills before every task. Once the skills are present the call does nothing. The ADK always wires the skill toolset when standalone skills are configured; skilltoolset lists the directory on every model request.

Can the secret end up in a golden snapshot?

No, for two reasons:

1. The secret never enters the Actor. The runtime only has a placeholder. The gateway reads the Secret through the credential provider at request time and overwrites the header on the TLS-intercepted leg. Revision provenance and the compiled config hold the Secret name and key, never the value. The behavior matches existing model and MCP credentials.
2. Nothing is fetched before the golden snapshot. The golden Actor never receives a task, so EnsureSkills never runs there: the snapshot contains neither the credential nor the private skill content. The first task of each session Actor does the fetch. By then networking is active and kagent has installed that Actor's egress policy and credential binding (created suspended, then policy, then resume). The fetched files end up in that session's own snapshots. Those hold skill content, never the credential.

Behavior changes

• Latency. The first task of every session Actor waits for the skill fetch (a depth-1 fetch of one pinned commit per skill). Later tasks of the same Actor reuse the files. Actors restored from a FULL snapshot already have them.
• Failure reporting. A fetch failure (bad token → 401, unreachable host, missing SKILL.md) fails the task instead of crashing the golden Actor. The task gets a failed status whose message is skills_unavailable: materialize skill "<name>": …, and the runtime logs the error ("standalone skills are unavailable" in the ADK). The next task retries. The revision itself becomes ready, because readiness no longer depends on the network.
• Public standalone skills (Git, OCI, S3) work again on Substrate. This fixes [BUG] Git, OCI, and S3 Agent Plugin resources cannot be materialized while creating Substrate golden Actors #2604 for skills[], not for plugins[].

Limits

• Plugins are unchanged. They are still fetched at startup, because they contribute MCP servers and Claude plugin directories while the runtime is built. That path is still affected by [BUG] Git, OCI, and S3 Agent Plugin resources cannot be materialized while creating Substrate golden Actors #2604, so authorizationFrom is rejected on plugins by CEL and by the compiler.
• Scope of the binding. Substrate matches credentials by hostname. The header is added to every request the Actor sends to that host, including requests from model-driven tools such as shell or MCP. The token never becomes readable, but its authority is usable from inside the Actor for its lifetime, the same as for MCP headers. Use a read-only token scoped to the skill repositories. All skills on one host share one credential.
• Only static header credentials are supported: no SSH, and no OAuth or app-token exchange.
• The Secret must be readable through the Substrate credential provider grant for the agent atespace (credentialProvider.namespacePolicies), like other gateway credentials.

Relation to other work

• feat(api): a per-source credential for private git skill and plugin sources #2899 (draft, giantswarm) adds a similar credentialRef routed through the gateway. It has three differences:

  • Its Secret holds base64(user:token) and the compiler adds Basic . Here the Secret holds the full header value, following the headersFrom convention, which also allows Bearer.
  • It sends the placeholder through an environment variable plus helper. Here git sends an http.extraHeader placeholder, with no environment variable and no Secret-backed EnvVar.
  • It keeps the fetch at startup, where [BUG] Git, OCI, and S3 Agent Plugin resources cannot be materialized while creating Substrate golden Actors #2604 means the gateway never sees the request. This PR moves the standalone-skill fetch to the first task, which is the part that makes the credential usable.

  Happy to rebase onto feat(api): a per-source credential for private git skill and plugin sources #2899's API shape if maintainers prefer it.

• [FEATURE] Private skill and plugin sources in the v1alpha3 API #2815 proposes fetching in the controller and mounting the result as an image volume. That design is blocked on Substrate (anonymous image-volume pulls, atelet: pull image credentials from the kubelet credential provider agent-substrate/substrate#917). This PR needs no Substrate change and does not preclude it.

• Closes the standalone-skill half of [BUG] Git, OCI, and S3 Agent Plugin resources cannot be materialized while creating Substrate golden Actors #2604; refs [FEATURE] Private skill and plugin sources in the v1alpha3 API #2815.

Tests

• api/v1alpha3: envtest CRD validation (TestConfigurationCRDValidation/git_skill_authorizationFrom). Cases: valid skill; missing key → Required value; empty key; namespace → unknown field under strict field validation; http:// URL → rejected; plugin → rejected.
• core/internal/translator: binding plus egress host and contract flag (TestCompilePrivateGitSkillBindsGatewayCredentialAndEgress), a Shared child template namespace, and conflicts, user info and plugins rejected.
• core/internal/skillsinit: TestCloneGitCommitThroughCredentialGateway runs a git http-backend behind an httptest server that requires Authorization. In front of it sits a fake gateway that, like Substrate, replaces the header only when it is present. Cases: placeholder → fetch succeeds and .git/config holds nothing; no placeholder → fails fast; placeholder sent directly → 401.
• core/pkg/agentplugins: Materialize does no standalone fetch; EnsureSkills fetches, resumes after a partial failure without refetching, and is a no-op on a zero value.
• adk/pkg/a2a: a failure produces a failed status with skills_unavailable, and the agent is not invoked; it runs once skills are present. adk/pkg/config, and the claude/codex driver and adapter tests, are updated for the deferred fetch.

go build ./..., go vet and go test pass for ./api/... ./adk/... ./harness/... ./core/pkg/... ./core/internal/{skillsinit,translator/...,substrate,egress}.

Not verified: an end-to-end run on a kind cluster with Substrate (make -C go e2e). There is no existing e2e coverage for skills; it would need a private Git fixture behind a DNS name. I have not observed the first-task fetch through the real gateway.

[QuentinBisson]

We hit the same failure on a Substrate line with replace-only replace_headers: a private git skill failed with could not read Username, because git sends no Authorization before a challenge. I checked that git sends a per-command http.extraHeader on its very first info/refs request, so the placeholder works through the gateway without a 401 round-trip.

A few notes:

1. Nit: http.<url>.extraHeader instead of http.extraHeader keeps the placeholder on the source's own URL (it can be passed per invocation through GIT_CONFIG_COUNT/GIT_CONFIG_KEY_0/GIT_CONFIG_VALUE_0 or -c).
2. In Limits: with replace-only semantics the credential goes on requests to that host that carry an Authorization header, not on every request. A tool can still set one, so the read-only, repository-scoped token advice stands.
3. Fetching at the first task is the right call while golden actors have no egress ([BUG] Git, OCI, and S3 Agent Plugin resources cannot be materialized while creating Substrate golden Actors #2604). It does put one fetch per repository in every session's first task, so host availability and rate limits now apply per session. Once golden-time egress exists (Configure egress access during golden initialization agent-substrate/substrate#2159 with Trigger CI workflows on all release/** branches #2117), moving the fetch back to golden boot would fetch once per revision, keep content in the golden snapshot, and let plugins use the same field. Glad to take that as a follow-up rather than grow this PR.

I'll close #2899 in favour of this one. The full-header Secret is the better shape.

[pood1e]

Tested this PR on a three-node K3s cluster with kagent v1.0.0-alpha9 (46fdd3d, this PR's base), Substrate 0.4.0-alpha1 and the microvm sandbox class (Kata Containers 4.1.0, Cloud Hypervisor v53.0). The Codex and Claude harness images were built from 46fdd3d plus this PR with upstream's Dockerfiles and default CLI versions; the controller is the stock alpha9 image.

Before (stock alpha9 harnesses). An AgentTemplate that selected skills through plugins[] from a public Git repository at a pinned commit never became ready: Ready=False, ActorTemplateFailed, GoldenActorCrashed. The worker logged the golden Actor's wakeup probe failing for 30 s (connection refused on :8081), and the egress gateway recorded no request, which matches #2604.

With this PR. The same templates use standalone skills[] (source.git with url and commit, and path: skills/<name>):

• The golden snapshot became ready (ActorTemplate golden snapshot is ready) for both the Codex and the Claude agent.
• On the first task, both runtimes fetched the skill through the egress gateway. Codex read it from /data/codex/skills/<name>/SKILL.md, and Claude loaded it with the Skill tool from /data/generated/claude/.claude/skills/<name>/SKILL.md.
• The unit tests of core/pkg/agentplugins, core/internal/skillsinit and harness/... pass on the PR head.

Not covered: authorizationFrom, because our skill repository is public. plugins[] still fails at golden creation, as the PR description says.

We now run these patched harness images in our development environment until a release includes this fix. Thanks for working on it!