feat(atenet): tell the actor why a credential was refused

FORK.md

@@ -73,6 +73,7 @@ Everything on `giantswarm` that is not in the pin (`git log v0.2.0-beta5..giants
 | The `rustfs-bucket-init` Job is a Helm hook (`post-install,post-upgrade`, delete policy `before-hook-creation,hook-succeeded`: a completed Job left behind would block a rollback to a release whose Job is a plain resource of the same name; `hack/install-microvm-deps.sh` and `hack/microvm-assets/stage-to-rustfs.sh` wait for the Job only while it exists), recreated on every install and upgrade under the same name, and `images.awsCli` is the gsoci copy `gsoci.azurecr.io/giantswarm/aws-cli:2.17.0` (upstream's digest) | a Job's pod template is immutable: as a plain release resource any change of its image failed the upgrade of every installation with `spec.template: … field is immutable` until the completed Job was deleted by hand, so the chart kept Docker Hub's short name `amazon/aws-cli` and the platform could not run from gsoci only (giantswarm/agent-platform#580). On an upgrade from a release where the Job is a release resource, Helm deletes the old Job (gone from the manifest) and the post-upgrade hook creates the new one. `charts/substrate/tests/rustfs_test.yaml` holds the hook | `74036f9a` ([#85](https://github.com/giantswarm/substrate/pull/85)) | to file for kagent-dev/substrate (the hook only; the gsoci default is fork-only), [giantswarm/giantswarm#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 94 |
 | The Go components honor `OTEL_METRICS_EXPORTER=none` (the chart's `otel.metrics.enabled: false`): `serverboot` installs no OTLP periodic reader and keeps the Prometheus one; `atecontroller` then registers its OTel instruments on controller-runtime's registry, so its metrics endpoint serves `ate_workerpool_*` next to `controller_runtime_*` | with the PodMonitors on and an OTLP endpoint set, every `ateapi`, `atelet`, `atenet-router` and `atecontroller` series reached the backend twice, once scraped (`ate_actor_crashes_total`) and once pushed (`ate_actor_crashes`), about 6,300 series on gazelle; the chart's switch rendered the variable and nothing read it ([giantswarm/giantswarm#36711](https://github.com/giantswarm/giantswarm/issues/36711)). `ateom` keeps pushing: its environment comes from the WorkerPool, not this chart | `18e2f1a0` ([#82](https://github.com/giantswarm/substrate/pull/82); code as in the upstream pull request; this line also updates the chart's `otel` comment and README row, which agent-substrate does not ship) | [agent-substrate/substrate#1878](https://github.com/agent-substrate/substrate/pull/1878) (issue [#1877](https://github.com/agent-substrate/substrate/issues/1877)), open (not a draft since 2026-09-25), carried until it merges and the next re-pin drops it; giantswarm/giantswarm#37742 row 95. The line stays on scrape-only metrics until OTLP metrics carry the Kubernetes labels in Mimir ([giantswarm/giantswarm#37935](https://github.com/giantswarm/giantswarm/issues/37935)) |
 | Actors run as their image's `USER` in its `WORKDIR` (atelet resolves `USER` as Docker does, against the image's own `/etc/passwd` and `/etc/group`; no `USER` is root; the pause container follows the same rule; the process starts in `WORKDIR`, default `/`, created in the actor's upper when the image lacks it), and what a non-root actor needs: the bundle's `rootfs` and overlay `upper` are `0755`, a `durableDir` volume is `0777`, and the kubectl manifest's atelet holds `CAP_DAC_OVERRIDE` to empty the directories a non-root actor left in it (the chart's atelet is privileged already) | atelet ran every container as root in `/` whatever its image declared, and on gVisor the container's `/` took the `0700` of the overlay upper, so a process that is not root could exec nothing. kagent's harness images declare `USER 65532` and were run as root regardless; giantswarm/kagent-upstream#125 separates the harness (root, declared as such in the image) from Claude Code and Codex (uid 65532) ([#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 85, giantswarm/giantswarm#38054) | `git cherry-pick -x` of the whole stack, in upstream's order: `8f7eb07d` (rootfs root, carried earlier as `916f3c6e`, same diff), with the fork-only test of a restored bundle; `afb62e18` (merged as agent-substrate/substrate#1910; the manifest keeps the fork's `/var/lib/ateom-gvisor` path); `895ec3ce` (#1906); `1e0466df`, `fa8578de`, `5925173f`, `d41a2fc2` (#1918). Adapted to the line: `ocispec.Options` keeps its `ActorUID`/`ContainerName` shape and gains only `User` and `Cwd`, paths stay in `ateompath`, and the fork's `TestPrepareOCIDirectory_RegistryRejectionIsTerminal` passes the bundle path | not ours: [agent-substrate/substrate#1918](https://github.com/agent-substrate/substrate/pull/1918) (author dims, open, five commits, rebased on 2026-10-01) and [#1906](https://github.com/agent-substrate/substrate/pull/1906) (open), which #1918 asks to merge first; #1910 is merged. All fall away at the re-pin onto the first release that carries #1918. **The upgrade onto this line breaks every gVisor snapshot**: `runsc restore` checks each container's `Process.User` and `Process.Cwd` against the checkpoint and the pause image sets `USER 65535`, so delete every gVisor actor before the roll and recreate every gVisor template after it (`docs/upgrade.md`, step 4); micro-VM snapshots resume as they were. The harness images must declare their identity (`USER 0:0`, `WORKDIR /`, kagent-upstream#125) before the roll, or the harness itself starts as 65532 |
+| The egress gateway tells the actor why a credential was refused (`mapCredentialProviderError` in `cmd/atenet/internal/router/egress/credentials.go`, the contract on `FetchSecret` in `pkg/proto/credproviderpb/credprovider.proto`: `NotFound`, `PermissionDenied`, `FailedPrecondition` and `Unauthenticated` deny as 403 with the provider's message, escaped and cut to 512 bytes; `Unavailable`, `DeadlineExceeded` and `ResourceExhausted` as 503; anything else, and an empty or unusable secret, as 502; every body names the provider and nothing of the transport) | kagent's caller credential provider refuses for reasons the actor can act on (no turn running, no grant at the token broker: sign in and retry), and every refusal reached the actor as `egress denied`; git prints a `text/plain` error body as `remote:` lines, so the reason is what a coding agent relays to its person. A provider bug answered 403 like a refusal, which an agent reads as a permission problem: 502 is the gateway's own outcome when the provider it relies on misbehaves. This departs from upstream's deliberate 403 for anything unexpected (chosen so a client does not retry); no answer becomes 404 or 400, which git and `gh` read as a missing repository or a malformed request. The agentgateway dataplane applies the same table (giantswarm/agentgateway-upstream `fork/credential-provider-errors`); giantswarm/giantswarm#38056 | `feat/credential-provider-errors` | to file, with the agentgateway half: the 502 for unexpected answers needs upstream's agreement |
 
 Fifteen patches change Substrate ahead of upstream — egress for an actor while it resumes, without which no
 skill-carrying agent of the platform boots, the atelet scheduling knobs, the keep policy on the CRD chart's

cmd/atenet/internal/router/egress/credentials.go

@@ -15,8 +15,13 @@
 package egress
 
 import (
+	"cmp"
 	"context"
 	"log/slog"
+	"strconv"
+	"strings"
+	"unicode"
+	"unicode/utf8"
 
 	corev3 "github.com/envoyproxy/go-control-plane/envoy/config/core/v3"
 	envoy_type "github.com/envoyproxy/go-control-plane/envoy/type/v3"
@@ -30,22 +35,55 @@ import (
 	"github.com/agent-substrate/substrate/pkg/proto/credproviderpb"
 )
 
-// mapCredentialProviderError converts a FetchSecret failure into a
-// client-facing ext_proc denial, mirroring mapEgressIdentityError: a credential
-// the provider does not hold or will not release (NotFound, PermissionDenied)
-// denies as 403 — retrying cannot succeed — while a transient provider failure
-// (Unavailable, DeadlineExceeded) fails closed as a retryable 503. Anything
-// unexpected denies rather than inviting retries of a request that cannot be
-// completed as the policy promised.
-func mapCredentialProviderError(err error) error {
+// maxProviderMessageBytes bounds the part of a provider's denial that reaches
+// the actor.
+const maxProviderMessageBytes = 512
+
+// mapCredentialProviderError converts a FetchSecret failure into the denial the
+// credprovider contract specifies: a credential the provider will not release
+// denies as 403 with the provider's own message, the only text of the provider
+// that reaches the actor; a transient provider failure fails closed as a
+// retryable 503; any other failure is the provider's or the gateway's fault, a
+// 502. The cause stays reachable through Unwrap for the caller's log.
+func mapCredentialProviderError(provider string, err error) error {
 	switch status.Code(err) {
-	case codes.NotFound, codes.PermissionDenied:
-		return extproc.WrapReqError(envoy_type.StatusCode_Forbidden, err, deniedBody)
-	case codes.Unavailable, codes.DeadlineExceeded:
-		return extproc.WrapReqError(envoy_type.StatusCode_ServiceUnavailable, err, deniedBody)
+	case codes.NotFound, codes.PermissionDenied, codes.FailedPrecondition, codes.Unauthenticated:
+		message := providerMessage(status.Convert(err).Message())
+		if message == "" {
+			return extproc.WrapReqError(envoy_type.StatusCode_Forbidden, err, "credential provider %s denied the request", provider)
+		}
+		return extproc.WrapReqError(envoy_type.StatusCode_Forbidden, err, "credential provider %s denied: %s", provider, message)
+	case codes.Unavailable, codes.DeadlineExceeded, codes.ResourceExhausted:
+		return extproc.WrapReqError(envoy_type.StatusCode_ServiceUnavailable, err, "credential provider %s unavailable", provider)
 	default:
-		return extproc.WrapReqError(envoy_type.StatusCode_Forbidden, err, deniedBody)
+		return credentialProviderFailed(provider, err)
+	}
+}
+
+// credentialProviderFailed is the 502 of a provider that answered with
+// something the gateway cannot use.
+func credentialProviderFailed(provider string, err error) error {
+	return extproc.WrapReqError(envoy_type.StatusCode_BadGateway, err, "credential provider %s failed", provider)
+}
+
+// providerMessage makes a provider's status message safe for a response body:
+// control characters are escaped, invalid UTF-8 is replaced, and the result is
+// cut to maxProviderMessageBytes on a rune boundary.
+func providerMessage(message string) string {
+	var b strings.Builder
+	for _, r := range strings.ToValidUTF8(message, string(utf8.RuneError)) {
+		var piece string
+		if unicode.IsControl(r) {
+			piece = strings.Trim(strconv.QuoteRune(r), "'")
+		} else {
+			piece = string(r)
+		}
+		if b.Len()+len(piece) > maxProviderMessageBytes {
+			break
+		}
+		b.WriteString(piece)
 	}
+	return strings.TrimSpace(b.String())
 }
 
 // applyEffects resolves a matched rule's credential injections and returns the
@@ -109,6 +147,12 @@ func (h *Handler) applyEffects(ctx context.Context, ref resources.ActorRef, dest
 			}
 		}
 
+		// The provider's name in the actor's denial.
+		provider, err := providerNameFromURI(inj.GetCredentialUri())
+		if err != nil {
+			provider = cmp.Or(h.providerName, "unknown")
+		}
+
 		resp, err := h.provider.FetchSecret(ctx, &credproviderpb.FetchSecretRequest{
 			Uri:           inj.GetCredentialUri(),
 			ActorSpiffeId: actorSpiffeID,
@@ -118,13 +162,13 @@ func (h *Handler) applyEffects(ctx context.Context, ref resources.ActorRef, dest
 			// must not let the request out without it.
 			slog.ErrorContext(ctx, "egress denied: credential fetch failed",
 				slog.Any("actor", ref), slog.String("host", dest.Hostname), slog.String("uri", inj.GetCredentialUri()), slog.Any("err", err))
-			return nil, mapCredentialProviderError(err)
+			return nil, mapCredentialProviderError(provider, err)
 		}
 		secret, err := sanitizeSecret(resp.GetOpaqueBytes())
 		if err != nil {
 			slog.ErrorContext(ctx, "egress denied: unusable credential",
 				slog.Any("actor", ref), slog.String("host", dest.Hostname), slog.String("uri", inj.GetCredentialUri()), slog.Any("err", err))
-			return nil, extproc.WrapReqError(envoy_type.StatusCode_ServiceUnavailable, err, deniedBody)
+			return nil, credentialProviderFailed(provider, err)
 		}
 
 		// Overwrite any header the actor set itself, so a client cannot pre-seed a

cmd/atenet/internal/router/egress/credentials_test.go

@@ -17,6 +17,7 @@ package egress
 import (
 	"context"
 	"errors"
+	"strings"
 	"testing"
 
 	corev3 "github.com/envoyproxy/go-control-plane/envoy/config/core/v3"
@@ -147,51 +148,121 @@ func TestInjectionDenials(t *testing.T) {
 		providerName string
 		leg          string
 		want         envoy_type.StatusCode
+		wantBody     string
 	}{
 		{
 			name:         "credential URI for another provider is refused",
 			provider:     &fakeProvider{resp: bearerTokenResponse("s3cr3t")},
 			providerName: "vault", // policy URI is ate-secret://k8s/...
 			leg:          extproc.EgressTLSMITMFilterChainName,
 			want:         envoy_type.StatusCode_InternalServerError,
+			wantBody:     deniedBody,
 		},
 		{
 			// A transient provider failure is retryable.
 			name:         "provider unavailable fails closed as retryable",
-			provider:     &fakeProvider{err: status.Error(codes.Unavailable, "provider down")},
+			provider:     &fakeProvider{err: status.Error(codes.Unavailable, "dial tcp provider.internal:50051: connection refused")},
 			providerName: injectionProviderName,
 			leg:          extproc.EgressTLSMITMFilterChainName,
 			want:         envoy_type.StatusCode_ServiceUnavailable,
+			wantBody:     "credential provider k8s unavailable",
+		},
+		{
+			name:         "provider deadline fails closed as retryable",
+			provider:     &fakeProvider{err: status.Error(codes.DeadlineExceeded, "deadline")},
+			providerName: injectionProviderName,
+			leg:          extproc.EgressTLSMITMFilterChainName,
+			want:         envoy_type.StatusCode_ServiceUnavailable,
+			wantBody:     "credential provider k8s unavailable",
+		},
+		{
+			name:         "exhausted provider fails closed as retryable",
+			provider:     &fakeProvider{err: status.Error(codes.ResourceExhausted, "quota")},
+			providerName: injectionProviderName,
+			leg:          extproc.EgressTLSMITMFilterChainName,
+			want:         envoy_type.StatusCode_ServiceUnavailable,
+			wantBody:     "credential provider k8s unavailable",
 		},
 		{
 			// A secret the provider does not hold cannot appear on retry.
-			name:         "secret not found denies as non-retryable",
+			name:         "secret not found denies with the provider's message",
 			provider:     &fakeProvider{err: status.Error(codes.NotFound, "no such secret")},
 			providerName: injectionProviderName,
 			leg:          extproc.EgressTLSMITMFilterChainName,
 			want:         envoy_type.StatusCode_Forbidden,
+			wantBody:     "credential provider k8s denied: no such secret",
 		},
 		{
-			name:         "provider refuses the actor denies as non-retryable",
+			name:         "provider refuses the actor denies with the provider's message",
 			provider:     &fakeProvider{err: status.Error(codes.PermissionDenied, "atespace not allowed")},
 			providerName: injectionProviderName,
 			leg:          extproc.EgressTLSMITMFilterChainName,
 			want:         envoy_type.StatusCode_Forbidden,
+			wantBody:     "credential provider k8s denied: atespace not allowed",
 		},
 		{
-			// An unclassified error denies rather than inviting retries.
-			name:         "unexpected provider error denies",
-			provider:     &fakeProvider{err: errors.New("provider down")},
+			name:         "failed precondition denies with the provider's message",
+			provider:     &fakeProvider{err: status.Error(codes.FailedPrecondition, "sign in to github and retry")},
+			providerName: injectionProviderName,
+			leg:          extproc.EgressTLSMITMFilterChainName,
+			want:         envoy_type.StatusCode_Forbidden,
+			wantBody:     "credential provider k8s denied: sign in to github and retry",
+		},
+		{
+			name:         "a denial without a message says so",
+			provider:     &fakeProvider{err: status.Error(codes.Unauthenticated, "")},
+			providerName: injectionProviderName,
+			leg:          extproc.EgressTLSMITMFilterChainName,
+			want:         envoy_type.StatusCode_Forbidden,
+			wantBody:     "credential provider k8s denied the request",
+		},
+		{
+			name:         "a denial's control characters are escaped",
+			provider:     &fakeProvider{err: status.Error(codes.PermissionDenied, "no grant\r\nSet-Cookie: x")},
 			providerName: injectionProviderName,
 			leg:          extproc.EgressTLSMITMFilterChainName,
 			want:         envoy_type.StatusCode_Forbidden,
+			wantBody:     `credential provider k8s denied: no grant\r\nSet-Cookie: x`,
 		},
 		{
-			name:         "empty secret fails closed",
+			name:         "a denial's message is truncated",
+			provider:     &fakeProvider{err: status.Error(codes.PermissionDenied, strings.Repeat("é", 400))},
+			providerName: injectionProviderName,
+			leg:          extproc.EgressTLSMITMFilterChainName,
+			want:         envoy_type.StatusCode_Forbidden,
+			wantBody:     "credential provider k8s denied: " + strings.Repeat("é", maxProviderMessageBytes/2),
+		},
+		{
+			name:         "internal provider error is a bad gateway without its message",
+			provider:     &fakeProvider{err: status.Error(codes.Internal, "panic in provider at /src/x.go")},
+			providerName: injectionProviderName,
+			leg:          extproc.EgressTLSMITMFilterChainName,
+			want:         envoy_type.StatusCode_BadGateway,
+			wantBody:     "credential provider k8s failed",
+		},
+		{
+			name:         "invalid argument is a bad gateway, not the actor's fault",
+			provider:     &fakeProvider{err: status.Error(codes.InvalidArgument, "bad actor identity")},
+			providerName: injectionProviderName,
+			leg:          extproc.EgressTLSMITMFilterChainName,
+			want:         envoy_type.StatusCode_BadGateway,
+			wantBody:     "credential provider k8s failed",
+		},
+		{
+			name:         "unclassified error is a bad gateway",
+			provider:     &fakeProvider{err: errors.New("provider down")},
+			providerName: injectionProviderName,
+			leg:          extproc.EgressTLSMITMFilterChainName,
+			want:         envoy_type.StatusCode_BadGateway,
+			wantBody:     "credential provider k8s failed",
+		},
+		{
+			name:         "empty secret is a bad gateway",
 			provider:     &fakeProvider{resp: bearerTokenResponse("")},
 			providerName: injectionProviderName,
 			leg:          extproc.EgressTLSMITMFilterChainName,
-			want:         envoy_type.StatusCode_ServiceUnavailable,
+			want:         envoy_type.StatusCode_BadGateway,
+			wantBody:     "credential provider k8s failed",
 		},
 	}
 	for _, tc := range tests {
@@ -200,6 +271,10 @@ func TestInjectionDenials(t *testing.T) {
 			_, err := h.HandleRequestHeaders(context.Background(),
 				innerMetadata(tc.leg, "GET", "api.example.com", nil))
 			wantStatus(t, err, tc.want)
+			var re *extproc.ReqError
+			if errors.As(err, &re) && re.Msg != tc.wantBody {
+				t.Errorf("body = %q, want %q", re.Msg, tc.wantBody)
+			}
 		})
 	}
 }

docs/kubernetes-credential-provider.md

@@ -81,6 +81,23 @@ Create the Secret and set an actor's egress policy header injection to use crede
 `authorization`, and prefix `Bearer `. Namespace grants alone do not create an
 egress policy. No ext_proc injector is needed.
 
+## Errors the actor sees
+
+A credential the gateway cannot fetch fails the request closed with a
+`text/plain` body, as the
+[FetchSecret contract](../pkg/proto/credproviderpb/credprovider.proto) specifies:
+
+| Provider answer | Status | Body |
+|---|---|---|
+| `NotFound`, `PermissionDenied`, `FailedPrecondition`, `Unauthenticated` | 403 | `credential provider <name> denied: <message>` |
+| `Unavailable`, `DeadlineExceeded`, `ResourceExhausted` | 503 | `credential provider <name> unavailable` |
+| any other code, or an empty or unusable secret | 502 | `credential provider <name> failed` |
+
+Only a denial's message reaches the actor, with control characters escaped and
+cut to 512 bytes. This provider's denials name the Secret and namespace, which
+the actor's own egress policy already names. The full error is in the gateway's
+log.
+
 ## Tests
 
 The Helm PR workflow installs the provider and MITM gateway from the start and