QuentinBisson · Oct 1, 2026 · Oct 1, 2026 · Oct 2, 2026
@@ -97,6 +97,7 @@ Everything on `giantswarm` that is not in the pin (`git log v0.4.0-alpha1..giant
 | The durable pause's background commit holds no lease on the actor: it writes the uploaded copy under a version precondition with the actor's lease as the write's fence (`store.Precondition.Unleased`: the write takes the lease for its own transaction, so a lifecycle operation arriving meanwhile waits for that transaction and reads the copy, and a write arriving while an operation holds the lease is refused with `ErrLeaseConflict`, which the commit waits out and reads again). `acquireActorLease` no longer retries a held lease: two lifecycle operations on one actor are `Aborted` at once, as upstream answers. A lease records its holder and when it was taken (`leases.holder`, `leases.acquired_at`, migration `migrations_fork/000001_lease_holder.sql`; `AcquireLease(ctx, key, holder)`), the store answers a conflict as `store.LeaseConflictError` naming them, and the acquire and release logs carry the holder, the hold's duration and, on a conflict, who held it for how long. The lifecycle e2e runner pauses and suspends directly again. Tests: the store contract (a fenced write refused while held, landed while free, waited for by an acquire that arrives during it), the lease row's holder, the commit waiting for a lifecycle holder, and the actor's next operation taking the lease at once while a commit with slow store round trips is in flight | the commit held the lease for its store round trips; on the loaded `E2E (agentgateway)` job those outlasted the six retries, about 0.7 s, of `acquireActorLease`, and a pause or suspend right after a resume from `PAUSED` failed with `Aborted` ([giantswarm/substrate#187](https://github.com/giantswarm/substrate/issues/187)): the only background holder of an actor's lease was that commit, and no wait a client can afford bounds a database under load. Now nothing but a lifecycle operation holds the lease, and the one write the commit makes is ordered against them by the store | `giantswarm`: [#193](https://github.com/giantswarm/substrate/pull/193) | to file with the durable-pause patch; the lease holder and the fenced update are upstream-shaped (no fork references). Its migration, released in 1.5.0 as `migrations/000003_lease_holder.sql`, is `migrations_fork/000001_lease_holder.sql` in the fork migration ledger since the 2026-10-08 re-pin (next row). Upstream's worker-assignment reconciler takes its leases as `worker-assignment-reconcile` (the worker's) and `worker-assignment-release` (an actor's) |
 | The line's own PostgreSQL migrations live in `cmd/ateapi/internal/store/atepg/migrations_fork/` with their own Goose ledger, `fork_schema_migrations`, applied after upstream's `migrations/` and its ledger `schema_migrations`, which carry upstream's files unchanged. A 1.5.x database, which applied the lease holder as version 3 of `schema_migrations`, is moved once at startup under the migration lock: the fork ledger is created with its version 1 applied and version 3 leaves `schema_migrations`, so upstream's `000003_access_policies.sql` applies as 3; `leases.holder` without the fork ledger in any other state is refused before any migration runs. `hack/verify/postgresql-migrations.sh` checks both directories. Tests: a 1.5 database (its lease row kept, the access-policy tables written, a restart moving nothing), a fresh one, and the refused state | renumbering upstream's migrations around the line's shifts every later upstream migration on every re-pin and leaves upstream files whose number differs on the line; with two ledgers a re-pin carries `migrations/` as upstream ships it. A 1.6 ateapi rolled back to 1.5.x finds `schema_migrations` at 3 and the lease columns it reads | `giantswarm`: the 2026-10-08 re-pin (`sync/20261008-v0.4.0-alpha1`) | none: fork-only, upstream carries no migrations but its own |
 | The atenet router answers a denial in the caller's protocol: a gRPC call (`content-type` `application/grpc`, alone or with a `+format` suffix or parameters) gets a trailers-only gRPC response — HTTP 200, no body, `content-type: application/grpc`, `grpc-status`, the percent-encoded `grpc-message`, and the code in the immediate response's `grpc_status` — whose code is the gRPC status of the denial's cause (Aborted for a held actor, NotFound for a missing one, ResourceExhausted for a full pool), else the code nearest the HTTP status, and whose message is the denial's client-safe message; every other request (gRPC-Web included) keeps its status code, `text/plain` and body (`cmd/atenet/internal/router/extproc`; unit tests on the mux, e2e `TestIngressDenialProtocol` in the networking suite) | a gRPC caller of a held actor (one whose lease another operation holds, answered Aborted by ate-api) got `Unavailable`: Envoy turned the router's plain-text 503 into a gRPC status by gRPC's HTTP-to-status table, so the caller could not tell a held actor, whose operation it should retry, from an unavailable control plane (a missing actor arrived as `Unimplemented`) ([giantswarm/substrate#158](https://github.com/giantswarm/substrate/issues/158); proven with the dataplane's Envoy v1.39 in the pull request). The AgentGateway dataplane, which this line's chart runs, answers its own ingress denials by the same fallback table (a missing actor `Unimplemented`, a held one `Unknown`), which the e2e records per dataplane; its counterpart is [giantswarm/agentgateway-upstream#43](https://github.com/giantswarm/agentgateway-upstream/issues/43) | `giantswarm`: [#200](https://github.com/giantswarm/substrate/pull/200) | to file: the upstream-shaped patch is branch [`upstream/atenet-grpc-denial-code`](https://github.com/giantswarm/substrate/tree/upstream/atenet-grpc-denial-code) here (`0a91139b`, on the mirror `main`); agent-substrate/substrate `main` (`296e329a`, 2026-10-08) still answers every denial as `text/plain` and no upstream issue or pull request covers it (searched 2026-10-08); [#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 137; a team member opens the agent-substrate/substrate pull request with DCO sign-off once #37742 has reviewed it |
+| The egress gateway takes further credential providers (`credentialProvider.additionalProviders`: `uriAuthority`, a lowercase DNS name, and `host`, `<service>.<namespace>.svc:<port>`, rendered on both egress listeners beside the bundled `k8s.io` provider, dialed with the gateway's pod identity and verified against the `servicedns.podcert.ate.dev` trust bundle; an entry cannot replace `k8s.io` or repeat an authority; `charts/substrate/tests/atenet_egress_credential_providers_test.yaml`) | kagent injects the caller of the current turn at egress through its own provider, so the caller's bearer never enters the actor: agentgateway already selects a provider by URI authority and fetches the actor's egress policy on every request, but the chart rendered one provider only (giantswarm/giantswarm#38054) | [#104](https://github.com/giantswarm/substrate/pull/104) | to file for kagent-dev/substrate (chart only); giantswarm/giantswarm#37742 row to add |
 
 Twenty-three patches change Substrate ahead of upstream — egress for an actor while it resumes, without which no
 skill-carrying agent of the platform boots, the atelet scheduling knobs, the keep policy on the CRD chart's

@@ -56,6 +56,7 @@ See `values.yaml` for the full set; the important keys:
 | `atelet.imageCache.pinnedImages` | `[]` | Image references every pass pulls and never evicts (the ConfigMap `atelet-pinned-images`, mounted for `--image-cache-pinned-images-file`): the pool's ActorTemplate images on nodes whose cache volume stays above the watermark. atelet reads the list at every pass, so a change restarts no atelet pod |
 | `credentialProvider.namespacePolicies` | `[]` | Default-deny atespace-to-namespace grants; the chart includes get-only Secret RBAC for the provider |
 | `ateController.gvisorWorker.followRelease` | `true` | Every gVisor WorkerPool runs this release's `ateom-gvisor` image (`--gvisor-worker-image`), composed like the other component images, instead of its own `spec.workerImage`; `false` runs each pool's `workerImage` |
+| `credentialProvider.additionalProviders` | `[]` | Further credential providers for the egress gateway, each `uriAuthority` (a lowercase DNS name) and `host` (`<service>.<namespace>.svc:<port>`); the gateway verifies the provider's `servicedns.podcert.ate.dev` serving certificate and presents its own pod identity, which the provider must require |
 | `ateApi.extraArgs` | `[]` | Additional command-line arguments appended to the ateapi defaults |
 | `atelet.nodeSelector` | `{}` | Nodes the atelet DaemonSet runs on; empty = every schedulable node |
 | `atelet.tolerations` | `[]` | Tolerations of the atelet pods, added to the `ate.dev/sandboxClass` toleration the chart always sets |

@@ -268,3 +268,44 @@ caps them all; unset, kubelet's default applies and nothing is rendered.
 maxExpirationSeconds: {{ int . }}
 {{- end }}
 {{- end }}
+
+{{/*
+The egress gateway's credential providers: the bundled k8s.io one and
+credentialProvider.additionalProviders, each dialed with the gateway's pod
+identity and verified against the servicedns.podcert.ate.dev trust bundle.
+*/}}
+{{- define "substrate.egressCredentialProviders" -}}
+{{- $providers := list (dict "uriAuthority" "k8s.io" "host" (printf "%s.%s.svc:50051" (include "substrate.fullname" (list "k8s-credential-provider" .)) .Release.Namespace)) -}}
+{{- range .Values.credentialProvider.additionalProviders -}}
+{{- if or (not .uriAuthority) (not .host) -}}
+{{- fail "credentialProvider.additionalProviders entries need uriAuthority and host" -}}
+{{- end -}}
+{{- $authority := toString .uriAuthority -}}
+{{- $host := toString .host -}}
+{{- if not (regexMatch "^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$" $authority) -}}
+{{- fail (printf "credentialProvider.additionalProviders: uriAuthority %q must be a lowercase DNS name" $authority) -}}
+{{- end -}}
+{{- if eq $authority "k8s.io" -}}
+{{- fail "credentialProvider.additionalProviders cannot replace the bundled k8s.io provider" -}}
+{{- end -}}
+{{- if not (regexMatch "^[a-z0-9]([-a-z0-9]*[a-z0-9])?\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?\\.svc:[0-9]{1,5}$" $host) -}}
+{{- fail (printf "credentialProvider.additionalProviders: host %q must be <service>.<namespace>.svc:<port>" $host) -}}
+{{- end -}}
+{{- $providers = append $providers (dict "uriAuthority" $authority "host" $host) -}}
+{{- end -}}
+{{- $seen := dict -}}
+{{- range $providers }}
+{{- if hasKey $seen .uriAuthority -}}
+{{- fail (printf "credentialProvider.additionalProviders names %s twice" .uriAuthority) -}}
+{{- end -}}
+{{- $_ := set $seen .uriAuthority true }}
+- uriAuthority: {{ .uriAuthority }}
+  target:
+    host: {{ .host }}
+    policies:
+      backendTLS:
+        cert: /run/podidentity.podcert.ate.dev/credential-bundle.pem
+        key: /run/podidentity.podcert.ate.dev/credential-bundle.pem
+        root: /run/servicedns.podcert.ate.dev/trust-bundle.pem
+{{- end }}
+{{- end -}}
@@ -75,14 +75,7 @@ data:
                   key: /run/podidentity.podcert.ate.dev/credential-bundle.pem
                   root: /run/servicedns.podcert.ate.dev/trust-bundle.pem
               credentialProviders:
-              - uriAuthority: k8s.io
-                target:
-                  host: {{ include "substrate.fullname" (list "k8s-credential-provider" .) }}.{{ .Release.Namespace }}.svc:50051
-                  policies:
-                    backendTLS:
-                      cert: /run/podidentity.podcert.ate.dev/credential-bundle.pem
-                      key: /run/podidentity.podcert.ate.dev/credential-bundle.pem
-                      root: /run/servicedns.podcert.ate.dev/trust-bundle.pem
+              {{- include "substrate.egressCredentialProviders" . | trim | nindent 14 }}
       - protocol: TLS
         hostname: "*"
         tcpRoutes:
@@ -102,14 +95,7 @@ data:
                   key: /run/podidentity.podcert.ate.dev/credential-bundle.pem
                   root: /run/servicedns.podcert.ate.dev/trust-bundle.pem
               credentialProviders:
-              - uriAuthority: k8s.io
-                target:
-                  host: {{ include "substrate.fullname" (list "k8s-credential-provider" .) }}.{{ .Release.Namespace }}.svc:50051
-                  policies:
-                    backendTLS:
-                      cert: /run/podidentity.podcert.ate.dev/credential-bundle.pem
-                      key: /run/podidentity.podcert.ate.dev/credential-bundle.pem
-                      root: /run/servicedns.podcert.ate.dev/trust-bundle.pem
+              {{- include "substrate.egressCredentialProviders" . | trim | nindent 14 }}
       - protocol: TCP
         tcpRoutes:
         - backends:

@@ -0,0 +1,96 @@
+# Copyright 2026 The Agent Substrate Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+suite: atenet egress credential providers
+templates:
+- atenet-egress.yaml
+tests:
+- it: resolves only the bundled k8s.io provider by default
+  documentSelector:
+    path: kind
+    value: ConfigMap
+  asserts:
+  - matchRegex:
+      path: data["config.yaml"]
+      pattern: '(?s)credentialProviders:\n\s+- uriAuthority: k8s\.io\n\s+target:\n\s+host: RELEASE-NAME-k8s-credential-provider\.NAMESPACE\.svc:50051'
+  - notMatchRegex:
+      path: data["config.yaml"]
+      pattern: '(?s)(?:uriAuthority:.*){3}'
+
+- it: adds a provider to both the HTTPS and the HTTP egress listener
+  set:
+    credentialProvider.additionalProviders:
+    - uriAuthority: kagent.dev
+      host: kagent-credentials.kagent.svc:50051
+  documentSelector:
+    path: kind
+    value: ConfigMap
+  asserts:
+  - matchRegex:
+      path: data["config.yaml"]
+      pattern: '(?s)protocol: HTTPS.*- uriAuthority: kagent\.dev\n\s+target:\n\s+host: kagent-credentials\.kagent\.svc:50051\n\s+policies:\n\s+backendTLS:\n\s+cert: /run/podidentity\.podcert\.ate\.dev/credential-bundle\.pem\n\s+key: /run/podidentity\.podcert\.ate\.dev/credential-bundle\.pem\n\s+root: /run/servicedns\.podcert\.ate\.dev/trust-bundle\.pem.*protocol: HTTP\n.*- uriAuthority: kagent\.dev'
+
+- it: refuses an entry without a host
+  set:
+    credentialProvider.additionalProviders:
+    - uriAuthority: kagent.dev
+  asserts:
+  - failedTemplate:
+      errorMessage: credentialProvider.additionalProviders entries need uriAuthority and host
+
+- it: refuses to replace the bundled provider
+  set:
+    credentialProvider.additionalProviders:
+    - uriAuthority: k8s.io
+      host: other.example.svc:50051
+  asserts:
+  - failedTemplate:
+      errorMessage: credentialProvider.additionalProviders cannot replace the bundled k8s.io provider
+
+- it: refuses the same authority twice
+  set:
+    credentialProvider.additionalProviders:
+    - uriAuthority: kagent.dev
+      host: a.kagent.svc:50051
+    - uriAuthority: kagent.dev
+      host: b.kagent.svc:50051
+  asserts:
+  - failedTemplate:
+      errorMessage: credentialProvider.additionalProviders names kagent.dev twice
+
+- it: refuses an entry without a uriAuthority
+  set:
+    credentialProvider.additionalProviders:
+    - host: kagent-credentials.kagent.svc:50051
+  asserts:
+  - failedTemplate:
+      errorMessage: credentialProvider.additionalProviders entries need uriAuthority and host
+
+- it: refuses a uriAuthority that is not a lowercase DNS name
+  set:
+    credentialProvider.additionalProviders:
+    - uriAuthority: "Kagent.dev\n  target: {}"
+      host: kagent-credentials.kagent.svc:50051
+  asserts:
+  - failedTemplate:
+      errorMessage: "credentialProvider.additionalProviders: uriAuthority \"Kagent.dev\\n  target: {}\" must be a lowercase DNS name"
+
+- it: refuses a host the servicedns certificate cannot name
+  set:
+    credentialProvider.additionalProviders:
+    - uriAuthority: kagent.dev
+      host: kagent-credentials.kagent.svc.cluster.local:50051
+  asserts:
+  - failedTemplate:
+      errorMessage: "credentialProvider.additionalProviders: host \"kagent-credentials.kagent.svc.cluster.local:50051\" must be <service>.<namespace>.svc:<port>"
@@ -132,6 +132,18 @@ credentialProvider:
   namespacePolicies: []
   # - atespace: team-a
   #   allowedNamespaces: [team-a-secrets]
+  # Further providers the egress gateway resolves ate-secret:// URIs through,
+  # by URI authority (a lowercase DNS name), beside the bundled k8s.io
+  # one. host is <service>.<namespace>.svc:<port>: the gateway verifies the
+  # provider's servicedns.podcert.ate.dev serving certificate, which names its
+  # Service in that form, and the Service must select the provider's pod. The
+  # gateway presents its pod identity certificate
+  # (spiffe://cluster.local/ns/<release namespace>/sa/<release>-atenet-egress);
+  # the provider must require that client certificate, chained to the
+  # podidentity.podcert.ate.dev trust bundle, and refuse any other caller.
+  additionalProviders: []
+  # - uriAuthority: kagent.dev
+  #   host: kagent-credentials.kagent.svc:50051
 
 atenetRouter:
   # Request parking on the actor routes (agentgateway's substrateIngress