Skip to content

S10.A2.b: BYO end-to-end deployment + raw_env #67

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Pal Lakatos-Toth (pallakatos) merged 1 commit into from
Apr 28, 2026
Merged

S10.A2.b: BYO end-to-end deployment + raw_env #67

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 63 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,69 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased] — Phase 2

### S10.A2.b `phase2-multi-runtime-byo` — BYO end-to-end deployment + `raw_env`

#### Added
- **`RuntimeDeploymentPlan.raw_env: Vec<serde_json::Value>`** —
captures structural env entries (e.g. `valueFrom: secretKeyRef:`)
from BYO `spec.runtime.byo.env`. Static `value:` entries continue
to flow via `runtime_extra_env: BTreeMap<String,String>` (S10.A2).
- **`plan_byo()` populates `raw_env`** from any env entry the static
flattener skipped — the existing reserved-prefix / NUL / dup name
filter applies to `raw_env` entries' `name` field; the
`valueFrom` payload renders verbatim.
- **`build_runtime_plan_dispatches_byo_to_producer`** unit test
asserting `RuntimeKind::BYO` no longer returns `AdapterMissing`.

#### Changed
- **`RuntimeKind::BYO` now routes through `Ok(plan_byo(cfg))`** in
`build_runtime_plan` (was `AdapterMissing`). BYO sandboxes get
end-to-end Pod deployment.
- **Reconciler `mod.rs`** gained `is_byo` flag derived from
`runtime_spec.kind`. The following env entries are skipped when
`is_byo`: `OPENCLAW_MODEL`, `OPENCLAW_GATEWAY_TOKEN`,
`FOUNDRY_DEPLOYMENTS`, `FOUNDRY_AGENT_ID`, `FOUNDRY_AGENT_TOOLS`.
Critical: `OPENCLAW_GATEWAY_TOKEN` references Secret
`gateway-token` which is OpenClaw-namespace-scoped — BYO
referencing it would `CreateContainerConfigError`.
- **Agent container extracted** into a `let agent_container = json!`
binding before the deployment macro. Conditional fields:
`name: "agent"` (BYO) vs `"openclaw"` (OpenClaw); port 18789 only
when `!is_byo`; admin-token volumeMount only when `!is_byo`;
`command` / `args` set from `plan.command` / `plan.args` when
`Some(...)`.
- **`raw_env` consumption block** added in the reconciler after the
static `runtime_extra_env` block. Defensive skip on entries
missing `name`.
- Renamed
`plan_returns_adapter_missing_for_each_non_openclaw_kind` →
`plan_returns_adapter_missing_for_each_unwired_non_openclaw_kind`
(BYO removed from cases vec; remaining unwired kinds:
`OpenAIAgents`, `MicrosoftAgentFramework`).

#### Tests
- 307/307 controller tests pass (was 306 in S10.A2; +1 new
dispatcher test).
- `cargo clippy --package azureclaw-controller --all-targets -- -D
warnings` clean.
- `cargo fmt --all -- --check` clean.

#### Audit
- `docs/security-audits/2026-04-28-phase2-multi-runtime-byo.md`
(threat model: gateway-token escape attempt, container-name
divergence, raw_env reserved-prefix coverage, post-deploy patch
compatibility).

#### Follow-ups not in this slice
- S10.A3 (`phase2-runtime-openai-agents`) — first runnable
non-OpenClaw runtime; will exercise BYO-shaped deployment path
with a real Python 3.12 image; first multi-runtime e2e Kind test.
- S10.A4 (`phase2-runtime-microsoft-agent-framework`) — flips §14.6
column 11 fully ✓.
- S10.B (`phase2-platform-mcp-server`) — Foundry-shim platform MCP
server in router. Should ship before S10.A3/A4 so adapters are
trivial.

### S10.A2 `phase2-multi-runtime-dispatch` — `RuntimeDeploymentPlan` dispatch seam

#### Added
Expand Down
214 changes: 154 additions & 60 deletions controller/src/reconciler/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -700,6 +700,12 @@ async fn reconcile(sandbox: Arc<ClawSandbox>, ctx: Arc<Context>) -> Result<Actio
.and_then(|b| b.per_request)
.unwrap_or(0);

// S10.A2.b: OpenClaw vs BYO branch the *agent container shape*.
// The router sidecar, init container, NetworkPolicy, SA, seccomp,
// volumes, and security context are runtime-agnostic. Only the
// agent container itself differs.
let is_byo = matches!(runtime_spec.kind, crate::crd::RuntimeKind::BYO);

// Build OpenClaw container env vars.
//
// OPENCLAW_GATEWAY_TOKEN is plumbed via secretKeyRef rather than a static
Expand All @@ -712,42 +718,56 @@ async fn reconcile(sandbox: Arc<ClawSandbox>, ctx: Arc<Context>) -> Result<Actio
// token; if env and Secret ever drift, the operator gets 401s on a
// rolled pod even though the Secret looks correct. valueFrom closes
// that drift window.
let mut openclaw_env = vec![
json!({"name": "OPENCLAW_MODEL", "value": inference_config.model}),
json!({"name": "AZURE_OPENAI_ENDPOINT", "value": &ctx.openai_endpoint}),
json!({"name": "AZURECLAW_AUTH_MODE", "value": "workload-identity"}),
json!({
//
// For BYO, OPENCLAW_* envs are skipped: the gateway-token Secret is
// OpenClaw-specific (azureclaw up provisions it for OpenClaw only),
// and a BYO pod referencing it without `optional: true` would
// ImagePullBackOff-style fail to start.
let mut openclaw_env: Vec<serde_json::Value> = Vec::new();
if !is_byo {
openclaw_env
.push(json!({"name": "OPENCLAW_MODEL", "value": inference_config.model.clone()}));
}
openclaw_env.push(json!({"name": "AZURE_OPENAI_ENDPOINT", "value": &ctx.openai_endpoint}));
openclaw_env.push(json!({"name": "AZURECLAW_AUTH_MODE", "value": "workload-identity"}));
if !is_byo {
openclaw_env.push(json!({
"name": "OPENCLAW_GATEWAY_TOKEN",
"valueFrom": {
"secretKeyRef": {
"name": "gateway-token",
"key": "token"
}
}
}),
];
}));
}
// Foundry project endpoint (for standalone APIs: Memory Store, Foundry IQ, etc.)
if !ctx.foundry_project_endpoint.is_empty() {
openclaw_env.push(
json!({"name": "FOUNDRY_PROJECT_ENDPOINT", "value": &ctx.foundry_project_endpoint}),
);
}
// Foundry deployments list (so plugin shows only deployed models, not full catalog)
if !ctx.foundry_deployments.is_empty() {
// Foundry deployments list (so plugin shows only deployed models, not full catalog).
// BYO agents bring their own Foundry client (or none); skipped to avoid leaking
// the deployment list into a runtime that doesn't need it.
if !is_byo && !ctx.foundry_deployments.is_empty() {
openclaw_env
.push(json!({"name": "FOUNDRY_DEPLOYMENTS", "value": &ctx.foundry_deployments}));
}
// Inject Foundry Agent ID if set in status (for tools needing agent runs)
if let Some(ref agent_id) = sandbox
.status
.as_ref()
.and_then(|s| s.foundry_agent_id.clone())
// Inject Foundry Agent ID if set in status (for tools needing agent runs).
// BYO doesn't go through the OpenClaw plugin path; skipped.
if !is_byo
&& let Some(ref agent_id) = sandbox
.status
.as_ref()
.and_then(|s| s.foundry_agent_id.clone())
&& !agent_id.is_empty()
{
openclaw_env.push(json!({"name": "FOUNDRY_AGENT_ID", "value": agent_id}));
}
// Signal configured Foundry agent tools
if let Some(ref tools) = agent_config.tools
// Signal configured Foundry agent tools (OpenClaw plugin reads this).
if !is_byo
&& let Some(ref tools) = agent_config.tools
&& !tools.is_empty()
{
openclaw_env.push(json!({"name": "FOUNDRY_AGENT_TOOLS", "value": tools.join(",")}));
Expand Down Expand Up @@ -880,6 +900,42 @@ async fn reconcile(sandbox: Arc<ClawSandbox>, ctx: Arc<Context>) -> Result<Actio
}
}

// S10.A2.b: append `plan.raw_env` entries (BYO `valueFrom` etc.).
// The producer guarantees these have a `name` field; we apply the
// same reserved-prefix / NUL / dup filter to the `name` only —
// the `valueFrom` payload itself is rendered verbatim.
if !runtime_plan.raw_env.is_empty() {
const RESERVED_PREFIXES: &[&str] =
&["AGT_", "FOUNDRY_AGENT_", "AZURE_", "IMDS_", "AZURECLAW_"];
let mut existing: std::collections::HashSet<String> = openclaw_env
.iter()
.filter_map(|v| v.get("name").and_then(|n| n.as_str()).map(String::from))
.collect();
for entry in &runtime_plan.raw_env {
let Some(name) = entry.get("name").and_then(|n| n.as_str()) else {
tracing::warn!("rawEnv: entry missing `name`, skipping");
continue;
};
if name.is_empty()
|| !name.chars().all(|c| c.is_ascii_alphanumeric() || c == '_')
|| name.chars().next().is_some_and(|c| c.is_ascii_digit())
{
tracing::warn!(key = %name, "rawEnv: invalid env var name, skipping");
continue;
}
if RESERVED_PREFIXES.iter().any(|p| name.starts_with(p)) {
tracing::warn!(key = %name, "rawEnv: key uses reserved prefix, skipping");
continue;
}
if existing.contains(name) {
tracing::debug!(key = %name, "rawEnv: overridden by reconciler, skipping");
continue;
}
openclaw_env.push(entry.clone());
existing.insert(name.to_string());
}
}

// Build the inference-router env array
let mut router_env = vec![
json!({"name": "AZURE_OPENAI_ENDPOINT", "value": &ctx.openai_endpoint}),
Expand Down Expand Up @@ -919,6 +975,87 @@ async fn reconcile(sandbox: Arc<ClawSandbox>, ctx: Arc<Context>) -> Result<Actio
router_env.push(json!({"name": "EGRESS_LEARN_MODE", "value": "true"}));
}

// ── Agent container ──────────────────────────────────────────
// S10.A2.b: branch the agent container shape on the runtime
// kind. OpenClaw container = "openclaw" with gateway port 18789
// + admin-token mount (plugin pushes trust to router after
// KNOCK). BYO container = "agent" with no port, no admin-token
// mount, command/args from the runtime plan. Everything else
// (env, security context, volumes, probes, resources) is
// identical across runtimes — they're platform contract,
// controller-enforced.
let agent_container_name = if is_byo { "agent" } else { "openclaw" };
let agent_resources = spec
.resources
.as_ref()
.map(|r| {
json!({
"requests": r.requests,
"limits": r.limits,
})
})
.unwrap_or(json!({
"requests": {"cpu": "500m", "memory": "1Gi"},
"limits": {"cpu": "2", "memory": "4Gi"},
}));
let mut agent_volume_mounts = vec![
json!({"name": "sandbox-data", "mountPath": "/sandbox"}),
json!({"name": "tmp", "mountPath": "/tmp"}),
];
if !is_byo {
// OpenClaw plugin needs admin token to authenticate trust
// mutations after KNOCK handshakes (pushTrustToRouter).
// BYO does not run the plugin — mount is omitted to keep the
// attack surface narrow (defense-in-depth § the principle of
// least privilege).
agent_volume_mounts.push(json!({
"name": "admin-token",
"mountPath": "/etc/azureclaw/secrets",
"readOnly": true,
}));
}
let mut agent_container = json!({
"name": agent_container_name,
"image": image,
"imagePullPolicy": pull_policy,
"env": openclaw_env,
"envFrom": [
{"secretRef": {"name": format!("{}-credentials", name), "optional": true}}
],
"securityContext": {
"runAsUser": 1000,
"allowPrivilegeEscalation": sandbox_config.allow_privilege_escalation,
"readOnlyRootFilesystem": sandbox_config.read_only_root_filesystem,
"capabilities": {"drop": ["ALL"]}
},
"volumeMounts": agent_volume_mounts,
"resources": agent_resources,
"livenessProbe": {
"exec": {
"command": ["sh", "-c", "test -f /proc/1/status"]
},
"initialDelaySeconds": 15,
"periodSeconds": 30
},
"readinessProbe": {
"exec": {
"command": ["sh", "-c", "test -f /proc/1/status"]
},
"initialDelaySeconds": 5,
"periodSeconds": 10
}
});
if !is_byo {
// OpenClaw gateway port (used by `azureclaw connect` port-forward).
agent_container["ports"] = json!([{"containerPort": 18789, "name": "gateway"}]);
}
if let Some(cmd) = &runtime_plan.command {
agent_container["command"] = json!(cmd);
}
if let Some(args) = &runtime_plan.args {
agent_container["args"] = json!(args);
}

// Build the pod spec — runtimeClassName only set for Kata (confidential)
let mut pod_spec = json!({
"serviceAccountName": "sandbox",
Expand Down Expand Up @@ -982,50 +1119,7 @@ async fn reconcile(sandbox: Arc<ClawSandbox>, ctx: Arc<Context>) -> Result<Actio
}
}],
"containers": [
{
"name": "openclaw",
"image": image,
"imagePullPolicy": pull_policy,
"ports": [{"containerPort": 18789, "name": "gateway"}],
"env": openclaw_env,
"envFrom": [
{"secretRef": {"name": format!("{}-credentials", name), "optional": true}}
],
"securityContext": {
"runAsUser": 1000,
"allowPrivilegeEscalation": sandbox_config.allow_privilege_escalation,
"readOnlyRootFilesystem": sandbox_config.read_only_root_filesystem,
"capabilities": {"drop": ["ALL"]}
},
"volumeMounts": [
{"name": "sandbox-data", "mountPath": "/sandbox"},
{"name": "tmp", "mountPath": "/tmp"},
// Plugin needs admin token to authenticate trust
// mutations after KNOCK handshakes (pushTrustToRouter).
{"name": "admin-token", "mountPath": "/etc/azureclaw/secrets", "readOnly": true}
],
"resources": spec.resources.as_ref().map(|r| json!({
"requests": r.requests,
"limits": r.limits
})).unwrap_or(json!({
"requests": {"cpu": "500m", "memory": "1Gi"},
"limits": {"cpu": "2", "memory": "4Gi"}
})),
"livenessProbe": {
"exec": {
"command": ["sh", "-c", "test -f /proc/1/status"]
},
"initialDelaySeconds": 15,
"periodSeconds": 30
},
"readinessProbe": {
"exec": {
"command": ["sh", "-c", "test -f /proc/1/status"]
},
"initialDelaySeconds": 5,
"periodSeconds": 10
}
},
agent_container,
{
"name": "inference-router",
"image": &ctx.inference_router_image,
Expand Down
Loading
Loading