feat(skills): request-scoped secrets for skills (closes #3861)

[fancyboi999]

What & why

Closes #3861. Lets a caller pass per-request, short-lived end-user credentials (e.g. an ERP token, a per-user cloud API key) to a skill's sandbox scripts without the value entering the prompt, tool arguments, the executed command string, traces, the checkpoint, or the run record — the capability the issue asks for, and the skill-side counterpart of the still-open #3322 (MCP per-user credentials).

The design follows LangGraph's "out-of-band runtime dependency injection" pattern (the same runtime.context DeerFlow already uses) and Anthropic Agent Skills' "credentials live in the execution environment, not the prompt" model.

How it works

caller run request
  └─ config.context.secrets = { "ERP_TOKEN": "<short-lived>" }   ← out-of-band, never a message
       └─ build_run_config → runtime.context["secrets"]  (not mirrored into configurable)
            └─ /skill-name activates → SkillActivationMiddleware resolves the skill's
               declared `required-secrets` against context.secrets → per-run injection set
                 └─ bash tool → Sandbox.execute_command(command, env=…)
                      ├─ LocalSandbox: subprocess env (platform creds scrubbed first)
                      └─ AioSandbox:  bash.exec(env=…) on a fresh session
                           └─ skill script reads $ERP_TOKEN  (value never in any message)

• Declare — a skill lists what it needs in SKILL.md frontmatter: required-secrets: (string list or {name, optional}). name is both the context.secrets key and the env var exposed to the script.
• Carry — the caller sends values in the run request's context.secrets (out-of-band). Reuses the existing context passthrough; never mirrored into configurable.
• Bind — only the slash-activated skill's declared secrets are injected, scoped to that run.
• Inject — Sandbox.execute_command gains an env parameter (Local + AIO). env=None is fully backward-compatible.
• Scrub — execute_command no longer leaks the Gateway's os.environ to skill subprocesses: secret-looking names (*KEY*/*SECRET*/*TOKEN*/*PASSWORD*/*CREDENTIAL*/*DSN* + a connection-string denylist like DATABASE_URL/REDIS_URL/GH_PAT) are dropped. A skill can only receive what the caller explicitly supplies; a caller-supplied value wins over a scrubbed host value of the same name (the per-user-key-overrides-shared-key case).

Leak surfaces — all sealed (verified)

Surface	How
prompt	value travels via runtime.context, never a message
trace	tracing/metadata.py never copies context; redact_secret_context_keys available defensively
checkpoint	secrets live on runtime.context, not graph state
audit	the activation journal records secret names only
bash stdout	mask_secret_values redacts injected values from tool output before it re-enters context
run record + run API	start_run persists redact_config_secrets(body.config), so runs.kwargs_json and RunResponse.kwargs never carry the secret
inherited env	platform creds scrubbed from the sandbox subprocess

Verification (evidence, not assertion)

• Unit/integration tests — new tests/test_skill_request_scoped_secrets.py (parser, carrier, activation binding, env scrub, the leak surfaces, a real-subprocess end-to-end) + updated tests/test_gateway_services.py and tests/test_local_sandbox_encoding.py.
• Full backend suite (CI-equivalent): green, all regression tests pass; blocking-IO gate passes.
• Real gateway end-to-end (real LLM + skill activation + bash): a request with context.secrets ran a skill whose script used the injected secret; the value reached the sandbox subprocess and appeared in none of the surfaces above (verified by scanning the run response, the run API, the trace events, and the full per-thread data dir).
• Real third-party cloud call: a skill called an external cloud API using a request-scoped API key supplied via context.secrets. The call succeeded (HTTP 200); a same-named key that happened to exist in the Gateway env was scrubbed and never used; the caller's key leaked nowhere. (Credential redacted from this report.)

Notes for reviewers

• This PR also fixes a pre-existing latent bug surfaced during end-to-end testing: InputSanitizationMiddleware wraps user input in --- BEGIN USER INPUT --- markers, but only preserved the original text for upload / IM-channel messages — so slash-skill activation silently failed for plain text messages. The sanitizer now preserves the original text in ORIGINAL_USER_CONTENT_KEY (additive; sanitization behaviour unchanged), which get_original_user_content_text already expected. Slash activation now works for all messages.
• build_run_config now sets configurable.thread_id on the context path too — the checkpointer requires it, and the prior behaviour produced a KeyError: 'thread_id' for context-driven runs.

Scope / non-goals

• Only /slash-activated skills receive secrets (autonomously invoked enabled skills do not).
• No persistence/vaulting — secrets are request-lived only.
• The MCP per-user-credential gap (MCP servers lack per-user credential isolation — all users share the same API keys #3322) is a sibling and is not addressed here, though context.secrets is designed to be reusable for it.

Not verified

• No live Langfuse/LangSmith backend was exercised; trace-cleanliness was verified against DeerFlow's metadata builder and the in-graph callback payloads, not a live tracing UI.
• The AIO/Docker bash.exec(env=…) path is covered by unit tests against the SDK signature/return shape; it was not run against a live Docker sandbox (local sandbox was used for the live end-to-end).

[yong326]

@fancyboi999 Code conflicts, please help resolve

[willem-bd]

Review findings from an automated pass. Focus: correctness bugs surfaced by the new env kwarg path, the secrets-carrier lifecycle, and output redaction. See inline comments.

[willem-bd]

bash.exec is called with no session_id, so consecutive env-bearing bash calls in the same skill activation share no cwd / env exports / venv state.

A typical skill run issues sequential bash calls like:

cd /mnt/user-data/workspace && source .venv/bin/activate && pip install foo
python run.py

Without secrets, both go through shell.exec_command on the persistent session and the second call sees the first call's cwd/PATH. With secrets injected, both go through _execute_with_env and each gets a fresh auto-created session: the second call starts in $HOME, sees no venv, no foo, and fails with No such file or command not found.

I understand the design tension (a reused session_id would keep secrets scoped-to-thread rather than scoped-to-call). Two options worth considering:

1. Use a stable per-thread session_id for env-bearing calls and rely on the fact that bash.exec's env overrides the session env per command (verify with the SDK — if that guarantee holds, secrets don't persist across calls).
2. If session reuse isn't safe, at least pass exec_dir (e.g. the thread's workspace) so relative paths behave the same way as on the legacy path.

[fancyboi999]

Intentionally keeping the fresh-session choice and documenting why on _execute_with_env.

The core tension is real, but a shared session_id for env-bearing calls would reintroduce the leak surface this PR exists to close: if bash.exec's env merges into the session env rather than being strictly per-command (the SDK does not contractually guarantee per-command isolation), a request-scoped secret would ride the session into later commands in the same skill — exactly the scoped-to-call invariant the design preserves. Without that SDK guarantee, reusing a session_id is a regression risk I'm not willing to take in a security-focused change.

The fresh-session model matches LocalSandbox, where each execute_command is already a fresh subprocess (cwd is recovered via the command-prefix; sourced venv / exports never cross calls either). So AIO-with-secrets is now consistent with the local backend, not an outlier.

Practical mitigation for the cwd/venv case, now stated in the docstring: skills that need setup must fold it into a single command (e.g. cd /mnt/user-data/workspace && source .venv/bin/activate && python run.py). That's the same pattern LocalSandbox skills already need.

If a later SDK version documents per-command env isolation on a reused session, exec_dir (for cwd) or a stable session_id become safe upgrades — happy to revisit then.

[starhobby]

When will 2.1.0 plan to launch?

[WillemJiang]

When will 2.1.0 plan to launch?

We will release it this month.

[fancyboi999]

Review fixes — push 926c73e5

Addressed the review from @willem-bd. CI is green across the board (backend-unit-tests, backend-blocking-io, lint-backend/frontend, Layer 1/2 golden).

Resolved (5)

• E2B bash breakage (tools.py) — E2BSandbox.execute_command now accepts env/timeout and routes them to commands.run(envs=, timeout=). The bash tool passes env= unconditionally, so the prior command-only signature raised TypeError on every e2b bash call. env=None is fully backward-compatible (no envs kwarg sent). Regression tests added.
• Stale active secrets (skill_activation_middleware.py) — _apply_skill_secrets now clears the active-secret set before each activation, so a later skill in the same run can never inherit an earlier skill's injection set. Two regression tests cover both reported paths (next skill declares nothing; caller omits the required value).
• hard_timeout vs no_change_timeout (aio_sandbox.py) — dedicated _DEFAULT_HARD_TIMEOUT for the env path with a docstring noting bash.exec exposes no idle/no-change timeout. Kept at 600 to stay aligned with the legacy idle budget; named independently so the two call sites evolve without silently altering each other.
• Short-value mask corruption (tools.py) — mask_secret_values skips values below _MIN_MASK_LENGTH (8 chars) so a short declared secret (region code, numeric id) cannot shred exit codes / timestamps / sizes in tool output. The secret is still injected into the subprocess; only the output mask skips short values.
• ErrorObservation retry (aio_sandbox.py) — the env path now detects _ERROR_OBSERVATION_SIGNATURE and retries on another fresh session, sharing the legacy persistent-shell recovery contract. Refactored into a _run_bash_exec helper so the retry stays on the env-bearing path (secrets never enter a shell session).

Left open (1) — intentional

• session_id reuse on the env path (aio_sandbox.py) — I kept the fresh-session choice. A shared session_id could let request-scoped secrets ride the session env into later commands, and the SDK does not contractually guarantee per-command env isolation, so reuse is a leak-surface regression I'm not willing to take in a security-focused change. Fresh session matches the LocalSandbox model (each execute_command is already a fresh subprocess there). The trade-off (consecutive env-bearing calls do not share cwd/venv/exports) is documented on _execute_with_env; skills fold setup into a single command. Happy to revisit if a future SDK version documents per-command env isolation on a reused session. Full rationale is in the thread.