fix(backend): unblock orchestrator dry-run end-to-end (canonical model + SDK auth) - #13180
Merged
Merged
Conversation
…del.value before injecting into OrchestratorBlock input The previous SECRT-2368 hotfix (#13177) taught ``LlmModel._missing_`` to resolve OpenRouter alias slugs like ``anthropic/claude-haiku-4-5`` back to the existing ``CLAUDE_4_5_HAIKU`` enum member, fixing Pydantic validation. But OrchestratorBlock inputs are validated by ``OrchestratorBlock.validate_data`` *first*, which runs ``jsonschema.validate`` against a schema whose ``enum`` is the literal list of ``LlmModel.value`` strings. The alias map only applies to Python-runtime enum lookups — it does not surface in the generated JSON Schema — so the OR-slug was rejected with:: 'anthropic/claude-haiku-4-5' is not one of ['o3-mini', ..., 'claude-haiku-4-5-20251001', ...] Failed validating 'enum' in schema['properties']['model'] Reproduced on dev-builder for every orchestrator dry-run after the deploy that included #13177. Fix: in ``prepare_dry_run``, run the configured simulator model through ``LlmModel(...).value`` before injecting into ``input["model"]``. This produces the canonical snapshot value (e.g. ``"claude-haiku-4-5-20251001"``), which IS in the JSON Schema ``enum``, so validation passes. The downstream Anthropic-compat endpoint OpenRouter exposes to the orchestrator's SDK accepts both the snapshot ID and the OR slug (verified empirically), so no other code path needs to change. Two new tests: - ``test_orchestrator_uses_simulation_model`` now asserts the injected model is a canonical ``LlmModel.value`` (in ``{m.value for m in LlmModel}``), not just LlmModel-parseable. - ``test_orchestrator_input_passes_jsonschema_validation`` calls ``OrchestratorBlock.input_schema.validate_data`` on the simulator's actual output to lock in end-to-end schema acceptance. End-to-end validated against real OpenRouter creds in a local script: bug reproduces with the OR slug, fix resolves at all three layers (jsonschema, Pydantic, OpenRouter Anthropic-compat), and a real ``messages.create`` call against ``ANTHROPIC_BASE_URL=https://openrouter.ai/api`` with the canonical snapshot returns a non-empty completion.
github-project-automation
Bot
moved this to 🆕 Needs initial review
in AutoGPT development kanban
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughCanonicalizes simulator-configured model slugs via ChangesDry-Run Simulator Model Resolution
Orchestrator OpenRouter Env Routing
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested labels
Suggested reviewers
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
🔍 PR Overlap DetectionThis check compares your PR against all other open PRs targeting the same branch to detect potential merge conflicts early. 🔴 Merge Conflicts DetectedThe following PRs have been tested and will have merge conflicts if merged after this PR. Consider coordinating with the authors.
🟢 Low Risk — File Overlap OnlyThese PRs touch the same files but different sections (click to expand)
Summary: 2 conflict(s), 0 medium risk, 3 low risk (out of 5 PRs with file overlap) Auto-generated on push. Ignores: |
CI lint step uses black; my local ruff format pass missed three whitespace differences. No semantic change.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@autogpt_platform/backend/backend/executor/simulator_test.py`:
- Around line 199-201: The test contains a redundant local import and a
function-local import that should be at module scope: remove the in-function
line "from unittest.mock import patch" (it's already imported at module scope)
and move "from backend.blocks.orchestrator import OrchestratorBlock" out of the
test function to the top-level imports in simulator_test.py so OrchestratorBlock
is imported alongside the other module imports; update any references in the
test to continue using OrchestratorBlock and ensure no other local imports
remain.
In `@autogpt_platform/backend/backend/executor/simulator.py`:
- Line 430: prepare_dry_run can crash if LlmModel(_simulator_model()).value
raises ValueError for a malformed CHAT_SIMULATION_MODEL; wrap the creation of
sim_model in a try/except that catches ValueError, logs a warning including the
offending config, and falls back to a safe default (e.g., a known-good model or
None) so dry-run continues; update the block around LlmModel, _simulator_model,
and sim_model to implement the guarded fallback and use the module's existing
logger to record the error.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 803ec4c9-fa24-4429-bd71-89d7466ce0fd
📒 Files selected for processing (2)
autogpt_platform/backend/backend/executor/simulator.pyautogpt_platform/backend/backend/executor/simulator_test.py
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (11)
- GitHub Check: check API types
- GitHub Check: Seer Code Review
- GitHub Check: end-to-end tests
- GitHub Check: test (3.11)
- GitHub Check: type-check (3.11)
- GitHub Check: test (3.12)
- GitHub Check: test (3.13)
- GitHub Check: type-check (3.13)
- GitHub Check: Check PR Status
- GitHub Check: Analyze (python)
- GitHub Check: Analyze (typescript)
🧰 Additional context used
📓 Path-based instructions (3)
autogpt_platform/backend/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
autogpt_platform/backend/**/*.py: Use Python 3.11 (required; managed by Poetry via pyproject.toml) for backend development
Always run 'poetry run format' (Black + isort) before linting in backend development
Always run 'poetry run lint' (ruff) after formatting in backend development
autogpt_platform/backend/**/*.py: Usepoetry run ...command for executing Python package dependencies
Use top-level imports only — avoid local/inner imports except for lazy imports of heavy optional dependencies likeopenpyxl
Use absolute imports withfrom backend.module import ...for cross-package imports; single-dot relative imports are acceptable for sibling modules within the same package; avoid double-dot relative imports
Do not use duck typing — avoidhasattr/getattr/isinstancefor type dispatch; use typed interfaces/unions/protocols instead
Use Pydantic models over dataclass/namedtuple/dict for structured data
Do not use linter suppressors — no# type: ignore,# noqa,# pyright: ignore; fix the type/code instead
Prefer list comprehensions over manual loop-and-append patterns
Use early return with guard clauses first to avoid deep nesting
Use%sfor deferred interpolation indebuglog statements for efficiency; use f-strings elsewhere for readability (e.g.,logger.debug("Processing %s items", count)vslogger.info(f"Processing {count} items"))
Sanitize error paths by usingos.path.basename()in error messages to avoid leaking directory structure
Be aware of TOCTOU (Time-Of-Check-Time-Of-Use) issues — avoid check-then-act patterns for file access and credit charging
Usetransaction=Truefor Redis pipelines to ensure atomicity on multi-step operations
Usemax(0, value)guards for computed values that should never be negative
Keep files under ~300 lines; if a file grows beyond this, split by responsibility (extract helpers, models, or a sub-module into a new file)
Keep functions under ~40 lines; extract named helpers when a function grows longer
...
Files:
autogpt_platform/backend/backend/executor/simulator_test.pyautogpt_platform/backend/backend/executor/simulator.py
autogpt_platform/{backend,autogpt_libs}/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
Format Python code with
poetry run format
Files:
autogpt_platform/backend/backend/executor/simulator_test.pyautogpt_platform/backend/backend/executor/simulator.py
autogpt_platform/backend/**/*_test.py
📄 CodeRabbit inference engine (autogpt_platform/backend/AGENTS.md)
autogpt_platform/backend/**/*_test.py: Use pytest with snapshot testing for API responses
Colocate test files with source files using*_test.pynaming convention
Mock at boundaries — mock where the symbol is used, not where it's defined; after refactoring, update mock targets to match new module paths
UseAsyncMockfromunittest.mockfor async functions in tests
When writing tests, use Test-Driven Development (TDD): write failing tests marked with@pytest.mark.xfailbefore implementation, then remove the marker once the implementation is complete
When creating snapshots in tests, usepoetry run pytest path/to/test.py --snapshot-update; always review snapshot changes withgit diffbefore committing
Files:
autogpt_platform/backend/backend/executor/simulator_test.py
🧠 Learnings (8)
📚 Learning: 2026-02-26T17:02:22.448Z
Learnt from: Pwuts
Repo: Significant-Gravitas/AutoGPT PR: 12211
File: .pre-commit-config.yaml:160-179
Timestamp: 2026-02-26T17:02:22.448Z
Learning: Keep the pre-commit hook pattern broad for autogpt_platform/backend to ensure OpenAPI schema changes are captured. Do not narrow to backend/api/ alone, since the generated schema depends on Pydantic models across multiple directories (backend/data/, backend/blocks/, backend/copilot/, backend/integrations/, backend/util/). Narrowing could miss schema changes and cause frontend type desynchronization.
Applied to files:
autogpt_platform/backend/backend/executor/simulator_test.pyautogpt_platform/backend/backend/executor/simulator.py
📚 Learning: 2026-03-05T15:42:08.207Z
Learnt from: ntindle
Repo: Significant-Gravitas/AutoGPT PR: 12297
File: .claude/skills/backend-check/SKILL.md:14-16
Timestamp: 2026-03-05T15:42:08.207Z
Learning: In Python files under autogpt_platform/backend (recursively), rely on poetry run format to perform formatting (Black + isort) and linting (ruff). Do not run poetry run lint as a separate step after poetry run format, since format already includes linting checks.
Applied to files:
autogpt_platform/backend/backend/executor/simulator_test.pyautogpt_platform/backend/backend/executor/simulator.py
📚 Learning: 2026-03-16T16:35:40.236Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 12440
File: autogpt_platform/backend/backend/api/features/workflow_import.py:54-63
Timestamp: 2026-03-16T16:35:40.236Z
Learning: Avoid using the word 'competitor' in public-facing identifiers and text. Use neutral naming for API paths, model names, function names, and UI text. Examples: rename 'CompetitorFormat' to 'SourcePlatform', 'convert_competitor_workflow' to 'convert_workflow', '/competitor-workflow' to '/workflow'. Apply this guideline to files under autogpt_platform/backend and autogpt_platform/frontend.
Applied to files:
autogpt_platform/backend/backend/executor/simulator_test.pyautogpt_platform/backend/backend/executor/simulator.py
📚 Learning: 2026-03-31T15:37:38.626Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 12623
File: autogpt_platform/backend/backend/copilot/tools/agent_generator/fixer.py:37-47
Timestamp: 2026-03-31T15:37:38.626Z
Learning: When validating/constructing Anthropic API model IDs in Significant-Gravitas/AutoGPT, allow the hyphen-separated Claude Opus 4.6 model ID `claude-opus-4-6` (it corresponds to `LlmModel.CLAUDE_4_6_OPUS` in `autogpt_platform/backend/backend/blocks/llm.py`). Do NOT require the dot-separated form in Anthropic contexts. Only OpenRouter routing variants should use the dot separator (e.g., `anthropic/claude-opus-4.6`); `claude-opus-4-6` should be treated as correct when passed to Anthropic, and flagged only if it’s used in the OpenRouter path where the dot form is expected.
Applied to files:
autogpt_platform/backend/backend/executor/simulator_test.pyautogpt_platform/backend/backend/executor/simulator.py
📚 Learning: 2026-04-15T02:43:36.890Z
Learnt from: ntindle
Repo: Significant-Gravitas/AutoGPT PR: 12780
File: autogpt_platform/backend/backend/copilot/tools/workspace_files.py:0-0
Timestamp: 2026-04-15T02:43:36.890Z
Learning: When reviewing Python exception handlers, do not flag `isinstance(e, X)` checks as dead/unreachable if the caught exception `X` is a subclass of the exception type being handled. For example, if `X` (e.g., `VirusScanError`) inherits from `ValueError` (directly or via an intermediate class) and it can be raised within an `except ValueError:` block, then `isinstance(e, X)` inside that handler is reachable and should not be treated as dead code.
Applied to files:
autogpt_platform/backend/backend/executor/simulator_test.pyautogpt_platform/backend/backend/executor/simulator.py
📚 Learning: 2026-04-22T11:46:04.431Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 12881
File: autogpt_platform/backend/backend/copilot/config.py:0-0
Timestamp: 2026-04-22T11:46:04.431Z
Learning: Do not flag the Claude Sonnet 4.6 model ID as incorrect when it uses the project’s established hyphenated convention: `anthropic/claude-sonnet-4-6`. This hyphen form is the intentional, production convention and should be treated as valid (including in files like llm.py, blocks tests, reasoning.py, `_is_anthropic_model` tests, and config defaults). Note that OpenRouter also accepts the dot variant `anthropic/claude-sonnet-4.6`, so either form may be tolerated, but `anthropic/claude-sonnet-4-6` should be considered the standard to match project usage.
Applied to files:
autogpt_platform/backend/backend/executor/simulator_test.pyautogpt_platform/backend/backend/executor/simulator.py
📚 Learning: 2026-04-22T11:46:12.892Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 12881
File: autogpt_platform/backend/backend/copilot/baseline/service.py:322-332
Timestamp: 2026-04-22T11:46:12.892Z
Learning: In this codebase (Significant-Gravitas/AutoGPT), OpenRouter-routed Anthropic model IDs should use the hyphen-separated convention (e.g., `anthropic/claude-sonnet-4-6`, `anthropic/claude-opus-4-6`). Although OpenRouter may accept both hyphen and dot variants, treat the hyphen-separated form as the intended, correct codebase-wide convention and do not flag it as an error. Only flag the dot-separated variant (e.g., `anthropic/claude-sonnet-4.6`) as incorrect when reviewing/validating model ID strings for OpenRouter-routed Anthropic models.
Applied to files:
autogpt_platform/backend/backend/executor/simulator_test.pyautogpt_platform/backend/backend/executor/simulator.py
📚 Learning: 2026-05-07T18:48:14.242Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 13040
File: autogpt_platform/backend/backend/blocks/llm.py:0-0
Timestamp: 2026-05-07T18:48:14.242Z
Learning: In this repository, isort may split imports from the same module into separate blocks when some imports are aliased (e.g., `from module import X as Y`) and others are not. Preserve the two-block layout when it results from isort (such as keeping `from openai.types.chat import ChatCompletion as OpenAIChatCompletion` separate from non-aliased imports from `openai.types.chat`). Do not treat that split as a style issue during review; merging them into a single block can fail CI with `Imports are incorrectly sorted and/or formatted`.
Applied to files:
autogpt_platform/backend/backend/executor/simulator_test.pyautogpt_platform/backend/backend/executor/simulator.py
🔇 Additional comments (2)
autogpt_platform/backend/backend/executor/simulator.py (1)
42-42: LGTM!Also applies to: 415-429
autogpt_platform/backend/backend/executor/simulator_test.py (1)
174-185: LGTM!Also applies to: 191-198, 203-230
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## dev #13180 +/- ##
==========================================
- Coverage 71.48% 71.46% -0.02%
==========================================
Files 2221 2221
Lines 167216 167249 +33
Branches 17048 17048
==========================================
- Hits 119533 119529 -4
- Misses 44119 44161 +42
+ Partials 3564 3559 -5
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
…-key to OpenRouter
The EXTENDED_THINKING SDK path with credentials.provider == "open_router"
previously set ``sdk_env["ANTHROPIC_API_KEY"] = ""`` to "force the CLI to
use AUTH_TOKEN". The Claude Agent SDK merges this dict on top of
``os.environ`` ([subprocess_cli.py:402](.venv claude_agent_sdk transport)),
so the spawned CLI sees ``ANTHROPIC_API_KEY=`` (present-but-empty) and
emits ``x-api-key:`` (empty header value) on the wire. OpenRouter
rejects with::
Error code: 401 — {'type': 'authentication_error',
'message': 'invalid x-api-key'}
This was latent before SECRT-2368 because every orchestrator dry-run
died at OrchestratorBlock.Input validation; #13177/#13180 made
execution reach the auth wiring and the 401 surfaced on dev-builder.
Fix: set ``ANTHROPIC_API_KEY`` to the same OpenRouter key. OpenRouter's
Anthropic-compat endpoint accepts either ``x-api-key`` or
``Authorization: Bearer`` with the OR key, so whichever the CLI happens
to send (or both) is a valid credential. We still need an explicit
value rather than omission because the SDK's merge would otherwise let
an inherited platform ``ANTHROPIC_API_KEY`` (e.g. a deployment's
direct-Anthropic key) leak through and 401 for a different reason.
majdyz
changed the title
12 tasks
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
EXTENDED_THINKING enters the Claude Agent SDK subprocess path, which has its own constraints — Claude-only model whitelist + brittle env-var auth wiring (the latter caused the ``invalid x-api-key`` 401 addressed in the previous commit). Every quirk on that path is a separate failure mode for the dry-run smoke check. BUILT_IN is the OpenAI-SDK-with-tool-calling path; it already exercises the orchestrator's tool-dispatch loop well enough to surface graph-wiring bugs (the actual purpose of dry-run). Users keep EXTENDED_THINKING in production; only the preview switches. Adds ``test_orchestrator_forces_built_in_execution_mode`` to pin the invariant. The orchestrator SDK auth fix in the previous commit is kept — it still protects real-run users on EXTENDED_THINKING + OpenRouter credentials from the same 401.
…orts Two PR-review follow-ups from CodeRabbit and Sentry: 1. **Guard invalid CHAT_SIMULATION_MODEL overrides.** If the env var is set to an unmapped slug, ``LlmModel(_simulator_model()).value`` raises ``ValueError`` and aborts every Orchestrator dry-run. Wrap in a try/except that logs a warning and falls back to ``_DEFAULT_SIMULATOR_MODEL`` so dry-run keeps working. Added ``test_orchestrator_invalid_sim_model_override_falls_back_to_default`` to pin. 2. **Move test imports to module scope.** CodeRabbit flagged the new ``test_orchestrator_input_passes_jsonschema_validation`` and ``test_orchestrator_forces_built_in_execution_mode`` for using function-local ``patch`` (already at module scope) and ``OrchestratorBlock``/``ExecutionMode`` imports. Hoist them.
majdyz
commented
May 21, 2026
…ompat path Sentry caught a MEDIUM in PR #13180's BUILT_IN dry-run flow: the canonical `LlmModel.value` ("claude-haiku-4-5-20251001") is rejected by OpenRouter's `/v1/chat/completions` endpoint with HTTP 400 "not a valid model ID". OR's OpenAI-compat endpoint only accepts `<vendor>/<model>` slugs. The bug is broader than dry-run — any caller hitting the `open_router` branch in `llm.llm_call` (backend/blocks/llm.py:1228) with an Anthropic model would 400. The dry-run regression in PR #13180 just made it observable on the default dry-run path. Fix: - New `openrouter_model_id(llm_model)` helper that returns the model identifier OR's OpenAI-compat actually accepts. Three cases: 1. Anthropic 4.5 snapshot models — reverse `_OPENROUTER_ALIASES` to drop the `-YYYYMMDD` suffix. 2. Anthropic 4.6/4.7+ (no snapshot) — prepend `anthropic/` since canonical already matches OR slug after the prefix. 3. Non-Anthropic OR-routed models — canonical IS the slug already. - Replace `model=llm_model.value` with `model=openrouter_model_id(llm_model)` in the open_router branch. - `_CANONICAL_TO_OPENROUTER_SLUG` reverse map computed once at module load. New `TestOpenRouterModelId` pins the contract: - 4.5 anthropic snapshot → OR slug reversal - 4.6/4.7 anthropic → prefix-prepend - Non-anthropic OR-routed → identity - Round-trip through `LlmModel._missing_` for every anthropic/open_router enum member — locks in the inverse-of-_missing_ invariant so a future enum addition that breaks the round-trip trips at CI time. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sentry was right twice in a row on this PR: 1. The previous commit's `openrouter_model_id` helper was dead code for Claude. `_llm_call` dispatches on `llm_model.metadata.provider`, not on credential type, so Claude always lands in the `anthropic` branch (line 1057) which uses the Anthropic Python SDK against api.anthropic.com. The `open_router` branch (line 1218) is only reached by models whose `metadata.provider == "open_router"` (Gemini, Mistral, Kimi, etc.) — and those already carry their `<vendor>/<model>` slug as their canonical value, so the helper was always identity on the reachable call sites. 2. Forcing `execution_mode = BUILT_IN` in `prepare_dry_run` was a defence against EXTENDED_THINKING SDK quirks, but with this PR's model override (always Claude `sim_model`) and the orchestrator's OR auth-env fix (line 1670-1691), both quirks are addressed. Meanwhile the force-BUILT_IN line was routing OR+Claude dry-runs through `llm.llm_call`'s anthropic branch against api.anthropic.com with an OR key — a 401 nobody had tested. EXTENDED_THINKING via the SDK subprocess hits OR's Anthropic-compat (which accepts both canonical and slug per PR #13177's empirical probe), so it's the actually-working dry-run target for OR+Claude. This commit: - Reverts the `openrouter_model_id` helper + its reverse-map + its `TestOpenRouterModelId` test class. - Restores `model=llm_model.value` in the open_router branch. - Drops the `execution_mode = BUILT_IN` override from `prepare_dry_run` (and the corresponding `ExecutionMode` import). - Renames the corresponding test to `test_orchestrator_preserves_user_execution_mode` and flips its assertion to require the user's choice to flow through. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ser's pick)
Sentry caught a HIGH on the previous revert: letting the user's
execution_mode flow through breaks the default-BUILT_IN case for
OR+Claude. The OrchestratorBlock default is BUILT_IN. With dry-run's
model override (Claude) + the platform key being an OR key, BUILT_IN
routes through `llm.llm_call`'s anthropic branch (dispatch keys on
`llm_model.metadata.provider`, not credential type) against
api.anthropic.com with an OR key — 401.
EXTENDED_THINKING is the path that actually works for OR+Claude:
the SDK subprocess's OR-credential branch in `orchestrator.py:1670-1691`
sets `ANTHROPIC_BASE_URL` to OR's Anthropic-compat endpoint and
`ANTHROPIC_API_KEY` to the OR key (this PR's own auth fix). The SDK
path's two prerequisites (`metadata.provider in {anthropic, open_router}`
and `model.value.startswith("claude")`) are both satisfied because
the dry-run always overrides `model = sim_model = Claude Haiku`.
So the right move is to force EXTENDED_THINKING (not BUILT_IN, not
user's choice). The previous force-BUILT_IN was defending against
EXTENDED_THINKING SDK quirks that this same PR fixes — but BUILT_IN
has its own showstopper for OR+Claude (the api.anthropic.com 401),
which this PR did *not* fix. EXTENDED_THINKING avoids both classes
of bug.
Test renamed to `test_orchestrator_forces_extended_thinking_execution_mode`
and flipped to assert the override (user picks BUILT_IN, dry-run
overrides to EXTENDED_THINKING). The PR description's force-BUILT_IN
justification stays mostly valid but applies in the *opposite*
direction now — the SDK path is the safer dry-run target post this PR.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Author
E2E Test Report — Live Dev PreviewTested combined branch
Screenshots1 — Login page (dev-builder.agpt.co) 2 — Post-login state 3 — Copilot landing with preview banner 4 — Chat ready to receive prompt 5 — Prompt submitted 6 — Copilot response: orchestrator-based agent created 7 — Agent persisted to library 8 — Agent detail view 9 — Agent graph in builder (proves the saved config) Negative coverageThe copilot's first attempt during this session returned "Validation failed with 1 error — The OrchestratorBlock requires at least one downstream tool block", and recovered by adding an AITextGenerator passthrough. This is the expected validation error path (legitimate user-error case: an orchestrator without tools is invalid), distinct from the SDK auth / model-enum failures that PR #13180 fixed. It proves the validation pipeline still rejects malformed graphs as it should. VerdictThe dev preview is functionally healthy after the combined merge of PR #13179 + PR #13180. The OrchestratorBlock dry-run — the specific path that motivated PR #13180 — works against the default ( |
Merged
4 tasks
… (not EXTENDED_THINKING)
Two coupled changes that converge on the simulator config that
satisfies every routing constraint at once. Both replace earlier
attempts on this PR that fixed one path but broke another.
**Default model: Claude Haiku → Gemini Flash-Lite.** The
LLM-simulation path (``_call_llm_for_simulation``) hits OpenRouter's
OpenAI-compat endpoint with ``response_format=json_object`` and
``json.loads()`` the response. Claude via OR's OpenAI-compat wraps
JSON in markdown fences (``\`\`\`json\\n{...}\\n\`\`\```) — empirically
verified with curl — and ``json.loads`` trips on the leading backtick
with ``Expecting value: line 1 column 1 (char 0)``. User hit this on
dev-builder for every non-Orchestrator block in their graph during a
dry-run. Gemini emits raw JSON so the parse succeeds.
**Force BUILT_IN, not EXTENDED_THINKING.** Previous commit on this PR
forced EXTENDED_THINKING because Haiku-default + BUILT_IN would route
through ``llm.llm_call``'s anthropic branch (dispatch keys on
``llm_model.metadata.provider``) against ``api.anthropic.com`` with
the platform OR key → 401. Switching the default to Flash-Lite
flips the optimal mode: Gemini has ``metadata.provider == "open_router"``
so BUILT_IN dispatches through the open_router branch (OpenAI SDK
against openrouter.ai) — works. Meanwhile EXTENDED_THINKING imposes
``model.value.startswith("claude")`` which Flash-Lite fails, so the
override has to be BUILT_IN under the new default.
Test ``test_orchestrator_forces_extended_thinking_execution_mode``
renamed + flipped to ``test_orchestrator_forces_built_in_execution_mode``;
``TestDefaultSimulatorModel`` pin updated + added
``test_default_provider_is_open_router`` to lock in the routing
constraint at unit-test time.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
majdyz
added a commit
that referenced
this pull request
May 21, 2026
github-project-automation
Bot
moved this from 🆕 Needs initial review
to ✅ Done
in AutoGPT development kanban
6 tasks
psbuilds
pushed a commit
to psbuilds/AutoGPT
that referenced
this pull request
May 28, 2026
…l + SDK auth) (Significant-Gravitas#13180) ### Why / What / How **Why:** Two follow-ups landed after PR Significant-Gravitas#13177 deployed and the user exercised the orchestrator dry-run on dev-builder. 1. **JSON Schema validation gap.** Significant-Gravitas#13177 taught `LlmModel._missing_` to resolve `anthropic/claude-haiku-4-5` → `CLAUDE_4_5_HAIKU`. That fixed Pydantic, but `OrchestratorBlock.Input` is validated by `validate_data` → `jsonschema.validate` *first*, against a schema whose `enum` is the literal list of `LlmModel.value` strings. The alias map is a Python runtime hook — it does not surface in the generated JSON Schema. So the OR-slug was rejected with `'anthropic/claude-haiku-4-5' is not one of [...]`. 2. **SDK auth: empty `x-api-key`.** Once jsonschema started passing, execution reached the EXTENDED_THINKING SDK path in [orchestrator.py:1670-1674](autogpt_platform/backend/backend/blocks/orchestrator.py#L1670). That code sets `sdk_env["ANTHROPIC_API_KEY"] = ""` to "force the CLI to use AUTH_TOKEN." But the Claude Agent SDK merges `options.env` on top of `os.environ` ([subprocess_cli.py:402](file:claude_agent_sdk/_internal/transport/subprocess_cli.py#L402)), so the spawned CLI sees `ANTHROPIC_API_KEY=` (present-but-empty) and emits `x-api-key:` (empty header) on the wire. OpenRouter returns `401 invalid x-api-key`. This bug was latent until Significant-Gravitas#13177 — every orchestrator dry-run died at jsonschema before reaching the auth wiring. **What:** - `simulator.py`: translate the configured simulator model to its canonical `LlmModel.value` via `LlmModel(_simulator_model()).value` before injecting into `input["model"]`. The OR-slug default (`anthropic/claude-haiku-4-5`) becomes `claude-haiku-4-5-20251001` — which IS in the JSON Schema enum — so `validate_data` passes. Downstream OpenRouter's Anthropic-compat endpoint accepts both forms, so no further translation is needed. - `orchestrator.py`: in the `provider == "open_router"` SDK branch, set `ANTHROPIC_API_KEY` to the same OpenRouter key (instead of `""`). OpenRouter accepts either `x-api-key` or `Authorization: Bearer` with the OR key, so whichever the CLI sends is valid. Explicit set (not omission) is required because the SDK's merge would otherwise let an inherited platform `ANTHROPIC_API_KEY` leak through. - `simulator_test.py`: strengthen `test_orchestrator_uses_simulation_model` to assert the injected model is in `{m.value for m in LlmModel}`; add `test_orchestrator_input_passes_jsonschema_validation` that calls `validate_data` on `prepare_dry_run`'s output (locks in the exact regression the user hit). **How:** End-to-end proof for the model translation, against real OpenRouter (4-step script): ``` === Step 1: reproduce the bug with the OR slug === ✓ validate_data REJECTS 'anthropic/claude-haiku-4-5' as expected 'anthropic/claude-haiku-4-5' is not one of ['o3-mini', ...]… === Step 2: canonical snapshot passes validate_data + Pydantic === ✓ validate_data accepts the canonical value ✓ Pydantic resolves Input.model to LlmModel.CLAUDE_4_5_HAIKU === Step 3: prepare_dry_run shape after canonical-model fix === prepare_dry_run produced model='anthropic/claude-haiku-4-5' Canonical-translated model='claude-haiku-4-5-20251001' ✓ dry-run input (post-fix) passes validate_data === Step 4: real LLM call via OpenRouter Anthropic-compat === ✓ Real LLM call succeeded with canonical snapshot ID ``` The SDK auth fix is supported by the user-reported 401 + the SDK-source trace ([subprocess_cli.py:402](file:claude_agent_sdk/_internal/transport/subprocess_cli.py#L402) merges options.env on top of os.environ; setting "" leaves the key set rather than unset). ### Changes 🏗️ - `prepare_dry_run` translates the simulator model to canonical `LlmModel.value`. - Orchestrator's SDK env now uses the OR key for both `ANTHROPIC_AUTH_TOKEN` and `ANTHROPIC_API_KEY` (was empty string). - Two strengthened/new tests pin the canonical-value invariant and jsonschema validation outcome. ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] `poetry run pytest backend/executor/simulator_test.py::TestPrepareDryRun backend/executor/simulator_test.py::TestDefaultSimulatorModel backend/copilot/tools/test_dry_run.py::test_prepare_dry_run_orchestrator_block backend/blocks/test/test_llm.py::TestLlmModelMissingHandler` — all green - [x] End-to-end validation against real OpenRouter — all 4 steps pass (see "How") - [x] User-reported 401 on dev-builder traced to the orchestrator SDK env-var; root cause + SDK-source line cited - [x] `poetry run black` + `poetry run ruff check` clean on changed files --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why / What / How
Why: Two follow-ups landed after PR #13177 deployed and the user exercised the orchestrator dry-run on dev-builder.
JSON Schema validation gap. fix(backend): teach LlmModel about OpenRouter Anthropic aliases + dry-run sim default Haiku OR-slug #13177 taught
LlmModel._missing_to resolveanthropic/claude-haiku-4-5→CLAUDE_4_5_HAIKU. That fixed Pydantic, butOrchestratorBlock.Inputis validated byvalidate_data→jsonschema.validatefirst, against a schema whoseenumis the literal list ofLlmModel.valuestrings. The alias map is a Python runtime hook — it does not surface in the generated JSON Schema. So the OR-slug was rejected with'anthropic/claude-haiku-4-5' is not one of [...].SDK auth: empty
x-api-key. Once jsonschema started passing, execution reached the EXTENDED_THINKING SDK path in orchestrator.py:1670-1674. That code setssdk_env["ANTHROPIC_API_KEY"] = ""to "force the CLI to use AUTH_TOKEN." But the Claude Agent SDK mergesoptions.envon top ofos.environ(subprocess_cli.py:402), so the spawned CLI seesANTHROPIC_API_KEY=(present-but-empty) and emitsx-api-key:(empty header) on the wire. OpenRouter returns401 invalid x-api-key. This bug was latent until fix(backend): teach LlmModel about OpenRouter Anthropic aliases + dry-run sim default Haiku OR-slug #13177 — every orchestrator dry-run died at jsonschema before reaching the auth wiring.What:
simulator.py: translate the configured simulator model to its canonicalLlmModel.valueviaLlmModel(_simulator_model()).valuebefore injecting intoinput["model"]. The OR-slug default (anthropic/claude-haiku-4-5) becomesclaude-haiku-4-5-20251001— which IS in the JSON Schema enum — sovalidate_datapasses. Downstream OpenRouter's Anthropic-compat endpoint accepts both forms, so no further translation is needed.orchestrator.py: in theprovider == "open_router"SDK branch, setANTHROPIC_API_KEYto the same OpenRouter key (instead of""). OpenRouter accepts eitherx-api-keyorAuthorization: Bearerwith the OR key, so whichever the CLI sends is valid. Explicit set (not omission) is required because the SDK's merge would otherwise let an inherited platformANTHROPIC_API_KEYleak through.simulator_test.py: strengthentest_orchestrator_uses_simulation_modelto assert the injected model is in{m.value for m in LlmModel}; addtest_orchestrator_input_passes_jsonschema_validationthat callsvalidate_dataonprepare_dry_run's output (locks in the exact regression the user hit).How:
End-to-end proof for the model translation, against real OpenRouter (4-step script):
The SDK auth fix is supported by the user-reported 401 + the SDK-source trace (subprocess_cli.py:402 merges options.env on top of os.environ; setting "" leaves the key set rather than unset).
Changes 🏗️
prepare_dry_runtranslates the simulator model to canonicalLlmModel.value.ANTHROPIC_AUTH_TOKENandANTHROPIC_API_KEY(was empty string).Checklist 📋
For code changes:
poetry run pytest backend/executor/simulator_test.py::TestPrepareDryRun backend/executor/simulator_test.py::TestDefaultSimulatorModel backend/copilot/tools/test_dry_run.py::test_prepare_dry_run_orchestrator_block backend/blocks/test/test_llm.py::TestLlmModelMissingHandler— all greenpoetry run black+poetry run ruff checkclean on changed files