Conversation
- Parse x-agenta-flags.is_chat from OpenAPI operations when available - Fall back to legacy heuristic based on messages fields - Thread is_chat into evaluation payload building and add temporary logs
The SDK (PR #3622) changed the OpenAPI vendor extension from a flat 'x-agenta-flags' key to a nested 'x-agenta: {flags: {...}}' structure. Update _get_openapi_chat_flag to read from the new nested path. Also removes unused imports (common, make_hash_id) caught by ruff.
…llback, restore removed variables
- Fix TODO comment: says 'messages' column, not 'chats'
- Remove datapoint.get('chat') fallback — 'chat' was the old column name,
the FE now uses 'messages'. No need for backward compat.
- Restore references/links variables + imports that were removed by ruff
as unused — they belong to a commented-out make_hash_id call and are
out of scope for this PR.
…SaveTestsetModal SaveTestsetModal.tsx hardcodes 'chat' as the column name when re-saving evaluation results to a testset. Several FE readers (DebugSection, CHAT_ARRAY_KEYS, legacy evaluation exports) also reference 'chat'. Keep the fallback: prefer 'messages', fall back to 'chat'.
Update all 4 frontend chat detection paths to prefer the explicit x-agenta.flags.is_chat flag from the SDK's OpenAPI extension, falling back to the existing messages-property heuristic for older apps. Detection paths updated: - genericTransformer/index.ts: detectChatVariantFromOpenAISchema() - appRevision/api/schema.ts: extractAllEndpointSchemas() - appRevision + legacyAppRevision runnableSetup.ts: isChatVariantAtomFamily (now reads pre-computed isChatVariant from schema state) - requestSchemaMeta.ts: hasMessages computation Stacked on PR #3622 (SDK flags support).
…allback Fix pre-existing bug surfaced by this PR: ep?.messagesSchema !== null evaluates to true for null endpoints (undefined !== null is true), causing every app without the x-agenta flag to be detected as chat. Change to !!ep?.messagesSchema for correct truthiness check.
…or traces and sessions
…line diff highlighting
The primary config-fetching endpoint used by the SDK middleware, SDK managers, frontend, and all documented curl examples was falling into the STANDARD rate limit tier (120 req/min on Hobby) instead of CORE_FAST (1,200 req/min).
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
dosubot
bot
added
the
size:S
fix(api): Add /variants/configs/fetch to CORE_FAST rate limit category
…ction feat(api): prefer x-agenta-flags for evaluation chat detection
dosubot
bot
added
size:L
…p-event fix(api): add cloud region metadata to signup event
…ding-video-reappears-when-no-traces-in-last-24h fix(frontend)[AGE-3638]: Observability onboarding video reappears when no traces in last 24h
…eplaces-entire-prompt-field
…al-replaces-entire-prompt-field fix(frontend)[AGE-3607]: Diff in commit modal replaces entire prompt field
dosubot
bot
added
size:XXL
…ts-still-returned-in-registryplayground-lists fix(frontend)[AGE-3630]: Deleted variants still returned in registry/playground lists
…points-to-empty-area-on-resize fix(frontend)[AGE 3613]: Onboarding widget points to empty area on resize
…k handlers Use ignore_inputs=["parameters"] on the @Instrument() decorator for completion_v0, chat_v0, and hook_v0 so that the configuration parameters are no longer captured inside ag.data.inputs. Also fix the ignore_inputs/ignore_outputs type hints on the instrument class to accept Union[bool, List[str]] matching the actual _redact implementation. Remove the frontend workaround in getAgDataInputs that was stripping parameters from inputs before display. Closes #3755
Metrics refresh only collected trace IDs from annotation steps, so cost and token metrics from invocation traces were never computed. This regression was introduced in 9500279 (2026-01-30, v0.81.2).
#3755 fix(SDK, frontend): Exclude parameters from trace inputs in chat/completion/hook handlers
fix(api): include invocation steps in evaluation metrics refresh
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Comment on lines
+593
to
+596
| operation, operation_path = _get_openapi_operation(schema, route_path) | ||
| is_chat = _get_openapi_chat_flag(operation, operation_path) | ||
| if is_chat is None: | ||
| is_chat = _fallback_chat_detection(properties, operation_path) |
Contributor
There was a problem hiding this comment.
🔴 Fallback chat detection returns False when schema resolution fails, silently dropping messages from payload
When get_parameters_from_openapi cannot resolve the request body schema (e.g., complex $ref chains, missing schema name), properties becomes {}. The new _fallback_chat_detection then returns False, causing is_chat to be False. In make_payload, the if is_chat: guard on line 190 prevents messages from being added to the payload. The old code unconditionally added messages at lines 185–188, so this is a behavioral regression for apps where schema resolution fails but the app still expects messages.
Root Cause and Impact
The old code in make_payload always ran:
messages_data = datapoint.get("messages", "[]")
messages = json.loads(messages_data)
payload["messages"] = messagesThe new code conditionally runs this only when is_chat is truthy:
if is_chat:
messages_data = datapoint.get("messages") or datapoint.get("chat", "[]")
...
payload["messages"] = messagesThe is_chat value comes from get_parameters_from_openapi (api/oss/src/services/llm_apps_service.py:593-596). If _get_openapi_chat_flag returns None (no x-agenta extension), _fallback_chat_detection is called with properties. But properties is derived only from the /test endpoint body schema (api/oss/src/services/llm_apps_service.py:577-591). If schema name resolution fails at line 586 (body_schema_name = ""), properties is {}, and _fallback_chat_detection returns False.
Impact: Chat applications whose OpenAPI body schema cannot be resolved will silently lose their message payloads during batch evaluation invocations, leading to incorrect or failed LLM calls.
Prompt for agents
In api/oss/src/services/llm_apps_service.py, the _fallback_chat_detection function (line 662) always returns a concrete bool. When properties is empty due to schema resolution failure, it returns False, which causes make_payload to skip adding messages to the payload. To fix this, _fallback_chat_detection should return Optional[bool] and return None when properties is empty (meaning we cannot determine chat status). Then in get_parameters_from_openapi (lines 593-596), if both _get_openapi_chat_flag and _fallback_chat_detection return None, is_chat should remain None. In make_payload (line 190), change the condition from 'if is_chat:' to 'if is_chat is not False:' (or 'if is_chat is None or is_chat:') so that the uncertain case still attempts to send messages (preserving old behavior when detection is inconclusive).
Was this helpful? React with 👍 or 👎 to provide feedback.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New version v0.85.6 in