Repository navigation
server:package paths and artifact names; VS Code installer and connec… - #2
Merged
Merged
Conversation
…tion UX
Server package (server:package)
------------------------------
- Staging dir: ./artifacts → ./build/artifacts (BUILD_ARTIFACTS_DIR).
Output archives: ./artifacts → ./dist/artifacts (DIST_ARTIFACTS_DIR).
Why: Keep build staging under ./build and distributable outputs under ./dist
so artifact layout matches common conventions and is easier to .gitignore.
- Artifact names: engine-v* → rocketride-v* (e.g. rocketride-v3.2.0-win64.zip).
Why: Align artifact naming with product name and avoid confusion with
generic "engine" in logs and release pages.
- Package Python client from packages/client-python/src/rocketride into
package dir rocketride/ (was clients/python).
Why: Python client was renamed to the rocketride package; package layout
must match so the server can import it at runtime.
- Create ./dist/artifacts before writing archives.
Why: Ensures the output directory exists so createArchive does not fail.
- Do not remove staging dir after package; leave ./build/artifacts/rocketride-v*/
in place for local testing.
Why: Allows running and debugging the packaged server from disk without
re-extracting the zip.
- server:clean: remove BUILD_ARTIFACTS_DIR and DIST_ARTIFACTS_DIR instead of
ARTIFACTS_DIR.
Why: Clean should remove the directories that server:package now uses.
VS Code engine installer
------------------------
- GitHub owner: rocketride-ai → rocketride-org.
Why: Point at the correct GitHub org for releases.
- Fetch release via listReleases({ per_page: 1 }) instead of getLatestRelease.
Why: Avoids relying on deprecated or inconsistent getLatestRelease behavior;
first page of releases is the latest.
- Expected asset name: engine-* → rocketride-* (getPlatformAssetName) to match
new server package artifacts.
Why: Installer looks up release assets by exact name; published artifacts
are now rocketride-v*.zip so the expected name must match.
- User-facing copy: "Engine" → "Server" (logs and progress).
Why: Users see "RocketRide server"; "engine" is an internal term.
Connection manager and UI
--------------------------
- connection.ts: Stop logging full WebSocket URL on connect. Add progressMessage
to connection status; log retry attempts, wait times, and errors; use
"server" in logs instead of "engine".
Why: Avoid logging sensitive or long URLs; give users clear feedback during
retries and failures; consistent "server" wording.
- SidebarConnectionProvider: Add getStateDetail() for progress detail during
connecting states; simplify status labels to "Connected" / "Disconnected" /
"Connecting..."; tooltips say "RocketRide server" instead of "engine".
Why: Single "Connecting" label with a detail line avoids label churn and
keeps tooltips accurate and user-friendly.
- PageConnection: Status label is "Connected" (no mode); connecting states show
"Connecting" + dots with state-based detail line; Connect button label stays
"Connect".
Why: Simpler status line; detail (e.g. "Downloading server...") is in the
subline; button text does not need to change to "Connecting...".
AI web server
-------------
- server.py: Reorder imports (rocketride before ai.constants).
Why: Satisfy linter/formatting (e.g. third-party before local).
4 of 6 tasks
6 of 7 tasks
3 tasks
ryan-t-christensen
added a commit
that referenced
this pull request
Jul 8, 2026
- EventArgs: make `properties` optional for events whose props are ALL optional
(pipeline:stop, auth:login_success, checkout:start, ...), not just NoProps
events. Switches the check to `{} extends EventProperties[E]`. (review #1)
- Add a reverse completeness guard: a type-only assertion that fails the build if
an orphaned EventProperties key drifts from EventName (the forward direction was
already enforced via the indexed access). (review #2)
- Add unit tests for the pure primitives: the CSPRNG UUID-v4 fallback (version/
variant bits + exact format), newAnonId throw path, getOrCreateAnonId, the
isOptedOut truth table, and standardProps raw-vs-posthog-js differences. (review #3)
- Reword "zero-runtime" -> "side-effect-free" in anonId/optOut headers; anonId and
standardProps ship pure runtime functions, so the accurate guarantee is
no-import-time-side-effects / no-SDK-imports. (review #4)
- Document that detectPlatform() never returns 'vscode' (extension host reports
'node', webview 'web'); VS Code callers must pass platform explicitly. (review #5)
- Replace the DOM `Crypto` type in the public anon-id API with a minimal structural
`CryptoLike`, so the published .d.ts stays lib-agnostic. (review #6)
Review #7 (Python SDK parity) is a future-work decision for when client.report()
lands; both current consumers are TS. Noted in the PR description, no code change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
3 of 5 tasks
4 of 7 tasks
dsapandora
added a commit
that referenced
this pull request
Aug 13, 2026
…view findings Replace the per-turn ContextVar with a module-global, lock-guarded usage collector scoped by turn_usage(). A ContextVar could not carry usage back from CrewAI/deepagent kickoffs, which run on a worker thread under contextvars.copy_context() — the agent's answer showed no tokens (dylan #3). The collector spans threads like the metrics singleton, and a seen-mark slice makes it nesting-safe: an agent can drive sub-agents, each its own scope, while the outermost scope reads the whole turn and clears the list so it stays bounded to one turn (asclearuc/dylan #2). Also from the reviews: - reasoning stream (try_openai_compat_reasoning_stream) now asks for include_usage, reads the usage-bearing final chunk, and reports on success — it billed zero before (dylan #1). It falls back to non-streaming if a strict proxy rejects the flag. - do not force stream_usage on a custom base URL (vLLM/proxies 400); usage is still read if the endpoint sends it (dylan #4). - drop the per-call metrics.event that fed an unbounded, cumulatively re-serialized events list (dylan #5); the counters carry the billable totals. - correct the max()-vs-additive comment (asclearuc/dylan #6). Tests: turn total, per-invoke slice, bounding, cross-thread, nested sub-agents, and no bleed between consecutive turns.
dsapandora
added a commit
that referenced
this pull request
Aug 17, 2026
…view findings Replace the per-turn ContextVar with a module-global, lock-guarded usage collector scoped by turn_usage(). A ContextVar could not carry usage back from CrewAI/deepagent kickoffs, which run on a worker thread under contextvars.copy_context() — the agent's answer showed no tokens (dylan #3). The collector spans threads like the metrics singleton, and a seen-mark slice makes it nesting-safe: an agent can drive sub-agents, each its own scope, while the outermost scope reads the whole turn and clears the list so it stays bounded to one turn (asclearuc/dylan #2). Also from the reviews: - reasoning stream (try_openai_compat_reasoning_stream) now asks for include_usage, reads the usage-bearing final chunk, and reports on success — it billed zero before (dylan #1). It falls back to non-streaming if a strict proxy rejects the flag. - do not force stream_usage on a custom base URL (vLLM/proxies 400); usage is still read if the endpoint sends it (dylan #4). - drop the per-call metrics.event that fed an unbounded, cumulatively re-serialized events list (dylan #5); the counters carry the billable totals. - correct the max()-vs-additive comment (asclearuc/dylan #6). Tests: turn total, per-invoke slice, bounding, cross-thread, nested sub-agents, and no bleed between consecutive turns.
5 of 7 tasks
dsapandora
added a commit
that referenced
this pull request
Aug 18, 2026
* feat(llm): capture exact provider token usage in the Adapter for per-user billing Read provider-reported usage (input/output tokens) at the normalized Adapter (#1683) and report it via metrics.counter so per-user token accounting flows through task_metrics/billing. One capture point covers all LangChain providers; the native Anthropic extended-thinking and OpenAI Responses paths are instrumented too. * feat(llm): cover OpenAI streaming usage and split cache tokens; add tests Addresses review on #1780: - Enable stream_usage for OpenAI-family models so streaming reports usage (LangChain does not emit usage_metadata for ChatOpenAI without it). Verified live: gpt-5.4-mini now reports input=21/output=11 where it reported nothing. - Report cache-read and cache-creation tokens on their own counters (disjoint from fresh input) since providers bill them at different rates; captured from the Anthropic native, OpenAI Responses, and LangChain paths. - Log suppressed reporting failures via debug instead of a silent pass. - Add unit tests for report_llm_tokens and the usage/cache split. * fix(llm): report token usage from the finally so partial-stream failures still bill Move report_llm_tokens into the finally of both native adapters (NativeAnthropicAdapter, NativeOpenAIResponsesAdapter) so a mid-stream error after message_start/message_delta still records the provider-reported usage. Adds a regression test that raises mid-stream and asserts the partial usage (input + cache) was recorded. * fix(llm): capture usage on failed OpenAI Responses terminal events response.failed can carry a usage object; read usage on failed/error terminal events (not only response.completed) so a failed-but-chargeable request reports tokens instead of zero. finish_reason stays 'error'. Adds a regression test. * fix(llm): treat response.incomplete as a terminal event in the OpenAI Responses adapter OpenAI Responses can emit response.incomplete separately from response.completed (e.g. max_output_tokens). Handle it as terminal: read its usage and map incomplete_details.reason to finish_reason, so a truncated-but-chargeable request still records tokens instead of zero. Adds a regression test. * fix(llm): capture LangChain token usage for every provider, not just OpenAI The stream_usage gate matched 'OpenAI' in the class name, so ChatXAI never asked for usage and silently reported none, even though it inherits the flag from BaseChatOpenAI; ChatBedrock does not declare the field at all. Gate on the declared flag instead. Report from a finally as well, so a stream that breaks mid-way still records the usage its chunks already carried, matching the two native adapters. * feat(llm): surface per-call token usage in the pipeline Trace report_llm_tokens publishes each call's usage on a ContextVar; the LLM node hangs it on Answer.tokens, which the Trace 'Tokens' grid already renders. One hook at the shared Adapter seam covers every provider. * fix(llm): accumulate per-turn token usage with a per-call breakdown for the Trace The counters and the llm_tokens event stay per-call (billing path unchanged), but the context var feeding answer.tokens now sums every model call in the turn and carries calls plus a breakdown listing each call's token cost. A ReAct/CrewAI turn that hits the model N times shows the whole turn's total and what each call cost in the Trace and action history, instead of only the last call. * fix(trace): render per-call token breakdown instead of crashing on the object The Tokens grid mapped answer.tokens as flat scalars; the new per-call breakdown (an array of objects) made React throw 'Objects are not valid as a React child'. Split the scalar totals (rendered as chips) from breakdown (one row per model call) and type answer.tokens as TokenUsage. * fix(agent): carry per-turn token usage onto the agent's answer The Trace "Tokens" grid renders answer.tokens, but that field was only ever set by the LLM node's own questions-lane path. An agent emits its own Answer from AgentBase.run_agent, so every agentic run reached the Trace with no usage at all — the case report_llm_tokens' calls/breakdown accumulation was written for. Read the same context var here and attach it when present. Covers all five agent frameworks, since they share this base. * feat(trace): show each model call's tokens on its own invoke row An 11-call agent run listed 11 breakdown blocks under the answer, none of them tied to the call that produced them. The data was only ever aggregated onto the final answer, which also arrives after every invoke has already rendered. ask() now attaches its own slice of the turn — the breakdown entries appended while it ran — so each Anthropic INVOKE row shows what that call cost, through the renderer the row already uses. The answer keeps the turn total and drops the per-call list. * fix(llm): thread- and nesting-safe turn usage collector; cover the review findings Replace the per-turn ContextVar with a module-global, lock-guarded usage collector scoped by turn_usage(). A ContextVar could not carry usage back from CrewAI/deepagent kickoffs, which run on a worker thread under contextvars.copy_context() — the agent's answer showed no tokens (dylan #3). The collector spans threads like the metrics singleton, and a seen-mark slice makes it nesting-safe: an agent can drive sub-agents, each its own scope, while the outermost scope reads the whole turn and clears the list so it stays bounded to one turn (asclearuc/dylan #2). Also from the reviews: - reasoning stream (try_openai_compat_reasoning_stream) now asks for include_usage, reads the usage-bearing final chunk, and reports on success — it billed zero before (dylan #1). It falls back to non-streaming if a strict proxy rejects the flag. - do not force stream_usage on a custom base URL (vLLM/proxies 400); usage is still read if the endpoint sends it (dylan #4). - drop the per-call metrics.event that fed an unbounded, cumulatively re-serialized events list (dylan #5); the counters carry the billable totals. - correct the max()-vs-additive comment (asclearuc/dylan #6). Tests: turn total, per-invoke slice, bounding, cross-thread, nested sub-agents, and no bleed between consecutive turns. * test(trace): cover the answer Tokens grid render Per the review: pin that the per-call breakdown is dropped from the chip grid, a tokens object holding only a breakdown renders no Tokens section, and the non-numeric model chip is stringified. Follows the media_renderers.test.tsx harness (node --import tsx --test via shared:test). * test(llm): assert no per-call event is emitted (it was dropped for O(N^2)) test_emits_llm_tokens_event_payload asserted the per-call llm_tokens metrics.event that this branch removed. Flip it: the event must NOT be emitted; the counters carry the totals. * fix(llm): per-turn usage collector safe under concurrent turns Move the turn collector from a module global to a ContextVar holding a mutable list appended in place. Overlapping turns (tool waves, one pipeline per message) each get their own list, so a Trace never shows another turn's tokens and the list dies with its scope (no global clear, no unbounded growth under load). copy_context() shares the list object, so worker-thread kickoffs still land in the open turn; run the rocketride tool wave under copy_context().run so its calls propagate. Meter the whole OpenAI-compatible family: ask for streaming usage on both the LangChain and native reasoning paths and retry once without the flag if the endpoint rejects it before the first chunk, instead of hard-disabling it for every custom base URL (xAI, DeepSeek, Qwen, Together, Groq, vLLM). An explicit stream_usage in stream_kwargs overrides the default. Cover the native reasoning capture point and the concurrent-turn, worker- thread and retry paths with tests. * fix(llm): isolate sibling invoke scopes; gate + log the LangChain usage retry Each turn_usage scope now owns its own list and folds it into the parent on exit, so two invokes running concurrently in one turn (a parallel tool wave) no longer cross-read each other's per-call breakdown in the Trace; the turn total is unchanged. On the LangChain usage-flag retry, warn before retrying (matching the native path) and gate it on status_code == 400 so a 401/429 surfaces without a second round trip, while a provider whose error carries no status_code keeps the broad retry. * fix(llm): gate the usage-flag retry to real 400s; stop reasoning-only double-bill Retry without stream_options only on a 400 (the flag rejection), on both the LangChain and native reasoning paths. A 401/429 or a transient connection error (no status_code) now re-raises / falls back instead of being retried without the flag: no silent unmetered success, and a rate limit stays at two round trips instead of four. On the reasoning path, report captured usage before the empty-content check and return the empty answer (the caller keeps it, since it tests `is not None`), so a reasoning-only turn is not discarded and re-issued via the non-streaming fallback — which billed the provider twice, metered once. Only a stream that produced nothing at all still falls back. * test(llm): cover the LLMBase token seams; widen the usage-flag retry to 4xx writeQuestions and ask both hang the turn's usage on the Answer and neither had a test — the plain llm_* node lane. Adds three, including a second ask on the same instance not inheriting the first's tokens. The usage-flag retry gated on status 400 only, so a FastAPI-based endpoint (vLLM's OpenAI-compatible server) answering 422 lost token-by-token streaming on every request. Both call sites now share is_usage_flag_rejection: any 4xx except 401/403/408/429. Drops the unused `seen` offset from usage_since (now aggregate_usage) — scopes isolate by owning their list, and the docstring still taught the replaced design. Barriers in the concurrency tests get a timeout so a dead thread fails instead of hanging. * fix(llm): keep the fallback for a stream that produced nothing include_usage is now always requested, so usage is set on essentially every successful stream — and gating the fallback on `usage is None` made it unreachable. A stream that emits neither content nor reasoning (a proxy that strips delta.reasoning_content, hitting max_tokens while reasoning) returned the empty string, which chat.py serves as the answer instead of re-issuing the request non-streaming. The usage report already runs before the guard, so ordering alone keeps the stream metered once; the fallback's own metering covers the second request, which the provider bills anyway. Also hang the turn's usage on the error answer: the adapters report from their finally, so the tokens are known — without this the Trace grid is empty for exactly the turn a user opens. * feat(llm): meter the five drivers that bypass the Adapter capture point LangChainAdapter meters every provider that routes through ChatBase, but five drivers override that seam and reach their SDK directly, so the Adapter never sees them and they billed zero. Two seams are overridden: _chat (gemini, ibm_watson, minimax) and the chat seam above it (perplexity, mistral). Gemini is the widest gap: it sets _client and never _llm, so the streaming gate in chat.py is never true and every call — streaming or not — took the override. Each now reports from its own call site, off its SDK's own usage object, keeping the four counters disjoint: Gemini counts prompt_token_count with the cached prefix included (subtracted back out) and bills thoughts_token_count as output; watsonx and Mistral carry no cache detail. _report_usage_metadata is now public so the two LangChain-backed drivers reuse the shared split instead of copying it. Also covers the error-path answer.tokens seam on LLMBase, which landed in 17ce15e without a test. The four llm_vision_* nodes and accessibility_describe override the same chat seam and stay unmetered: they are image-priced, so they need their own accounting decision rather than a copy of this one. * fix(llm): never let a usage read cost the user the answer they paid for The usage split runs before report_llm_tokens' own guard, so a provider or proxy answering a non-numeric count raised out of the metering code and into the caller — which is a retry loop. Verified against the real driver: a successful, already-billed Perplexity response came back to the user as "Perplexity API error: invalid literal for int() with base 10: 'n/a'". report_usage_metadata was already exposed to this before the five drivers were wired to it (LangChainAdapter.collect() calls it on every non-streaming turn); the new call sites widened the same hole to five more providers. Guards the split in report_usage_metadata and in the three SDK readers, sharing one debug_usage_failure() so every capture point fails the same way — the trace loses that call's tokens, the turn keeps its answer. One regression test per driver. * refactor(llm): drop the redundant usage guard in the two pass-through readers report_llm_tokens coerces its own arguments inside its own try and reports the failure through the same debug_usage_failure, so the local try in the watson and mistral readers could not change the outcome. Pass the raw values through: a missing count still reads as zero, a non-numeric one is swallowed one frame deeper, and the per-driver regression test holds. llm_gemini keeps its guard — it subtracts and adds the counts before calling.
erichanwang
pushed a commit
to erichanwang/rocketride-server
that referenced
this pull request
Sep 7, 2026
…ride-org#1780) * feat(llm): capture exact provider token usage in the Adapter for per-user billing Read provider-reported usage (input/output tokens) at the normalized Adapter (rocketride-org#1683) and report it via metrics.counter so per-user token accounting flows through task_metrics/billing. One capture point covers all LangChain providers; the native Anthropic extended-thinking and OpenAI Responses paths are instrumented too. * feat(llm): cover OpenAI streaming usage and split cache tokens; add tests Addresses review on rocketride-org#1780: - Enable stream_usage for OpenAI-family models so streaming reports usage (LangChain does not emit usage_metadata for ChatOpenAI without it). Verified live: gpt-5.4-mini now reports input=21/output=11 where it reported nothing. - Report cache-read and cache-creation tokens on their own counters (disjoint from fresh input) since providers bill them at different rates; captured from the Anthropic native, OpenAI Responses, and LangChain paths. - Log suppressed reporting failures via debug instead of a silent pass. - Add unit tests for report_llm_tokens and the usage/cache split. * fix(llm): report token usage from the finally so partial-stream failures still bill Move report_llm_tokens into the finally of both native adapters (NativeAnthropicAdapter, NativeOpenAIResponsesAdapter) so a mid-stream error after message_start/message_delta still records the provider-reported usage. Adds a regression test that raises mid-stream and asserts the partial usage (input + cache) was recorded. * fix(llm): capture usage on failed OpenAI Responses terminal events response.failed can carry a usage object; read usage on failed/error terminal events (not only response.completed) so a failed-but-chargeable request reports tokens instead of zero. finish_reason stays 'error'. Adds a regression test. * fix(llm): treat response.incomplete as a terminal event in the OpenAI Responses adapter OpenAI Responses can emit response.incomplete separately from response.completed (e.g. max_output_tokens). Handle it as terminal: read its usage and map incomplete_details.reason to finish_reason, so a truncated-but-chargeable request still records tokens instead of zero. Adds a regression test. * fix(llm): capture LangChain token usage for every provider, not just OpenAI The stream_usage gate matched 'OpenAI' in the class name, so ChatXAI never asked for usage and silently reported none, even though it inherits the flag from BaseChatOpenAI; ChatBedrock does not declare the field at all. Gate on the declared flag instead. Report from a finally as well, so a stream that breaks mid-way still records the usage its chunks already carried, matching the two native adapters. * feat(llm): surface per-call token usage in the pipeline Trace report_llm_tokens publishes each call's usage on a ContextVar; the LLM node hangs it on Answer.tokens, which the Trace 'Tokens' grid already renders. One hook at the shared Adapter seam covers every provider. * fix(llm): accumulate per-turn token usage with a per-call breakdown for the Trace The counters and the llm_tokens event stay per-call (billing path unchanged), but the context var feeding answer.tokens now sums every model call in the turn and carries calls plus a breakdown listing each call's token cost. A ReAct/CrewAI turn that hits the model N times shows the whole turn's total and what each call cost in the Trace and action history, instead of only the last call. * fix(trace): render per-call token breakdown instead of crashing on the object The Tokens grid mapped answer.tokens as flat scalars; the new per-call breakdown (an array of objects) made React throw 'Objects are not valid as a React child'. Split the scalar totals (rendered as chips) from breakdown (one row per model call) and type answer.tokens as TokenUsage. * fix(agent): carry per-turn token usage onto the agent's answer The Trace "Tokens" grid renders answer.tokens, but that field was only ever set by the LLM node's own questions-lane path. An agent emits its own Answer from AgentBase.run_agent, so every agentic run reached the Trace with no usage at all — the case report_llm_tokens' calls/breakdown accumulation was written for. Read the same context var here and attach it when present. Covers all five agent frameworks, since they share this base. * feat(trace): show each model call's tokens on its own invoke row An 11-call agent run listed 11 breakdown blocks under the answer, none of them tied to the call that produced them. The data was only ever aggregated onto the final answer, which also arrives after every invoke has already rendered. ask() now attaches its own slice of the turn — the breakdown entries appended while it ran — so each Anthropic INVOKE row shows what that call cost, through the renderer the row already uses. The answer keeps the turn total and drops the per-call list. * fix(llm): thread- and nesting-safe turn usage collector; cover the review findings Replace the per-turn ContextVar with a module-global, lock-guarded usage collector scoped by turn_usage(). A ContextVar could not carry usage back from CrewAI/deepagent kickoffs, which run on a worker thread under contextvars.copy_context() — the agent's answer showed no tokens (dylan rocketride-org#3). The collector spans threads like the metrics singleton, and a seen-mark slice makes it nesting-safe: an agent can drive sub-agents, each its own scope, while the outermost scope reads the whole turn and clears the list so it stays bounded to one turn (asclearuc/dylan rocketride-org#2). Also from the reviews: - reasoning stream (try_openai_compat_reasoning_stream) now asks for include_usage, reads the usage-bearing final chunk, and reports on success — it billed zero before (dylan #1). It falls back to non-streaming if a strict proxy rejects the flag. - do not force stream_usage on a custom base URL (vLLM/proxies 400); usage is still read if the endpoint sends it (dylan rocketride-org#4). - drop the per-call metrics.event that fed an unbounded, cumulatively re-serialized events list (dylan rocketride-org#5); the counters carry the billable totals. - correct the max()-vs-additive comment (asclearuc/dylan rocketride-org#6). Tests: turn total, per-invoke slice, bounding, cross-thread, nested sub-agents, and no bleed between consecutive turns. * test(trace): cover the answer Tokens grid render Per the review: pin that the per-call breakdown is dropped from the chip grid, a tokens object holding only a breakdown renders no Tokens section, and the non-numeric model chip is stringified. Follows the media_renderers.test.tsx harness (node --import tsx --test via shared:test). * test(llm): assert no per-call event is emitted (it was dropped for O(N^2)) test_emits_llm_tokens_event_payload asserted the per-call llm_tokens metrics.event that this branch removed. Flip it: the event must NOT be emitted; the counters carry the totals. * fix(llm): per-turn usage collector safe under concurrent turns Move the turn collector from a module global to a ContextVar holding a mutable list appended in place. Overlapping turns (tool waves, one pipeline per message) each get their own list, so a Trace never shows another turn's tokens and the list dies with its scope (no global clear, no unbounded growth under load). copy_context() shares the list object, so worker-thread kickoffs still land in the open turn; run the rocketride tool wave under copy_context().run so its calls propagate. Meter the whole OpenAI-compatible family: ask for streaming usage on both the LangChain and native reasoning paths and retry once without the flag if the endpoint rejects it before the first chunk, instead of hard-disabling it for every custom base URL (xAI, DeepSeek, Qwen, Together, Groq, vLLM). An explicit stream_usage in stream_kwargs overrides the default. Cover the native reasoning capture point and the concurrent-turn, worker- thread and retry paths with tests. * fix(llm): isolate sibling invoke scopes; gate + log the LangChain usage retry Each turn_usage scope now owns its own list and folds it into the parent on exit, so two invokes running concurrently in one turn (a parallel tool wave) no longer cross-read each other's per-call breakdown in the Trace; the turn total is unchanged. On the LangChain usage-flag retry, warn before retrying (matching the native path) and gate it on status_code == 400 so a 401/429 surfaces without a second round trip, while a provider whose error carries no status_code keeps the broad retry. * fix(llm): gate the usage-flag retry to real 400s; stop reasoning-only double-bill Retry without stream_options only on a 400 (the flag rejection), on both the LangChain and native reasoning paths. A 401/429 or a transient connection error (no status_code) now re-raises / falls back instead of being retried without the flag: no silent unmetered success, and a rate limit stays at two round trips instead of four. On the reasoning path, report captured usage before the empty-content check and return the empty answer (the caller keeps it, since it tests `is not None`), so a reasoning-only turn is not discarded and re-issued via the non-streaming fallback — which billed the provider twice, metered once. Only a stream that produced nothing at all still falls back. * test(llm): cover the LLMBase token seams; widen the usage-flag retry to 4xx writeQuestions and ask both hang the turn's usage on the Answer and neither had a test — the plain llm_* node lane. Adds three, including a second ask on the same instance not inheriting the first's tokens. The usage-flag retry gated on status 400 only, so a FastAPI-based endpoint (vLLM's OpenAI-compatible server) answering 422 lost token-by-token streaming on every request. Both call sites now share is_usage_flag_rejection: any 4xx except 401/403/408/429. Drops the unused `seen` offset from usage_since (now aggregate_usage) — scopes isolate by owning their list, and the docstring still taught the replaced design. Barriers in the concurrency tests get a timeout so a dead thread fails instead of hanging. * fix(llm): keep the fallback for a stream that produced nothing include_usage is now always requested, so usage is set on essentially every successful stream — and gating the fallback on `usage is None` made it unreachable. A stream that emits neither content nor reasoning (a proxy that strips delta.reasoning_content, hitting max_tokens while reasoning) returned the empty string, which chat.py serves as the answer instead of re-issuing the request non-streaming. The usage report already runs before the guard, so ordering alone keeps the stream metered once; the fallback's own metering covers the second request, which the provider bills anyway. Also hang the turn's usage on the error answer: the adapters report from their finally, so the tokens are known — without this the Trace grid is empty for exactly the turn a user opens. * feat(llm): meter the five drivers that bypass the Adapter capture point LangChainAdapter meters every provider that routes through ChatBase, but five drivers override that seam and reach their SDK directly, so the Adapter never sees them and they billed zero. Two seams are overridden: _chat (gemini, ibm_watson, minimax) and the chat seam above it (perplexity, mistral). Gemini is the widest gap: it sets _client and never _llm, so the streaming gate in chat.py is never true and every call — streaming or not — took the override. Each now reports from its own call site, off its SDK's own usage object, keeping the four counters disjoint: Gemini counts prompt_token_count with the cached prefix included (subtracted back out) and bills thoughts_token_count as output; watsonx and Mistral carry no cache detail. _report_usage_metadata is now public so the two LangChain-backed drivers reuse the shared split instead of copying it. Also covers the error-path answer.tokens seam on LLMBase, which landed in 17ce15e without a test. The four llm_vision_* nodes and accessibility_describe override the same chat seam and stay unmetered: they are image-priced, so they need their own accounting decision rather than a copy of this one. * fix(llm): never let a usage read cost the user the answer they paid for The usage split runs before report_llm_tokens' own guard, so a provider or proxy answering a non-numeric count raised out of the metering code and into the caller — which is a retry loop. Verified against the real driver: a successful, already-billed Perplexity response came back to the user as "Perplexity API error: invalid literal for int() with base 10: 'n/a'". report_usage_metadata was already exposed to this before the five drivers were wired to it (LangChainAdapter.collect() calls it on every non-streaming turn); the new call sites widened the same hole to five more providers. Guards the split in report_usage_metadata and in the three SDK readers, sharing one debug_usage_failure() so every capture point fails the same way — the trace loses that call's tokens, the turn keeps its answer. One regression test per driver. * refactor(llm): drop the redundant usage guard in the two pass-through readers report_llm_tokens coerces its own arguments inside its own try and reports the failure through the same debug_usage_failure, so the local try in the watson and mistral readers could not change the outcome. Pass the raw values through: a missing count still reads as zero, a non-numeric one is swallowed one frame deeper, and the per-driver regression test holds. llm_gemini keeps its guard — it subtracts and adds the counts before calling.
4 of 7 tasks
dylan-savage
added a commit
that referenced
this pull request
Sep 11, 2026
- D13: add tests/test_validate_node_readme.py as a third, blocking phase
of makeValidateAction in docs/docusaurus/scripts/tasks.js, reusing the
shared runPytest() helper (scripts/lib/pytest.js, exported from
scripts/lib) with engine: 'python3' — a compiled engine binary isn't
available in the docs pipeline, and runPytest's own docstring allows
"python" as the engine arg. nodes/scripts/tasks.js's nodes:test still
runs this same file too (left in place, per plan).
- Checked .github/workflows/docs-schemas.yml: it sets up Node only, no
Python tooling, and installs no pytest anywhere (confirmed: neither
validate-client-docs.py nor validate-node-readme.py needs it — only
the moved unit-test file does). Added the one missing line ("Install
validator test dependencies": `python3 -m pip install
--disable-pip-version-check pytest==8.4.1`, same pin used before this
file's schemas job was removed) so docs:test can run it on ubuntu-
latest's already-present python3.
- Updated docs/development/ci-gates.md:178-182 to say the validator's
own unit tests run inside docs:validate on every PR now, dropping the
os-matrix wording; nodes:test's own invocation of the same file is
noted as unchanged.
- Pipeline Minor #2: drop the now-redundant `docs/docusaurus/**` path
filter (a subset of `docs/**` since the Docusaurus package moved under
docs/).
- Pipeline Minors #3 and #4: reword absolutizeImageLinks' doc comment in
scripts/lib/markdown.js — list titled markdown images and single-quoted
<img src='...'> as additional untouched forms, and stop implying `../`
segments would resolve incorrectly (client-side URL normalization
handles them; no code change, no current file uses either form).
Gates: ./builder docs:test, ./builder docs:build, ./builder docs:check
all end "Builder complete!".
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017bVEisz8uY1PzUgZ1iLPjy
dylan-savage
added a commit
that referenced
this pull request
Sep 11, 2026
…redundant path filter - docs-schemas.yml's path filters (both pull_request and push) dropped tests/test_validate_node_readme.py when the schemas job that used to run it was removed, and no other workflow filters on tests/**, so a PR touching only that test file triggered no CI job even though docs:validate now runs it (this commit's earlier D13 change). Added it back to both lists. - Pipeline Minor #2, second instance: docs.yml also carried the now- redundant docs/docusaurus/** path filter (a subset of docs/** since the Docusaurus package moved under docs/); dropped it, matching the same fix already made to docs-schemas.yml in the previous commit. Gates: ./builder docs:test, ./builder docs:build, ./builder docs:check all end "Builder complete!"; both workflow files parse as valid YAML. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017bVEisz8uY1PzUgZ1iLPjy
6 of 7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…tion UX
Server package (server:package)
VS Code engine installer
Connection manager and UI
AI web server