Repository navigation
🧾 fix: Bill Subagent Child-Run Model Usage in Parent Transactions - #13683
Conversation
There was a problem hiding this comment.
Pull request overview
This PR fixes a billing gap where token usage from subagent child runs was not being recorded/spent, by introducing a subagentUsageSink that funnels child-run usage into the existing collectedUsage → transaction pipeline and accounting for it separately as usage_type: 'subagent'.
Changes:
- Added
createSubagentUsageSink(collectedUsage)and updatedrecordCollectedUsageto bill subagent usage under a dedicated'subagent'transaction context while excluding subagent completions from reportedoutput_tokens. - Threaded
subagentUsageSinkthroughcreateRunand wired it into the main agent execution surfaces (chat, OpenAI-compatible completions, and Responses API). - Added/updated unit tests and mocks to cover the new sink behavior and subagent usage segregation.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/api/src/stream/interfaces/IJobStore.ts | Extends UsageMetadata.usage_type union to include 'subagent'. |
| packages/api/src/agents/usage.ts | Adds createSubagentUsageSink and bills usage_type: 'subagent' separately, excluding it from reported output totals. |
| packages/api/src/agents/usage.spec.ts | Adds unit coverage for subagent segregation and sink behavior. |
| packages/api/src/agents/run.ts | Extends createRun signature to accept/pass subagentUsageSink into Run.create. |
| api/server/controllers/agents/responses.js | Passes subagentUsageSink when creating runs for Responses API. |
| api/server/controllers/agents/openai.js | Passes subagentUsageSink when creating runs for OpenAI-compatible chat completions. |
| api/server/controllers/agents/client.js | Passes subagentUsageSink for the main chat agent client runs. |
| api/server/controllers/agents/tests/responses.unit.spec.js | Updates @librechat/api mock to include createSubagentUsageSink. |
| api/server/controllers/agents/tests/openai.spec.js | Updates @librechat/api mock to include createSubagentUsageSink. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| appConfig, | ||
| subagentUsageSink, | ||
| streaming = true, | ||
| streamUsage = true, | ||
| }: { |
| import { logger } from '@librechat/data-schemas'; | ||
| import { Providers } from 'librechat-data-provider'; | ||
| import type { SubagentUsageEvent } from '@librechat/agents'; | ||
| import type { TCustomConfig, TTransactionsConfig } from 'librechat-data-provider'; |
|
Reviewed against the context usage tracker (#13670) — the sink design and the One integration seam once both PRs land: the sink pushes child usage straight into |
GitNexus: ❌ deploy failedThe deploy failed — the previous index (if any) continues to be served. |
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e0995b6438
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| initialSessions, | ||
| calibrationRatio, | ||
| indexTokenCountMap, | ||
| subagentUsageSink, |
There was a problem hiding this comment.
This wires subagentUsageSink into Run.create, but the repository is still locked to @librechat/agents 3.2.33 (package-lock.json resolves node_modules/@librechat/agents to 3.2.33), and the new comment here says that versions <= 3.2.33 ignore this field at runtime. In installs that use the committed lockfile, subagent child runs will never call the sink, so their usage still will not be added to collectedUsage or billed; bump the dependency/lockfile to the SDK version that actually supports the option before relying on this path.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Correct, and intentional — called out in the PR description: the sink is inert on the locked 3.2.33 and activates with the dependency/lockfile bump once LibreChat-AI/agents#236 ships in a release. The structural typing exists precisely so this PR can merge green ahead of that release without a hard ordering constraint; the bump (plus swapping the structural mirrors for the SDK's Pick<RunConfig, 'subagentUsageSink'> / type import, commented at each site) is the follow-up that turns billing on.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e0995b6438
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| initialSessions, | ||
| calibrationRatio, | ||
| indexTokenCountMap, | ||
| subagentUsageSink, |
There was a problem hiding this comment.
In lockfile-based installs (Dockerfile and CI run npm ci), package-lock.json still resolves @librechat/agents to 3.2.33, and the new comment directly above this block says that SDK version ignores subagentUsageSink at runtime. In that environment Run.create never calls the sink, so subagent child-run usage still never reaches collectedUsage and remains unbilled; update the dependency/lockfile to a version that implements the callback before wiring billing through it.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Resolved by b6e85b8 (pushed after this review): both api/package.json and packages/api/package.json now pin ^3.2.35, and package-lock.json resolves @librechat/agents to 3.2.35 — the release that implements subagentUsageSink. So npm ci in Docker/CI installs the version that actually invokes the callback, and subagent child-run usage reaches collectedUsage and gets billed. Good catch flagging the lockfile specifically.
| user: { id: userId }, | ||
| /** Bills subagent child-run model calls (reported outside the | ||
| * streamEvents loop) into the same collectedUsage array. */ | ||
| subagentUsageSink: createSubagentUsageSink(collectedUsage), |
There was a problem hiding this comment.
For Agents API requests that invoke a subagent, this sink feeds only the billing array; the user-visible Chat Completions usage is still built later from tracker/aggregator.usage, which is updated only by the on_chat_model_end stream handler above. That means recordCollectedUsage charges the child-run tokens, but the API response's usage totals omit them, so API clients see lower token counts than the balance actually spent. Please also fold sink events into the API usage tracker or derive the response usage from the billed usage.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Valid observation, but deferring this one consistent with the OpenAI-compatible surface's move into the @librechat/agents SDK — api/server/controllers/agents/openai.js and responses.js are part of the handler surface being migrated, and handler fixes here were already descoped from #13636 for the same reason. Scope of the discrepancy: billing is correct (recordCollectedUsage charges the child-run tokens from the sink), and the main chat path is unaffected (its response tokenCount derives from recordCollectedUsage's result, which intentionally excludes subagent output from the parent's context count while still billing it). The only gap is the Chat-Completions API usage totals under-reporting vs. balance for API clients that invoke a subagent. Tracking it as a known deferred item to fold into the API usage tracker (or derive response usage from billed usage) as part of the SDK migration, rather than growing the legacy controller here.
e0995b6 to
b6e85b8
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b6e85b8baf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| user: { id: userId }, | ||
| /** Bills subagent child-run model calls (reported outside the | ||
| * streamEvents loop) into the same collectedUsage array. */ | ||
| subagentUsageSink: createSubagentUsageSink(collectedUsage), |
There was a problem hiding this comment.
When an OpenAI-compatible request runs an agent that calls a subagent, this sink now appends the child call to collectedUsage so it is billed, but the usage returned to the API client is still built from the streaming tracker/non-streaming aggregator updated only by on_chat_model_end. Those handlers never see child-run events, so clients receive a usage total that omits tokens they were charged for; compose the sink to update the API usage tracker/aggregator (and apply the same pattern to the Responses API wiring) or otherwise expose the billed subagent usage.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Same item as the earlier thread on this line — re-detected on the lockfile-bump commit (the openai.js code is unchanged). Intentionally deferred: openai.js/responses.js are part of the OpenAI-compatible surface being migrated into the @librechat/agents SDK (handler fixes here were descoped from #13636 for the same reason). Billing is correct via the sink; only the Chat-Completions usage field under-reports for API clients that invoke a subagent, and the main chat path is unaffected. Tracked as a known deferred item to fold into the API usage tracker (or derive response usage from billed usage) during the SDK migration, rather than growing the legacy controller in this PR.
GitNexus: ❌ deploy failedThe deploy failed — the previous index (if any) continues to be served. |
|
@codex review |
|
Codex Review: Didn't find any major issues. Breezy! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
…nts 3.2.35) Brings in #13683 (subagent usage billing via subagentUsageSink, 3-way recordCollectedUsage partition, agents SDK 3.2.35 lockfile). usage.ts and IJobStore.ts auto-merged cleanly — the context-usage changes (splitUsage, resolveCompletionTokens, inputTokensIncludesCache, contextUsage/tokenUsage fields) compose with the subagent segregation. No code conflicts.
…breChat-AI#13683) * 🧾 fix: Bill Subagent Child-Run Model Usage in Parent Transactions * 🩹 fix: Type Subagent Usage Sink Structurally Until SDK Release * 🔧 chore: Update @librechat/agents dependency to version 3.2.35 in package-lock.json and related package.json files
…breChat-AI#13683) * 🧾 fix: Bill Subagent Child-Run Model Usage in Parent Transactions * 🩹 fix: Type Subagent Usage Sink Structurally Until SDK Release * 🔧 chore: Update @librechat/agents dependency to version 3.2.35 in package-lock.json and related package.json files
Summary
I fixed a billing leak where model calls made inside subagent child runs were never billed: the child graph executes outside the run's
streamEventsloop, so itschat_model_endevents never reachModelEndHandler— child usage never landed incollectedUsage,spendTokensnever recorded it, and balance enforcement was blind to it. The fix consumes the newsubagentUsageSinkfrom the agents SDK (LibreChat-AI/agents#236) and routes child usage through the existing transaction pipeline.createSubagentUsageSink(collectedUsage)topackages/api/src/agents/usage.ts: receives the SDK's per-callSubagentUsageEvents and pushes them onto the samecollectedUsagearrayModelEndHandlerfills, taggedusage_type: 'subagent'with the child's model and provider (the provider drives the existing subset-vs-additive cache pricing semantics insplitUsage).createRuncall sites —client.js(chat),openai.js(OpenAI-compat completions), andresponses.js(Responses API, both handlers) — so every surface that can spawn subagents bills them; nested subagents report through the same sink.usage_type: 'subagent'entries into their own group inrecordCollectedUsage, billed in full (transactions + balance) under transaction context'subagent'with the child's model. The abort path reusesrecordCollectedUsage, so child usage on aborted runs is billed too.output_tokenstotal: that result becomes the parent response message'stokenCount(viagetStreamUsage), and child output the parent never saw would distort next-turn context accounting;input_tokenslikewise stays pinned to the parent's first message usage.subagentUsageSinkthroughcreateRunintoRun.create, and widened theUsageMetadata.usage_typeunion with'subagent'.SubagentUsageEvent,Run.createconfig built as a variable) so the build and CI pass against the published@librechat/agents@3.2.33. The PR is safe to merge before the SDK release: older SDK versions ignore the extra config field, and the sink activates once the dependency is bumped past 3.2.33 (💸 feat: Report Subagent Child-Run Model Usage via Usage Sink agents#236). The structural types can then be swapped forPick<RunConfig, 'subagentUsageSink'>and the SDK type import — call sites are commented accordingly.Change Type
Testing
packages/api/src/agents/usage.spec.ts: subagent entries billed under context'subagent'with the child model while excluded from reported totals; leading subagent entry cannot hijackinput_tokens; cache-token entries route through structured spend;createSubagentUsageSinktagging, detail preservation, missing-model/provider omission, no-usage guard; and a sink →collectedUsage→recordCollectedUsageround-trip asserting child transactions alongside the parent's.@librechat/apimocks inopenai.spec.js/responses.unit.spec.jsfor the new export.@librechat/agents@3.2.33:tscforpackages/apiclean, ESLint/Prettier/import-sort clean on all changed files, and all touched suites pass (107 packages/api usage/run tests, 284server/controllers/agentstests, full packages/api unit suite green locally).ModelEndHandlerpath.Test Configuration:
@librechat/agents@3.2.33for type/lint/test verification; locally built SDK from 💸 feat: Report Subagent Child-Run Model Usage via Usage Sink agents#236 for the live end-to-end run.Checklist