feat(routing): per-tier and per-provider stream warmup timeout override - #2774
Open
crashf wants to merge 4 commits into
Open
feat(routing): per-tier and per-provider stream warmup timeout override#2774crashf wants to merge 4 commits into
crashf wants to merge 4 commits into
Conversation
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
14 issues found across 19 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/backend/src/routing/proxy/chatgpt-adapter.ts">
<violation number="1" location="packages/backend/src/routing/proxy/chatgpt-adapter.ts:375">
P2: When `response.function_call_arguments.delta` arrives without a parsed `response.output_item.added`, `sawToolCall` stays false and an empty completed output is finalized as `stop`. Set `sawToolCall` in the arguments-delta branch as well, since that branch already emits a tool-call chunk.</violation>
</file>
<file name="packages/backend/src/entities/tenant-provider.entity.ts">
<violation number="1" location="packages/backend/src/entities/tenant-provider.entity.ts:46">
P1: On databases upgraded from an existing installation, this entity field is active before its migration runs because `AddStreamWarmupMs1802100000000` is missing from the explicit migration registry. Register the migration in `data-source-definitions.ts`; otherwise provider reads and writes can fail with a missing-column error.</violation>
</file>
<file name="packages/backend/src/database/migrations/1802100000000-AddStreamWarmupMs.ts">
<violation number="1" location="packages/backend/src/database/migrations/1802100000000-AddStreamWarmupMs.ts:17">
P2: When `npm run migration:revert` runs this `down()` method with `--transaction none`, `SET LOCAL` has no effect outside a transaction, leaving the column drops unbounded. Use session-scoped `SET lock_timeout = '5s'` with a `RESET lock_timeout` in cleanup, or run this rollback in a transaction.</violation>
<violation number="2" location="packages/backend/src/database/migrations/1802100000000-AddStreamWarmupMs.ts:17">
P1: Every execution of this migration fails at the first `SET LOCAL` because PostgreSQL cannot parse the unquoted `5s` value, so neither stream-warmup column is added. Quote the duration in both `up()` and `down()`.</violation>
</file>
<file name="packages/frontend/src/components/HeaderTierModal.tsx">
<violation number="1" location="packages/frontend/src/components/HeaderTierModal.tsx:188">
P1: When the timeout is invalid, this check runs after the tier and response-mode API calls have already persisted changes. Validate the warmup value before any mutation so a failed create does not leave an undisplayed tier or partial edit.</violation>
<violation number="2" location="packages/frontend/src/components/HeaderTierModal.tsx:192">
P2: Fractional values pass this client check but fail the endpoint's integer validation. Require `Number.isInteger(warmupVal)` here so the form rejects the same values before sending the request.</violation>
</file>
<file name="packages/backend/src/routing/proxy/stream-warmup.ts">
<violation number="1" location="packages/backend/src/routing/proxy/stream-warmup.ts:29">
P2: clampStreamWarmupMs has no unit tests even though its sibling functions are tested in stream-warmup.spec.ts. Its rounding, string/number coercion, and the 1s–120s bound where out-of-range returns undefined (triggering inherit/fallback) is exactly the boundary logic that should be pinned down, and it now gates per-tier/per-provider routing overrides.</violation>
</file>
<file name="packages/frontend/src/components/CustomProviderForm.tsx">
<violation number="1" location="packages/frontend/src/components/CustomProviderForm.tsx:368">
P2: When adding a provider, the new timeout control is shown, but `handleCreate` never includes `stream_warmup_ms` in its payload. The value is silently lost and only becomes configurable after a later edit; persist it during creation or hide the control until edit mode.</violation>
</file>
<file name="packages/backend/src/routing/proxy/proxy.service.ts">
<violation number="1" location="packages/backend/src/routing/proxy/proxy.service.ts:613">
P2: When the primary fails before this block, `tryFallbackChain` returns the first successful fallback stream directly, so this resolver is never called for that route. Apply the warmup check and resolved tier/provider timeout to fallback successes too; otherwise a slow fallback can stall the client without the new override.</violation>
</file>
<file name="packages/backend/src/routing/dto/custom-provider.dto.ts">
<violation number="1" location="packages/backend/src/routing/dto/custom-provider.dto.ts:107">
P2: The provider DTO validates stream_warmup_ms with @Min(1000) but no upper bound, despite the PR specifying 1s–120s and the header-tier DTO enforcing @Max(120000). Values above 120000 are persisted un-clamped, then silently dropped by clampStreamWarmupMs in the proxy, so a user who sets e.g. 300s on a provider gets the default 15s with no error or feedback. Add @Max(120000) to match the tier validation and the documented bounds.</violation>
<violation number="2" location="packages/backend/src/routing/dto/custom-provider.dto.ts:107">
P2: The provider-level `stream_warmup_ms` DTO is missing `@Max(120000)`, which the tier endpoint (`StreamWarmupBody` in header-tier.controller.ts has `@Min(1000) @Max(120000)`) and the proxy's `clampStreamWarmupMs` both enforce. A value above 120000 is accepted and stored on `tenant_providers`, but then `clampStreamWarmupMs` returns undefined at resolve time, so the override is silently ignored and the tier/global value applies instead. Add the same max bound so out-of-range provider values are rejected at the API boundary.</violation>
</file>
<file name="packages/backend/src/routing/routing-core/provider.service.ts">
<violation number="1" location="packages/backend/src/routing/routing-core/provider.service.ts:454">
P2: When the companion row this finds is inactive (or inactive rows exist), the new warmup value is written to a row the proxy never reads, because resolveStreamWarmupMs only selects rows with `is_active`. The setting then silently has no effect. Match the reader's predicate by adding `is_active: true` to the findOne (and consider ordering by updated_at) so the setter always targets the row the proxy resolves.</violation>
</file>
<file name="packages/backend/src/routing/custom-provider/custom-provider.service.ts">
<violation number="1" location="packages/backend/src/routing/custom-provider/custom-provider.service.ts:317">
P2: The stream-warmup sync runs before the apiKey/upsert logic below it, but it keys on (tenant_id, provider) only (setCustomProviderStreamWarmup uses providerRepo.findOne without auth_type) while upsertProvider/retagAuthType can reseat the companion row under a different auth_type. When the same edit both renames the provider across categories (e.g. "LM Studio" → a freeform name) and supplies apiKey, upsertProvider creates a fresh tenant_providers row for nextAuthType because the unique index is (tenant_id, provider, auth_type), leaving stream_warmup_ms written on the old row. The value then only reaches the proxy through the leftover stale row (the resolver's `ORDER BY updated_at DESC` matches it), so the override is not reliably attached to the row that ends up routing traffic. Compute nextAuthType/nameCategoryChanged and run the stream warmup write after the apiKey/retag block targeting the final row.</violation>
</file>
<file name="docker/DOCKER_README.md">
<violation number="1" location="docker/DOCKER_README.md:389">
P3: The doc states the warmup window is "clamped to 1–120 s", but that is not how the resolution works. Tier and provider overrides are run through `clampStreamWarmupMs`, which returns `undefined` for values outside 1000–120000 ms — meaning out-of-bounds overrides are *dropped* and fall through to the next level, not clamped to the boundary. The global `STREAM_WARMUP_MS` env var is not clamped at all: `parseStreamWarmupMs` accepts any positive integer (e.g. 30000 or 200000) and uses it verbatim. A user following the env table could reasonably set `STREAM_WARMUP_MS=180000` expecting it to be capped at 120 s; it is used as-is. Consider rewording to "first non-null wins; tier/provider overrides outside 1–120 s are ignored".</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| is_active!: boolean; | ||
|
|
||
| /** Per-provider stream warmup override (ms). null => global default applies. */ | ||
| @Column('integer', { nullable: true }) |
There was a problem hiding this comment.
P1: On databases upgraded from an existing installation, this entity field is active before its migration runs because AddStreamWarmupMs1802100000000 is missing from the explicit migration registry. Register the migration in data-source-definitions.ts; otherwise provider reads and writes can fail with a missing-column error.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/backend/src/entities/tenant-provider.entity.ts, line 46:
<comment>On databases upgraded from an existing installation, this entity field is active before its migration runs because `AddStreamWarmupMs1802100000000` is missing from the explicit migration registry. Register the migration in `data-source-definitions.ts`; otherwise provider reads and writes can fail with a missing-column error.</comment>
<file context>
@@ -42,6 +42,10 @@ export class TenantProvider {
is_active!: boolean;
+ /** Per-provider stream warmup override (ms). null => global default applies. */
+ @Column('integer', { nullable: true })
+ stream_warmup_ms!: number | null;
+
</file context>
| public async up(queryRunner: QueryRunner): Promise<void> { | ||
| // Catalog-only adds, but they take ACCESS EXCLUSIVE — bound the wait so a | ||
| // deploy queues behind a long read instead of blocking the table. | ||
| await queryRunner.query(`SET LOCAL lock_timeout = 5s`); |
There was a problem hiding this comment.
P1: Every execution of this migration fails at the first SET LOCAL because PostgreSQL cannot parse the unquoted 5s value, so neither stream-warmup column is added. Quote the duration in both up() and down().
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/backend/src/database/migrations/1802100000000-AddStreamWarmupMs.ts, line 17:
<comment>Every execution of this migration fails at the first `SET LOCAL` because PostgreSQL cannot parse the unquoted `5s` value, so neither stream-warmup column is added. Quote the duration in both `up()` and `down()`.</comment>
<file context>
@@ -0,0 +1,33 @@
+ public async up(queryRunner: QueryRunner): Promise<void> {
+ // Catalog-only adds, but they take ACCESS EXCLUSIVE — bound the wait so a
+ // deploy queues behind a long read instead of blocking the table.
+ await queryRunner.query(`SET LOCAL lock_timeout = 5s`);
+ await queryRunner.query(
+ `ALTER TABLE "header_tiers" ADD COLUMN IF NOT EXISTS "stream_warmup_ms" integer DEFAULT NULL`,
</file context>
| saved = await setHeaderTierResponseMode(props.agentName, saved.id, newMode); | ||
| } | ||
| // Persist stream warmup override if changed | ||
| const warmupRaw = streamWarmup().trim(); |
There was a problem hiding this comment.
P1: When the timeout is invalid, this check runs after the tier and response-mode API calls have already persisted changes. Validate the warmup value before any mutation so a failed create does not leave an undisplayed tier or partial edit.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/frontend/src/components/HeaderTierModal.tsx, line 188:
<comment>When the timeout is invalid, this check runs after the tier and response-mode API calls have already persisted changes. Validate the warmup value before any mutation so a failed create does not leave an undisplayed tier or partial edit.</comment>
<file context>
@@ -178,6 +184,21 @@ const HeaderTierModal: Component<Props> = (props) => {
saved = await setHeaderTierResponseMode(props.agentName, saved.id, newMode);
}
+ // Persist stream warmup override if changed
+ const warmupRaw = streamWarmup().trim();
+ const warmupVal = warmupRaw === '' ? null : Number(warmupRaw);
+ if (
</file context>
| if (!data) return null; | ||
| const item = isObjectRecord(data.item) ? data.item : undefined; | ||
| if (item?.type !== 'function_call') return null; | ||
| if (state) state.sawToolCall = true; |
There was a problem hiding this comment.
P2: When response.function_call_arguments.delta arrives without a parsed response.output_item.added, sawToolCall stays false and an empty completed output is finalized as stop. Set sawToolCall in the arguments-delta branch as well, since that branch already emits a tool-call chunk.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/backend/src/routing/proxy/chatgpt-adapter.ts, line 375:
<comment>When `response.function_call_arguments.delta` arrives without a parsed `response.output_item.added`, `sawToolCall` stays false and an empty completed output is finalized as `stop`. Set `sawToolCall` in the arguments-delta branch as well, since that branch already emits a tool-call chunk.</comment>
<file context>
@@ -370,6 +372,7 @@ export function transformResponsesStreamChunk(
if (!data) return null;
const item = isObjectRecord(data.item) ? data.item : undefined;
if (item?.type !== 'function_call') return null;
+ if (state) state.sawToolCall = true;
return formatSSE(
{
</file context>
| public async up(queryRunner: QueryRunner): Promise<void> { | ||
| // Catalog-only adds, but they take ACCESS EXCLUSIVE — bound the wait so a | ||
| // deploy queues behind a long read instead of blocking the table. | ||
| await queryRunner.query(`SET LOCAL lock_timeout = 5s`); |
There was a problem hiding this comment.
P2: When npm run migration:revert runs this down() method with --transaction none, SET LOCAL has no effect outside a transaction, leaving the column drops unbounded. Use session-scoped SET lock_timeout = '5s' with a RESET lock_timeout in cleanup, or run this rollback in a transaction.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/backend/src/database/migrations/1802100000000-AddStreamWarmupMs.ts, line 17:
<comment>When `npm run migration:revert` runs this `down()` method with `--transaction none`, `SET LOCAL` has no effect outside a transaction, leaving the column drops unbounded. Use session-scoped `SET lock_timeout = '5s'` with a `RESET lock_timeout` in cleanup, or run this rollback in a transaction.</comment>
<file context>
@@ -0,0 +1,33 @@
+ public async up(queryRunner: QueryRunner): Promise<void> {
+ // Catalog-only adds, but they take ACCESS EXCLUSIVE — bound the wait so a
+ // deploy queues behind a long read instead of blocking the table.
+ await queryRunner.query(`SET LOCAL lock_timeout = 5s`);
+ await queryRunner.query(
+ `ALTER TABLE "header_tiers" ADD COLUMN IF NOT EXISTS "stream_warmup_ms" integer DEFAULT NULL`,
</file context>
Comment on lines
+107
to
+110
| @Min(1000) | ||
| @Type(() => Number) | ||
| stream_warmup_ms?: number | null; | ||
|
|
There was a problem hiding this comment.
P2: The provider DTO validates stream_warmup_ms with @min(1000) but no upper bound, despite the PR specifying 1s–120s and the header-tier DTO enforcing @max(120000). Values above 120000 are persisted un-clamped, then silently dropped by clampStreamWarmupMs in the proxy, so a user who sets e.g. 300s on a provider gets the default 15s with no error or feedback. Add @max(120000) to match the tier validation and the documented bounds.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/backend/src/routing/dto/custom-provider.dto.ts, line 107:
<comment>The provider DTO validates stream_warmup_ms with @Min(1000) but no upper bound, despite the PR specifying 1s–120s and the header-tier DTO enforcing @Max(120000). Values above 120000 are persisted un-clamped, then silently dropped by clampStreamWarmupMs in the proxy, so a user who sets e.g. 300s on a provider gets the default 15s with no error or feedback. Add @Max(120000) to match the tier validation and the documented bounds.</comment>
<file context>
@@ -102,6 +102,12 @@ export class ProbeCustomProviderDto {
export class UpdateCustomProviderDto {
+ @IsOptional()
+ @IsNumber()
+ @Min(1000)
+ @Type(() => Number)
+ stream_warmup_ms?: number | null;
</file context>
Suggested change
| @Min(1000) | |
| @Type(() => Number) | |
| stream_warmup_ms?: number | null; | |
| @IsOptional() | |
| @IsNumber() | |
| @Min(1000) | |
| @Max(120000) | |
| @Type(() => Number) | |
| stream_warmup_ms?: number | null; |
Comment on lines
+454
to
+456
| const row = await this.providerRepo.findOne({ | ||
| where: { tenant_id: tenantId, provider: providerKey }, | ||
| }); |
There was a problem hiding this comment.
P2: When the companion row this finds is inactive (or inactive rows exist), the new warmup value is written to a row the proxy never reads, because resolveStreamWarmupMs only selects rows with is_active. The setting then silently has no effect. Match the reader's predicate by adding is_active: true to the findOne (and consider ordering by updated_at) so the setter always targets the row the proxy resolves.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/backend/src/routing/routing-core/provider.service.ts, line 454:
<comment>When the companion row this finds is inactive (or inactive rows exist), the new warmup value is written to a row the proxy never reads, because resolveStreamWarmupMs only selects rows with `is_active`. The setting then silently has no effect. Match the reader's predicate by adding `is_active: true` to the findOne (and consider ordering by updated_at) so the setter always targets the row the proxy resolves.</comment>
<file context>
@@ -442,6 +442,24 @@ export class ProviderService {
+ providerKey: string,
+ streamWarmupMs: number | null,
+ ): Promise<void> {
+ const row = await this.providerRepo.findOne({
+ where: { tenant_id: tenantId, provider: providerKey },
+ });
</file context>
Suggested change
| const row = await this.providerRepo.findOne({ | |
| where: { tenant_id: tenantId, provider: providerKey }, | |
| }); | |
| const row = await this.providerRepo.findOne({ | |
| where: { tenant_id: tenantId, provider: providerKey, is_active: true }, | |
| order: { is_active: 'DESC', updated_at: 'DESC' }, | |
| }); |
|
|
||
| // Stream warmup lives on the companion tenant_providers row (the row the | ||
| // proxy reads). Sync it when the modal sends the field; null clears it. | ||
| if ('stream_warmup_ms' in dto) { |
There was a problem hiding this comment.
P2: The stream-warmup sync runs before the apiKey/upsert logic below it, but it keys on (tenant_id, provider) only (setCustomProviderStreamWarmup uses providerRepo.findOne without auth_type) while upsertProvider/retagAuthType can reseat the companion row under a different auth_type. When the same edit both renames the provider across categories (e.g. "LM Studio" → a freeform name) and supplies apiKey, upsertProvider creates a fresh tenant_providers row for nextAuthType because the unique index is (tenant_id, provider, auth_type), leaving stream_warmup_ms written on the old row. The value then only reaches the proxy through the leftover stale row (the resolver's ORDER BY updated_at DESC matches it), so the override is not reliably attached to the row that ends up routing traffic. Compute nextAuthType/nameCategoryChanged and run the stream warmup write after the apiKey/retag block targeting the final row.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/backend/src/routing/custom-provider/custom-provider.service.ts, line 317:
<comment>The stream-warmup sync runs before the apiKey/upsert logic below it, but it keys on (tenant_id, provider) only (setCustomProviderStreamWarmup uses providerRepo.findOne without auth_type) while upsertProvider/retagAuthType can reseat the companion row under a different auth_type. When the same edit both renames the provider across categories (e.g. "LM Studio" → a freeform name) and supplies apiKey, upsertProvider creates a fresh tenant_providers row for nextAuthType because the unique index is (tenant_id, provider, auth_type), leaving stream_warmup_ms written on the old row. The value then only reaches the proxy through the leftover stale row (the resolver's `ORDER BY updated_at DESC` matches it), so the override is not reliably attached to the row that ends up routing traffic. Compute nextAuthType/nameCategoryChanged and run the stream warmup write after the apiKey/retag block targeting the final row.</comment>
<file context>
@@ -312,6 +312,16 @@ export class CustomProviderService {
+ // Stream warmup lives on the companion tenant_providers row (the row the
+ // proxy reads). Sync it when the modal sends the field; null clears it.
+ if ('stream_warmup_ms' in dto) {
+ await this.providerService.setCustomProviderStreamWarmup(
+ tenantId,
</file context>
| export class UpdateCustomProviderDto { | ||
| @IsOptional() | ||
| @IsNumber() | ||
| @Min(1000) |
There was a problem hiding this comment.
P2: The provider-level stream_warmup_ms DTO is missing @Max(120000), which the tier endpoint (StreamWarmupBody in header-tier.controller.ts has @Min(1000) @Max(120000)) and the proxy's clampStreamWarmupMs both enforce. A value above 120000 is accepted and stored on tenant_providers, but then clampStreamWarmupMs returns undefined at resolve time, so the override is silently ignored and the tier/global value applies instead. Add the same max bound so out-of-range provider values are rejected at the API boundary.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/backend/src/routing/dto/custom-provider.dto.ts, line 107:
<comment>The provider-level `stream_warmup_ms` DTO is missing `@Max(120000)`, which the tier endpoint (`StreamWarmupBody` in header-tier.controller.ts has `@Min(1000) @Max(120000)`) and the proxy's `clampStreamWarmupMs` both enforce. A value above 120000 is accepted and stored on `tenant_providers`, but then `clampStreamWarmupMs` returns undefined at resolve time, so the override is silently ignored and the tier/global value applies instead. Add the same max bound so out-of-range provider values are rejected at the API boundary.</comment>
<file context>
@@ -102,6 +102,12 @@ export class ProbeCustomProviderDto {
export class UpdateCustomProviderDto {
+ @IsOptional()
+ @IsNumber()
+ @Min(1000)
+ @Type(() => Number)
+ stream_warmup_ms?: number | null;
</file context>
| fallback tier, so slow-but-healthy local models get skipped. | ||
|
|
||
| The warmup window resolves in this order (first non-null wins, clamped to | ||
| 1–120 s): |
There was a problem hiding this comment.
P3: The doc states the warmup window is "clamped to 1–120 s", but that is not how the resolution works. Tier and provider overrides are run through clampStreamWarmupMs, which returns undefined for values outside 1000–120000 ms — meaning out-of-bounds overrides are dropped and fall through to the next level, not clamped to the boundary. The global STREAM_WARMUP_MS env var is not clamped at all: parseStreamWarmupMs accepts any positive integer (e.g. 30000 or 200000) and uses it verbatim. A user following the env table could reasonably set STREAM_WARMUP_MS=180000 expecting it to be capped at 120 s; it is used as-is. Consider rewording to "first non-null wins; tier/provider overrides outside 1–120 s are ignored".
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At docker/DOCKER_README.md, line 389:
<comment>The doc states the warmup window is "clamped to 1–120 s", but that is not how the resolution works. Tier and provider overrides are run through `clampStreamWarmupMs`, which returns `undefined` for values outside 1000–120000 ms — meaning out-of-bounds overrides are *dropped* and fall through to the next level, not clamped to the boundary. The global `STREAM_WARMUP_MS` env var is not clamped at all: `parseStreamWarmupMs` accepts any positive integer (e.g. 30000 or 200000) and uses it verbatim. A user following the env table could reasonably set `STREAM_WARMUP_MS=180000` expecting it to be capped at 120 s; it is used as-is. Consider rewording to "first non-null wins; tier/provider overrides outside 1–120 s are ignored".</comment>
<file context>
@@ -377,6 +377,28 @@ For vLLM, text-generation-webui, TogetherAI proxies, Azure OpenAI gateways, or a
+fallback tier, so slow-but-healthy local models get skipped.
+
+The warmup window resolves in this order (first non-null wins, clamped to
+1–120 s):
+
+1. **Tier override** — per custom routing tier (tier edit → *Stream timeout
</file context>
crashf
force-pushed
the
feat/stream-warmup-timeout
branch
from
August 27, 2026 15:43
1a6c215 to
1cf01d5
Compare
- HeaderTier + TenantProvider entities: stream_warmup_ms integer nullable column - PATCH /routing/:agentName/header-tiers/:id/stream-warmup (validate 1s-120s, null=inherit) - proxy resolveStreamWarmupMs: tier -> provider -> STREAM_WARMUP_MS env -> 15s default (60s cache) - stream-warmup.ts: clampStreamWarmupMs + MIN/MAX bounds - Header-tier modal: Stream timeout (ms) field
…odal - custom-provider update DTO accepts stream_warmup_ms (1s-120s clamp, null=inherit) - service syncs the value onto the companion tenant_providers row the proxy reads - controller returns stream_warmup_ms so the form reflects the current value - provider.service.setCustomProviderStreamWarmup helper - CustomProviderForm: Stream timeout (ms) field under Base URL with cold-load hint - docs: STREAM_WARMUP_MS env row + full precedence section in DOCKER_README
- AddStreamWarmupMs1802100000000: nullable stream_warmup_ms on header_tiers and tenant_providers (no backfill; null = inherit, keeps default behavior) - .changeset/stream-warmup-timeout.md: minor bump, release notes
crashf
force-pushed
the
feat/stream-warmup-timeout
branch
from
August 27, 2026 15:47
1cf01d5 to
3d41fd6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When Manifest proxies a streaming request, it has to decide how long to wait for the first token before concluding the chosen model is unresponsive and trying the next route. That window is currently a fixed 15 seconds.
Cloud models respond well within it. Self-hosted local models often don't: a cold Ollama model load or an LM Studio just-in-time load can take 30–60 s before the server starts streaming. The result is that slow-but-healthy local models get skipped entirely — Manifest falls over to a fallback tier, or returns a failure, even though the primary would have answered seconds later.
Fix
Make the stream warmup timeout configurable at three levels, with the first match winning:
stream_warmup_mscolumn onheader_tierstream_warmup_mscolumn ontenant_providerSTREAM_WARMUP_MSenv var (default: 15000)Changes
header-tier.entity.ts/tenant-provider.entity.ts—stream_warmup_msnullable integer columnheader-tier.controller.ts/header-tier.service.ts—PATCH /:id/stream-warmupendpoint (validate 1s–120s, null = inherit)custom-provider.controller.ts/custom-provider.service.ts/custom-provider.dto.ts— stream warmup field on provider editproxy.service.ts—resolveStreamWarmupMs: tier → provider → env → 15s default (60s cache)stream-warmup.ts—clampStreamWarmupMs+ MIN/MAX boundsprovider.service.ts— provider-level resolve helperHeaderTierModal.tsx— stream timeout field in tier editorCustomProviderForm.tsx— stream timeout field in provider editorapi/header-tiers.ts/api/routing.ts— frontend API bindings1802100000000-AddStreamWarmupMsstream-warmup-timeout.mdDOCKER_README.md—STREAM_WARMUP_MSdocumentationSummary by cubic
Makes the stream warmup timeout—how long the proxy waits for the first token before failing over—configurable per tier and per provider. Previously a fixed 15s caused slow-starting local models (cold Ollama or LM Studio loads taking 30–60s) to be skipped even when healthy. Also fixes GPT/Codex streams ending with
finish_reason: stopwhen a tool call was streamed but the completed payload arrives with an empty output array.Bug Fixes
finish_reasonistool_callseven when the completed payload's output is empty.Written for commit 3d41fd6. Summary will update on new commits.