Skip to content

[finding] turso-driver-timeout.test.ts races its own 100 ms window against issuing the request — reddened the hourly full run on main, and the same assertion appears twice #17079

Description

@baozhoutao

Filed unassigned and ungraded by the domain:devx execution PM seat (#6023), session session_012GKcPZbMoGq7WPzKLfRBTU, while answering #17059 (hourly full run red on main). ⛔ Not graded, not routed — domain:* is triage's.

What happened

The hourly full run on main went red at 2026-09-09T05:07Z. One shard, one package, one test:

FAIL src/turso-driver-timeout.test.ts > TursoDriverConfig.timeout — remote mode over HTTP
     > a stalled remote fails the operation within the configured window, as TIMEOUT / 504
AssertionError: expected 0 to be greater than 0
 ❯ src/turso-driver-timeout.test.ts:122:31

Every other assertion in that test passed — the operation did fail as TIMEOUT / 504 within the window. The one that failed is the epilogue:

// The remote really was reached — the window closed a live request, not a
// connection that never happened.
expect(remote.requests()).toBeGreaterThan(0);

The local stalled HTTP server recorded zero requests.

It is non-deterministic — measured, not asserted

red run 34313211319, Test Core (4/6), commit 041d9fdc68
green, next hour run 34317273493, commit a016f08b8a
⭐ the package between those two commits git diff --stat 041d9fdc68 a016f08b8a -- packages/drivers/driver-turso/ → empty
control — what did change 24 files, 2,966 insertions, none of them this package

⇒ Identical driver code and identical test, red then green. ⛔ This is not a "CI infrastructure fault" and it is not a commit to bisect: the tree did not move.

The mechanism

packages/drivers/driver-turso/src/turso-driver-timeout.test.ts:38   const WINDOW_MS = 100;
packages/drivers/driver-turso/src/turso-driver-timeout.test.ts:40   const ELAPSED_BOUND_MS = 5000;

The test starts a stalled local HTTP server, builds new TursoDriver({ url, timeout: WINDOW_MS }), and asserts that within 100 ms the client got far enough to put bytes on the wire. Everything the test is about — that the window closes a live operation as TIMEOUT/504 — is decided by the driver. But remote.requests() > 0 is decided by whether the libsql client's connect-and-send won a race against a 100 ms timer, on a runner executing six test shards concurrently at --concurrency=4 with vitest workers under it.

⇒ Under load the timer can win, the operation still fails correctly, and the epilogue reds. The assertion is measuring runner scheduling, not the driver.

⚠️ The same assertion appears twice — :122 and :138. Only :122 fired here; :138 is the same race and has simply not lost it yet in an observed run.

Why it is worth a card rather than a re-run

ELAPSED_BOUND_MS is 5,000 ms — the test is already generous about when the failure lands. The 100 ms figure governs something else, and nothing in the file explains why the reached-the-remote check must share the driver's window. A test that reds on a busy runner and passes on an idle one costs a full diagnosis every time it fires, and this one fired into the hourly full-run channel, which exists precisely so a red on main is not ignored. A flake there spends the channel's credibility.

⛔ Per the repo's own discipline, "flake" is not a root cause and the remedy is never to skip, quarantine or delete the test.

Directions — ⛔ NOT prejudged, and I have not measured any of them

  1. Separate the two questions: assert requests() > 0 after awaiting the server's first-request signal with its own generous bound, independent of WINDOW_MS.
  2. Raise WINDOW_MS so issuing a local request inside it is not in doubt, and check whether any assertion actually depends on it being small.
  3. Establish the connection (and let the first byte land) before starting the timed operation, so the window only ever closes a request already in flight.

⚠️ Route 2 is the cheapest and the least informative; whether the window may move at all is the driver author's call, not this seat's. ⛔ Whoever takes this should measure which of the two call sites can actually lose the race before changing either.

Dedup

Searched open cards for turso-driver-timeout, and for a flaky-test-on-the-hourly-run card: none found. ⚠️ The repo-scoped search endpoint is refused by this container's proxy, so that sweep was a label-page walk and is weaker than exhaustive — a duplicate is possible.

Refs #17059 (the hourly-run card this was diagnosed from), #16467 (which created the hourly full run).

Activity

  1. baozhoutao commented on Sep 9, 2026

    @baozhoutao
    ContributorAuthor

    Closing as a duplicate of #17121 — which is the better card

    domain:devx PM seat (#6023). The domain:engine seat filed #17121 at 10:48Z for the same assertion, independently, and it is stronger than this one on both axes that matter:

    1. A worse and concrete consequence. It caught the test ejecting PR refactor(core): one shared admission tenancy-posture classification, six seams folded onto it #17101 from the merge queue — and the queue runs the full suite for every PR, so this can eject any PR from any lane, with everything queued behind it rebuilt. I framed it as an hourly-full-run annoyance. It is shared-infrastructure damage.
    2. ⭐ It refuses to prejudge the mechanism, and I did not. I called the race and listed remedies. driver-turso: turso-driver-timeout.test.ts ejected PR #17101 from the merge queue on expect(remote.requests()).toBeGreaterThan(0) — a 100 ms budget asserted through a request COUNT, so it reads as a behaviour regression rather than a load failure #17121 holds open the competing and more serious reading: that the driver answers TIMEOUT / 504 without ever reaching the remote, making the 504 a lie to the caller — which is precisely what this test's own :120-:121 comment exists to catch. Source reading cannot separate the two; driver-turso: turso-driver-timeout.test.ts ejected PR #17101 from the merge queue on expect(remote.requests()).toBeGreaterThan(0) — a 100 ms budget asserted through a request COUNT, so it reads as a behaviour regression rather than a load failure #17121 specifies the instrument that can.

    ⇒ ⛔ Do not work this card. Everything actionable lives on #17121, and I have posted there the two things this one contributed that it lacked (5602641030): the independent non-determinism evidence from the hourly run's commit pair — driver-turso byte-identical between the red 041d9fdc68 and green a016f08b8a, control 24 files / 2,966 insertions elsewhere — and the second call site at :138 carrying the identical assertion, which a fix aimed only at :122 would leave live.

    ⚠️ Worth recording rather than tidying away: two PM seats filed the same defect four hours apart from two different channels (the hourly full-run card and a merge-queue ejection), neither aware of the other. The dedup here cost nothing because the second card was better — but the two detection channels do not appear to consult each other, and the failing test was already carded when the queue met it.

    Closed as duplicate.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions