Skip to content

Flaky integration test: blob.test.mjs 'Per-device-type LMDB database sharding' — schema missing from describe_all (shard 3/6) #1557

Description

@heskew

Symptom

integrationTests/apiTests/blob.test.mjs → "Per-device-type LMDB database sharding" fails intermittently in CI on Integration Tests shard 3/6, and passes on re-run with no code change. Two occurrences on the same PR (#1106) on 2026-07-01/02 alone — a PR whose diff is a pure formatter + test files that never touch blob/sharding code:

  • Run 28542021803, Node v22: AssertionError: doorlock schema missing (in all three device-type schemas are visible in describe_all), then cascading SQL queries target each device database independently: expected 200 "OK", got 404 "Not Found", with HdbError: Table 'doorlock.DoorlockBlob' does not exist in the Harper log. Passed on re-run.
  • Run 28558632042, Node v22 (re-run): same shape, different fixture — AssertionError: sensor schema missing, then the same 200→404 cascade. Passed on re-run.

Both times the same shard passed simultaneously on Bun, Node v24, Node v26, and Windows — the failure is timing-dependent, not version-deterministic (observed twice on v22, but the mechanism looks like scheduling luck rather than a v22 behavior).

Shape of the race

The failing assertion is schema visibility: a device-type schema (doorlock, sensor — varies per occurrence) created during setup is missing from a subsequent describe_all, and SQL against that device database then 404s. The suite itself times out around ~30s (Per-device-type LMDB database sharding (29992ms / 35174ms)). This smells like create-schema/table completing on one worker thread before the metadata is visible to the thread serving the next request — same eventual-consistency family as #1497 (async ITC not awaited) and adjacent to #687 (create_table returns success before table is available). Possibly a condition-wait issue per the AGENTS.md guidance (prefer waitFor(condition) over fixed sleeps).

Context

Acceptance

  • Root-cause the visibility race (or confirm it's a fixture-setup race) and fix, OR make the test's setup wait on actual schema visibility (waitFor-style condition on describe_all contents) rather than assuming synchronous visibility.
  • No recurrence across ~20 consecutive shard-3 CI runs.

🤖 Generated with Claude Code

Activity

  1. added this to the v5.2 milestone on Aug 7, 2026
  2. added theissue type on Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    P2

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions