Skip to content

eve 0.31.3 bump — unblocked: the boot failure was our own probe, not eve - #36

Open
SammyTourani wants to merge 3 commits into
mainfrom
eve-0.31.3
Open

eve 0.31.3 bump — unblocked: the boot failure was our own probe, not eve#36
SammyTourani wants to merge 3 commits into
mainfrom
eve-0.31.3

Conversation

@SammyTourani

@SammyTourani SammyTourani commented Aug 9, 2026

Copy link
Copy Markdown
Owner

Unblocked. The boot failure was ours, not eve's, and the fix is on main.

CORRECTION — this PR previously said "Blocked… the version bump must not ship." That was
written when the cause was unknown, and the conclusion it reached was wrong. Superseded by
#40, which is merged.

What the failure actually was

agent_v31 refused to boot under 0.31.3, four times out of four, with two Zod errors on
completedAt and a Uint8Array. The database was not evidence about eve. Our own runtime
probe had poisoned it.

WorkflowRunSchema's pending/running branch declares output, error and completedAt as
z.undefined().optional(). A single row carrying status='running' and a completed_at
emits both of those messages together, from one parse. Probe 12 is the only thing in the
repository that UPDATEs that table: it forced status='running', spent seconds calling the
fleet API, then wrote the stale status back with no guard. world.start() parses every row
before filtering and outside the try, so one bad row bricks the deployment permanently — which
is why a restart never recovered it.

eve 0.30.8 fails identically on the same row. This was never a regression.

What that retires

Three claims this PR used to rest on, all withdrawn:

  • "0.31.3 cannot restart against a database that holds runs." False — three populated
    databases boot fine, and that was already corrected once before the real cause was found.
  • "The cause is not isolated." It is: The 0.31.3 blocker was our own probe, plus four more the hunt confirmed #40 reproduced it against the installed schema.
  • "A reproducible failure to boot, on data eve itself produced, is disqualifying." The data
    was not produced by eve. It was produced by our probe, writing a shape eve does not allow.

The migrations, unchanged and still correct

  1. ScheduleHandlerArgs.receivetoreceive(channel, {target, message, auth}) becomes
    to(channel, target).send(message, {auth}). The template's heartbeat is the only caller.
  2. createSession no longer demands a continuation token — 0.31.3 returns
    {ok, sessionId, status} and publishes the token on session.waiting. Requiring it made
    every session start a 502 "returned no handles". Contracts pin routes and module exports,
    not response bodies, and this is JSON parsed at runtime — so typecheck saw nothing and only
    the live seam probes caught it.

Measured against 0.31.3 with both: contracts 22/495 green, the Postgres probe tier
13 probes / 239 checks green, the agent tier green, and 5 of 6 seam probes green — the
sixth being the fleet sweep's own anti-vacuity guard reporting 0 candidates, which is the
harness, not a 0.31.3 behaviour.

Before merging

Rebase onto current main and re-run the seam tier. Those numbers were taken before #40, #43
and #45, and a measurement worth quoting is worth retaking.

SammyTourani and others added 2 commits August 9, 2026 01:11
eve-watch opened #31 headed "CONTRACTS BROKEN". That was the detector, not eve —
contract 05 pinned eve's internal constant NAMES, fixed separately in #34. With
it fixed, the upgrade is small and this commit is the whole of it.

MEASURED against a real install of 0.31.3, not read off a changelog:

  contracts   22 contracts, 495 assertions — all green
  build       clean
  typecheck   clean across 10 packages
  routes      every /eve/v1/session* path byte-identical to 0.30.8, so the
              dashboard's and the CLI's hardcoded paths are untouched

── the one breaking change ──────────────────────────────────────────────────

ScheduleHandlerArgs dropped `receive` for `to`, and the call shape moved from
one object to a two-step select-then-send:

  0.30.8   receive(channel, { target, message, auth })
  0.31.3   to(channel, target).send(message, { auth })

Same three inputs, same returned Session promise. templates/default's heartbeat
schedule is the only caller in the workspace, so it is the only file that
changed — the compiler found it, which is the argument for the peer ranges
being `>=0.30.0 <1.0.0` rather than pinned.

The `waitUntil(dispatch); await dispatch;` pair below it is untouched and still
matters for the same reason it did before: `to(...).send()` resolves as soon as
the workflow run has started, so awaiting it records the DISPATCH and not the
turn. That distinction is documented at length in the file and 0.31.3 does not
change it.

Also corrected one doc line in that file's header that still described handing
the turn over "with `receive()`".

── what is NOT in this commit ───────────────────────────────────────────────

Runtime probes against 0.31.3. They need a booted agent and a database, and are
run next rather than assumed — #31's own body reported them failing and nobody
has re-run them since the contract fix.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…cker behind it

Two things the runtime probes found that contracts and typecheck could not, on an
upgrade I had already called "one TypeScript error in one file". That claim was
wrong, and it was wrong because I made it on contracts and typecheck alone.

── FIXED: createSession demanded a continuation token ───────────────────────

0.30.x returned one from POST /eve/v1/session. 0.31.3 answers
`{ok, sessionId, status}` and publishes the token on the `session.waiting`
stream event instead — where it becomes meaningful, since a session that has not
parked has nothing to continue from. Verified against a running 0.31.3: the
event still carries data.continuationToken, unchanged.

Requiring it turned every session start into
`502 "The agent accepted the session but returned no handles"` — the dashboard's
only way to begin a conversation. Nothing else saw it: contract 05 pins routes
and module exports, not response bodies, and this is JSON parsed at runtime, so
typecheck had nothing to check. seam/chat-stream and seam/chat-mutations, both
driving a live agent, failed immediately.

The session id is the handle. Every caller that needs a token already resolves
one from the durable stream — the follow-up route does it whenever the caller
omits one, and the fork route polls getSessionSnapshot for a token it has not
already spent. Neither ever used this field.

After: 5 of 6 seam probes green against 0.31.3. The sixth is the fleet sweep's
own anti-vacuity guard reporting 0 candidates, which is the invalid model key in
that harness closing turns instantly, not a 0.31.3 behaviour.

── NOT FIXED, AND IT BLOCKS THE UPGRADE ─────────────────────────────────────

eve 0.31.3 cannot restart against a database that holds runs. Four Zod errors,
no HTTP listener, agent never comes up:

    "Invalid input: expected undefined, received Date"        path: completedAt
    "Invalid input: expected undefined, received Uint8Array"

Controlled, because the first control I ran was invalid — 0.30.8 appeared to
fail the same way and was actually failing to BUILD in a cold worktree with no
dist/, which is the cold-checkout problem ci.yml documents, not a boot failure.
With the workspace built:

    eve 0.30.8, db with 20 runs    up in 1s, 0 zod errors,
                                   "[world-postgres] Re-enqueued 12 active run(s)"
    eve 0.31.3, db with its runs   never came up, 4 zod errors
    eve 0.31.3, FRESH db           up in 3s, 0 zod errors

So it is specifically the resume path, and it is a regression: 0.30.8 resumes
the same shape of data cleanly. Not a store mismatch either — peer ranges are
identical between the two eve versions, and bumping
@workflow/world-postgres from 5.0.0-beta.31 to beta.32 reproduces it exactly.
That bump is reverted; it changed nothing.

A deployment on 0.31.3 would come up once and fail to come up again. This branch
keeps the migration ready and the version bump does NOT ship until eve fixes it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Aug 9, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
evestack Ready Ready Preview Aug 9, 2026 10:33am

The previous commit says "eve 0.31.3 cannot restart against a database that
holds runs". That is wrong. Three databases holding runs boot fine, in 3–4
seconds, with zero errors.

Same eve 0.31.3, pinned exactly, same build, same process recipe, one database
at a time:

    agent_v3           runs=20  open=12  came up 4s   zod=0
    evestack_agent_v2  runs=9   open=6   came up 3s   zod=0
    agent_v31          runs=27  open=0   NEVER CAME UP  zod=4

What IS established, and is reproducible — I have now booted against agent_v31
four times and it has never come up:

  - eve 0.31.3 fails to boot against that specific database, with
    "Invalid input: expected undefined, received Date" at path completedAt and
    "expected undefined, received Uint8Array".
  - It is not merely "a database with runs".

What I have RULED OUT by experiment rather than argument:

  - Having runs at all: two other populated databases boot.
  - Having completed runs: agent_v3 has 8 and boots.
  - Having a cancelled run: I copied evestack_agent_v2, marked one untagged
    running row cancelled with a completed_at, and it still booted in 4s. The
    cancelled-run hypothesis was mine and the experiment refuted it.
  - A store version mismatch: @workflow/world-postgres beta.31 and beta.32
    behave identically, and eve's peer ranges are unchanged between 0.30.8 and
    0.31.3.

What is still UNKNOWN: what it is about agent_v31. It is the only database the
full seam probe suite ran against, the only one written by 0.31.3 itself, the
only one with zero open runs and zero workflow_waits, and it has 72 stream
chunks against 1688–2140 in the others. The "received Uint8Array" half points at
binary data, so the stream chunks are the next thing to look at. I did not get
there.

Also corrected in the earlier commit: the bisect that produced the first
comparison used "eve": "^0.31.0", which resolves to 0.31.3 — so the run labelled
0.31.0 was 0.31.3. Nothing was concluded from it, and the version is pinned
exactly now.

WHAT DOES NOT CHANGE: this branch still must not merge. A reproducible failure
to boot, on data eve itself produced, is disqualifying whether or not I can name
the row that causes it. The two migrations in it remain correct and verified.
@SammyTourani SammyTourani changed the title DO NOT MERGE — eve 0.31.3 cannot restart against a database with runs DO NOT MERGE — eve 0.31.3 fails to boot against one database, cause not isolated Aug 9, 2026
SammyTourani added a commit that referenced this pull request Aug 9, 2026
…not read

CI caught the first fix being incomplete: probe 12 went red on a real race,
with the session row left `running` while carrying completed_at and output_cbor.
The guard added yesterday protected the exit and not the entry — the statement
that FORCED status='running' onto a row the engine had already settled was
itself the poisoning act, and the guarded restore then correctly declined to
touch what it had just broken.

The root cause is that status, completed_at, output_cbor and error_cbor are not
four columns. They are one value, and WorkflowRunSchema's discriminated union
(runs.ts:131-160) constrains them jointly. The old fixture wrote them in two
separate statements, each a valid half of a write that is only valid whole:

  SET completed_at = NULL     on the turn row, status untouched
  SET status = 'running'      on the session row, payload untouched

Both were measured against the installed schema; both are rejected. The second
shape is the subtle one and was missed entirely by the check added yesterday: a
terminal row with a NULL completed_at reaches the schema as `undefined`, not
null, because world-postgres maps null to undefined in compact() (util.js:8-19).
Fed null, z.coerce.date() would quietly coerce to 1970-01-01 and boot. Fed
undefined it builds an Invalid Date and throws. A helper three files away is
what makes it fatal rather than merely wrong.

So reopen() now writes the whole branch in one statement and returns the prior
values through a CTE, and release() puts them back only if the row is still the
one it left — a run the engine settled in the meantime stays settled rather than
being resurrected into the next reenqueue sweep. Verified against Postgres on a
scratch table: the guarded restore matches 1 row when the fixture still owns it
and 0 when the engine took it.

The "was it restored" assertion is gone, replaced by a note. It was the reason
CI went red for doing the right thing: the turn is EXPECTED to finish inside the
window the probe holds it open, and when it does, keeping the engine's row is
correct. What matters is not who won the race but whether the surviving row is
readable, which is now asserted directly — both poison shapes, over the whole
table, plus a check that the fixture's backdating never leaks.

Confirmed end to end against the real thing. agent_v31 held exactly one bad row,
`running` with completed_at and output_cbor. Repaired, then driven through the
actual read path world.start() uses:

  poisoned  runs.list({status:'running'})  THREW   (the reenqueue call)
  repaired  runs.list({limit:1000})        27 rows parsed, no throw

That is PR #36's blocker, and it was never eve 0.31.3.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
SammyTourani added a commit that referenced this pull request Aug 9, 2026
…ed (#40)

* The database that would not boot was poisoned by our own probe, not by eve 0.31.3

PR #36 holds the eve 0.30.8 -> 0.31.3 bump with DO NOT MERGE, because 0.31.3
never boots against one database — 4 failures out of 4 — with two zod errors:

    Invalid input: expected undefined, received Date          path: completedAt
    Invalid input: expected undefined, received Uint8Array

Both come from one schema and one row shape. `WorkflowRunSchema` (@workflow/world
runs.ts:130-160) is a discriminatedUnion on `status`, and its pending/running
branch declares `output`, `error` and `completedAt` as `z.undefined().optional()`.
Reproduced against the installed schema: a single row of

    status='running', completed_at=<Date>, output_cbor=<Uint8Array>

emits exactly those two messages, together, from one parse. A clean running row
passes. Those are the only `z.undefined()` calls in the package, so nothing else
in the stack can produce that wording — which pins the failure to
`workflow.workflow_runs` and rules out steps, waits, events and stream chunks.

Why it is fatal rather than noisy: `world.start()` calls `reenqueueActiveRuns`,
which does `runs.list({status:'running'})`, and `runs.list` parses EVERY row
before it filters. The parse sits outside the try that wraps the enqueue, so one
bad row aborts recovery, rejects `start()`, and does it again on every boot.

── who wrote the row ────────────────────────────────────────────────────────

`contract/runtime/probes/12-fleet-live-classification.probe.mjs`, the only thing
in this repository that UPDATEs that table. It reads `status`, forces 'running',
then spends several seconds calling the fleet API — and its own header says the
turn fails at the model call "in well under a second" without a provider key. The
engine settles the run inside that window, writing status, completed_at and
error_cbor in one statement. The probe's `finally` then put the stale 'running'
back with `WHERE id = $1` and no guard.

Both restores now name the value the probe itself wrote —
`WHERE id = $1 AND status = 'running' AND completed_at IS NULL` — so a row the
engine has moved on from is left alone.

`restored` came from having reached the line, after two `.catch(() => {})`. It
now comes from rowCount, and a new assertion checks the post-condition directly:
no pending/running run may carry completed_at, output_cbor or error_cbor. A probe
that had just bricked the agent's database reported "the fixture leaves nothing
behind", which is why this was silent for four days.

── the bump is not the cause ────────────────────────────────────────────────

PR #36 rules out 0.30.8 by watching it resume a DIFFERENT, clean database. eve
0.30.8 bundles the identical union (dist/src/compiled/@workflow/world/runs.d.ts),
so that control tests nothing about this failure; 0.30.8 against the same database
would fail the same way. The upgrade is being held by a fault that predates it.

── it can also happen without us ────────────────────────────────────────────

Upstream, `@workflow/world-postgres` guards run_completed, run_failed and
run_cancelled with `notInArray(status, TERMINAL_WORKFLOW_RUN_STATUSES)` and does
NOT guard run_started (verified by reading all four transitions in dist/storage.js).
Two workers racing one run can therefore leave a completed row marked 'running'
with no probe involved. So docs/troubleshooting.mdx gains the symptom, the query
that finds the row — it is invisible to the obvious one, because the poisoned row
HAS a completed_at and the dashboard defines an open turn as completed_at IS NULL,
so a database in this state reports zero open runs while refusing to boot — and
the repair, behind a back-up-first warning.

22 contracts / 499 assertions. The probe still skips cleanly with no stack and
still exits 1 under --require.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* A .env written on Windows read as empty, so attach gave the project a second database

`readEnvFiles` in attach.mjs split on "\n" and matched each line against a regex
ending in `$`. Without the `m` flag `$` anchors to end of input, and `.` does not
match CR, so against a CRLF file every line fails and the function returns an
empty Map. Measured: 0 of 2 lines parsed on CRLF, 2 of 2 on LF. A file written on
Windows, or checked out with `core.autocrlf=true`, is enough.

An empty Map is not a cosmetic loss. It is read in two places:

  attach.mjs:420  `existingUrl` is undefined, so attach takes the else branch and
                  adds its own Postgres — the case the comment three lines below
                  spells out as the thing that must not happen, because "a second
                  Postgres would split the session history in half and neither
                  half would be complete".
  attach.mjs:703  every key looks absent, so the block appended to the env file
                  re-declares keys already in it. Later wins, so the new
                  container's WORKFLOW_POSTGRES_URL silently overrides the
                  project's real one and the agent starts writing its sessions to
                  an empty database while the operator's own sits untouched.

`attach` is the command that edits a project someone already has, which is the
worst possible place for this.

The sibling readers — evestack-cli/src/project.mjs:108 and
templates/default/scripts/checks.mjs:41 — `.trim()` each line and were never
affected. This one matched the raw line. Fixed by splitting on /\r?\n/.

The regression test attaches to a project whose .env.local is CRLF and asserts
both consequences: WORKFLOW_POSTGRES_URL appears once and still points at the
project's own database, and no second Postgres service was added. Negative
control run: with the fix reverted the test fails, with it restored it passes.
82 scaffolder tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* "Most expensive sessions" was empty on every install, and the catch that hid it

/costs asks for the ten most expensive SESSIONS. `session_id` is declared
`groupable: false` in the catalog (metrics.ts:319), `compileMetricQuery` throws a
MetricQueryError for a non-groupable dimension, `ranked()`'s caller caught
everything and returned `[]`, and TopList rendered "Nothing to rank". The panel
has therefore been empty since it shipped, on every install, with the UI carrying
a sentence that made it look like an answer.

Two separate faults, both fixed.

── the dimension ────────────────────────────────────────────────────────────

`groupable: false` on session_id and run_id is right for its stated purpose:
keeping a hundred thousand distinct values off a chart axis. It is wrong for a
top-N list, which is exactly what `ranked()` builds.

Rather than making the dimension groupable — which would let it back onto an axis
everywhere — the query gains an explicit `topN: true`, and `ranked()` is the only
caller that sets it. A caller wanting a bounded ordered list says so; a caller
that forgets still cannot chart it. Verified by compiling the real query: refused
without the flag, `GROUP BY "d_session_id"` with it, and the unrelated guard
against ordering a bucketed query by a measure still fires.

── the catch ────────────────────────────────────────────────────────────────

`try { … } catch { return [] }` in overview.ts turned a malformed query into an
empty chart. A query this file built wrong is a bug in this file, and rendering it
as "Nothing to rank" is indistinguishable from a quiet month — which is how a
broken panel survived. MetricQueryError is deterministic and reachable in CI, so
both `ranked` and `stacked` now re-throw it and keep degrading only for runtime
failures, where an unreachable database should not take a page down.

516 dashboard tests, typecheck clean, 22 contracts / 499 assertions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Nothing asked the running dashboard what it was, which is why nobody noticed it was four days old

The published image sat at 0.1.0 for four days while packages/dashboard took 51
commits and +25,621 lines, and no check anywhere could tell. `evestack verify`
printed

    dashboard   answering at http://127.0.0.1:4000, database connected

against the current image and against the four-day-old one, identically. That is
the whole blind spot in one line: four verification tiers, all of them validating
the checkout, none of them asking the running artifact what it is.

The Dockerfile does set OCI labels. A label is visible to `docker inspect`, not
to the person reading a green verify and being told everything is fine.

So the artifact says. `/api/health` gains a `version`, on every branch including
the failures — a stale image that cannot reach Postgres still needs to be
identifiable. `lib/version.ts` reads it from the package.json the Dockerfile
already copies in (Dockerfile:151), by the same walk-up `lib/facts.ts` uses for
sql/facts.sql, because the working directory differs between `next dev`,
`next start` and the container and that shape already survives all three. It
matches on `name === "@evestack/dashboard"` rather than taking the first manifest
it finds, since walking up from a nested cwd reaches the monorepo root first. It
never throws: a health endpoint that 500s because it could not name itself is
worse than one answering "unknown", and "unknown" is still not "yes".

`npm run verify` then compares that to the tag the project's own compose file
pins — the file `docker compose up` actually obeys, not an env var that could
disagree with it — and says so on its own line. A warn and never a fail: running
a newer or locally built image is legitimate, and a verify that refused to go
green over it would be wrong more often than right. It just has to be said.

An image published before this change reports no version at all, and that is
reported as its own warning rather than as a match, because "cannot tell" and
"correct" must not print the same. Which is the same rule this commit exists to
enforce.

516 dashboard tests, 6 template tests, typecheck clean, 22 contracts / 499
assertions. Pin reader checked against a real generated project: reads 0.2.0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* An undercharged cap, an open redirect, and three surfaces disagreeing about "wedged"

Three findings from a seven-hunter sweep, each adversarially refuted before being
acted on, each reproduced here before being changed.

── the cap did not count cache writes ───────────────────────────────────────

`costUsd(model, input, output, cacheRead, cacheWrite)` takes five arguments and
hook.ts called it with four, so `cacheWriteTokens` fell to its `= 0` default on
every step. eve reports the number — its own ZERO_TOKEN_USAGE is
`{cacheReadTokens, cacheWriteTokens, inputTokens, outputTokens}` — and
pricing.ts has always accepted it. Only the call site dropped it.

Reads and writes are PARTS of inputTokens, not additions, so the miss is the gap
between two rates rather than a whole extra charge. Measured against the shipped
catalog, 1M input tokens of which 400k were writes:

    anthropic/claude-sonnet-5   charged $2.000, correct $2.200   (-10%)
    openai/gpt-5-mini           charged $0.250, correct $0.250   (none)

Anthropic prices a write at 1.25x input; gpt-5-mini publishes no write rate, so
pricing.ts falls back to input and the buckets cost the same. The bug therefore
undercharges on Anthropic and is a no-op on the DEFAULT provider, which is why it
survived. It still matters: `cost` is what the cap is measured against, so an
Anthropic prompt-caching workload passes its limit before anything trips.

`budget_steps` has no cache_write_tokens column, so the count is still not stored
per step — only its cost is now correct. That column needs a migration against
tables created with CREATE TABLE IF NOT EXISTS, which is more than this bug needs.

Guarded by a source-text assertion, the technique composio-identity.test.mjs
already uses here, because the hook needs Postgres and a live session to run and
a four-argument call is exactly the shape that regressed.

── the sign-in redirect could leave the origin ──────────────────────────────

`safeNextPath` rejected the `//` and `/\` prefixes. The WHATWG URL parser strips
tab, LF and CR BEFORE parsing, so the string a browser resolves is not the string
that was validated. Measured with Node's own implementation:

    given                  guard returned         browser resolved to
    "/\t/evil.example"     "/\t/evil.example"     https://evil.example
    "/\n/evil.example"     "/\n/evil.example"     https://evil.example
    "/\r/evil.example"     "/\r/evil.example"     https://evil.example

A link to /signin?next=/%09/evil.example sent the operator off-site immediately
after they typed the deployment password — the one moment a convincing clone is
worth most. Stripping those three characters first means the string validated is
the string emitted; every existing check then sees the normalised value.

── "wedged" meant two different things ──────────────────────────────────────

The alert hardcoded 15 minutes while facts.sql, lib/fleet.ts and the `wedged`
outcome all use STUCK_TURN_MS = 1 hour. So this alert counted turns the fleet
banner called healthy — and after 95ba65a wired its link to filter the sessions
list on outcome=wedged, it would report a number and then link to a shorter list.
It now takes the shared constant as a bound parameter and describes it in words
derived from the same value, so the prose cannot drift from the query.

While editing it I put backticks in a SQL comment inside a template literal and
broke the parse — the trap the comment four lines above warns about, in as many
words. Caught by typecheck immediately. Left the warning where it is.

843 tests across nine packages (5 new), 22 contracts / 499 assertions, typecheck
clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* The probe wrote half a row at a time, and half a row is a row eve cannot read

CI caught the first fix being incomplete: probe 12 went red on a real race,
with the session row left `running` while carrying completed_at and output_cbor.
The guard added yesterday protected the exit and not the entry — the statement
that FORCED status='running' onto a row the engine had already settled was
itself the poisoning act, and the guarded restore then correctly declined to
touch what it had just broken.

The root cause is that status, completed_at, output_cbor and error_cbor are not
four columns. They are one value, and WorkflowRunSchema's discriminated union
(runs.ts:131-160) constrains them jointly. The old fixture wrote them in two
separate statements, each a valid half of a write that is only valid whole:

  SET completed_at = NULL     on the turn row, status untouched
  SET status = 'running'      on the session row, payload untouched

Both were measured against the installed schema; both are rejected. The second
shape is the subtle one and was missed entirely by the check added yesterday: a
terminal row with a NULL completed_at reaches the schema as `undefined`, not
null, because world-postgres maps null to undefined in compact() (util.js:8-19).
Fed null, z.coerce.date() would quietly coerce to 1970-01-01 and boot. Fed
undefined it builds an Invalid Date and throws. A helper three files away is
what makes it fatal rather than merely wrong.

So reopen() now writes the whole branch in one statement and returns the prior
values through a CTE, and release() puts them back only if the row is still the
one it left — a run the engine settled in the meantime stays settled rather than
being resurrected into the next reenqueue sweep. Verified against Postgres on a
scratch table: the guarded restore matches 1 row when the fixture still owns it
and 0 when the engine took it.

The "was it restored" assertion is gone, replaced by a note. It was the reason
CI went red for doing the right thing: the turn is EXPECTED to finish inside the
window the probe holds it open, and when it does, keeping the engine's row is
correct. What matters is not who won the race but whether the surviving row is
readable, which is now asserted directly — both poison shapes, over the whole
table, plus a check that the fixture's backdating never leaks.

Confirmed end to end against the real thing. agent_v31 held exactly one bad row,
`running` with completed_at and output_cbor. Repaired, then driven through the
actual read path world.start() uses:

  poisoned  runs.list({status:'running'})  THREW   (the reenqueue call)
  repaired  runs.list({limit:1000})        27 rows parsed, no throw

That is PR #36's blocker, and it was never eve 0.31.3.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Nineteen agents, two adversarial rounds, and the fixes that needed fixing

The remaining audit findings, plus the release engineering that was the real
answer to "what is stopping this from being finished". Every lane was reviewed
by an agent whose job was to refute it, and five of nine did not survive the
first pass — the dominant failure being fixes that introduced NEW defects. Those
are fixed here too, and named, because the pattern is the point.

WHAT WAS ACTUALLY BROKEN

  A filter matching nothing exited green. `--only=seem` instead of `--only=seam`
  turned a whole verification tier into a no-op that reported success. Fixing it
  then introduced an unhandled EPIPE: waiting for the flush keeps the process
  alive long enough to receive the error the old fire-and-forget code outran, and
  an 'error' event with no listener is rethrown. `node contract/run.mjs
  --format=json | head` printed a stack trace.

  The same vacuity one level up: `node --test` exits 0 when its glob matches
  nothing (measured), and the new runner-regression suite was wired into CI
  through exactly such a glob while sitting untracked. It would have gone green
  running nothing, under a step name promising otherwise. Counted first now.

  The release gate was in the job that runs after publication, then moved to the
  right job but carrying `if: github.ref_type == 'tag'` — exempting manual
  dispatch, which the workflow's own header calls "the first publish, and
  re-running one that failed", and which still pushes `:<version>` and moves
  `:latest`. It gates the image, so it runs always.

  /schedules projected a DST fall-back an entire day early and flagged it
  `pinned`, which SUPPRESSES the hedge — a confident wrong answer where an unsure
  one had been. Verified against the real runner, not against a description of
  it: `30 1 * * *` in America/New_York standing on 01:30 EDT projected the 01:30
  EST repeat; the runner fires the next day. Fixing it initially broke spring
  forward in the other direction, because the resume line sat outside the branch
  it belonged to — caught by a differential harness against
  @evestack/schedules' own nextFire(), which is now the regression test.

  MCP's floor path let the accounting crowd out the answer. With enough
  shrinkable nodes the `cuts` list sank a document that fitted comfortably: 80
  arrays whose floored data measured 43,832 bytes returned 914 bytes and zero
  rows, under a notice claiming the data was too big. It was not. A caller can
  act on one real row and can do nothing with the 200th entry of a list of paths,
  so the cuts list yields and `cutsOmitted` counts what it stopped naming.

  A three-state SQL column read as a boolean, on the page users read. `priced` is
  TRUE / FALSE / NULL — sql/facts.sql:155 says so — and `!== true` folded the last
  two together, so every turn that never reached a provider was reported as
  unpriced spend. The empty parenthetical in "N turns ran a model with no catalog
  price ()" was the visible tell: a turn that called no model has no model to name.

  An ingest token interpolated unquoted into a printed shell command. Cosmetic
  while the token was always minted hex; the same change that made attach reuse
  the operator's token made `$(...)` in it executable.

  A security claim that was false in ten places, one of which wrote it into every
  generated docker-compose.yml. The dashboard does not answer 503 on every route
  including sign-in; PUBLIC_PATHS holds three paths and proxy.ts gates on method.

  The repair SQL shipped yesterday for the unbootable database DID NOT RUN —
  `status` is an enum and the statement fed it text. Both statements now dry-run
  clean against the real schema, and the find-query covers the mirror-image shape
  it never looked for.

RELEASE ENGINEERING

  A CHANGELOG reconstructed from real history for all 28 released versions, one
  tag convention, and a support statement that says Windows is untested rather
  than implying otherwise. @evestack/budget 0.2.1 so its undercharge fix can
  reach anyone; @evestack/sandbox-opensandbox 0.4.0 for a compatibility break.
  @evestack/mcp re-cut 0.2.1 -> 0.3.0: it was a patch while the change was
  README-only, and an output cap with a new environment variable is not a patch.

CLAIMS THAT WERE FALSE, INVESTIGATED AND DROPPED

  No SQL or shell injection in MCP — the package has neither. No
  `initialNetworkPolicy` was being dropped, because no such option existed. The
  memory page already showed a real total. Four tags cover 2 of 28 published
  versions, not 2 of 10.

  22 contracts, 508 assertions; 1,034 tests across nine packages; typecheck clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* The third job to trip over a gitignored dist, with the note about the first two right there

`dashboard-build` runs the dashboard's unit tests before the Next build on
purpose — a 300ms signal ahead of a two-minute compile. Its comment justified
that with "their whole import graph is node builtins plus pg (checked: no
workspace package among them)", and the new schedules test broke that
precondition: it checks the page's projection against @evestack/schedules'
own nextFire(), the function the runner actually fires on, which is the only
oracle that can say whether the two agree.

That import resolves through package exports to a gitignored dist/, so on a
cold checkout the whole suite died with ERR_MODULE_NOT_FOUND before running a
single test. It passed locally only because an earlier build had left dist/
lying around — the test was reading the machine, not the repo.

The comment eleven lines below already describes this exact trap for the
typecheck and Next-build steps, and ends "fixing only the typecheck job left
this one red for exactly one commit". This is the third.

Building the one package it needs rather than the workspace keeps the fast
signal: tsc on @evestack/schedules is seconds. The stale claim above it is
corrected rather than deleted, since it is the reason the step is ordered
this way.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* A schedule named __proto__ lost its next-fire silently, and took the prototype with it

`nextFires[schedule.name] = …` on a plain object literal, where the name is
whatever the operator typed. For every name but one that is an ordinary
assignment; for `__proto__` it sets the prototype instead of creating a
property. Measured:

  const a = {}; a["__proto__"] = { at: "x" };
  Object.hasOwn(a, "__proto__")  ->  false
  JSON.stringify(a)              ->  "{}"

So the entry does not exist, the row renders with no projection and no error,
and the object handed across the server/client boundary has had its prototype
replaced by a NextFire.

`Object.fromEntries` uses CreateDataProperty, which makes it an ordinary own
key. Preferred over `Object.create(null)` — that also fixes the key but hands
RSC a null-prototype object, trading a wrong answer for a serialisation
failure.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
SammyTourani added a commit that referenced this pull request Aug 10, 2026
…x was sitting in a stranded PR (#43)

An independent five-lens audit of the whole day found one thing genuinely broken on a
documented path, and it is not subtle.

  docs/upgrading.mdx  ->  npm install eve@latest
  npm latest          ->  0.31.3
  eve 0.30.8 docs     ->  the response "returns sessionId and the continuationToken"
  eve 0.31.3 docs     ->  bodies "do not accept or return continuation tokens"
  agent-client.ts:297 ->  throws 502 without it

So a self-hoster following our own upgrade instructions broke every new chat and every
fork — the dashboard's only route for starting a conversation. Nothing in the green path
catches it: tour.mjs checks only `sessionId`, verify.mjs probes only /api/health, and
attach.mjs prints a green tick for any eve newer than 0.30.8. Fresh scaffolds were safe
(the template pins ^0.30.8); upgraders were not.

The fix was written, reviewed and measured a day ago — and stranded in PR #36 behind a
version bump that never landed. Cherry-picked here on its own, because it is forward
compatibility rather than an upgrade: the session id is the handle, the token moved to
the `session.waiting` event where it becomes meaningful, and every caller that needs one
already resolves it from the durable stream. The dashboard now works against both lines.
The upgrade page pins 0.30.8 and explains why, instead of recommending `@latest`.

── the DST fix from this morning was half a fix ────────────────────────────────

`0 2 * * *` in America/St_Johns, standing on 2026-10-31T20:00Z: projected
2026-11-02T05:30Z against the runner's 2026-11-01T05:30Z. Twenty-five hours late, and
`pinned`, which suppresses the page's hedge.

0.3.0 stepped over the entire repeated interval. That is right for a reading inside it —
New York's `30 1 * * *` — and wrong for one on its far edge, which has not happened yet
and occurs exactly once. The rule is about the candidate, not the interval: look ahead at
the new offset, and step over the window only when what is found inside it is a reading
the clock has already spent. Both cases now fall out of one test.

Measured against the runner, 180 differential cases per zone, both transitions:

  America/St_Johns     4 wrong-and-pinned  ->  0
  Australia/Lord_Howe  4                   ->  0
  Pacific/Chatham      0                   ->  0
  Europe/Berlin        0                   ->  0
  America/New_York     3                   ->  2

Of the two left in New York, one is not ours: the runner's own `nextFire()` returns an
instant EARLIER than its `after` argument inside the repeated hour, so the page is right
and the oracle is wrong. The other is real and is written into the changelog rather than
hidden — `0 2 * * *` the day after a spring-forward, an hour late, still pinned.

A half-hour zone did not cause any of this; it only made a 30-minute error impossible to
mistake for rounding.

── everything else the audit surfaced ──────────────────────────────────────────

contract/floor.json was nine assertions behind the suite. Two CHANGELOG upgrade notes
warned in the present tense about bugs that npm has served fixes for since this morning —
anyone landing there was told the current release is broken. Counts that were true when
written and are not now: four tags (ten), verify's ten checks (eleven), /api/health
returning "only {ok, database}" (it returns `version`, the headline of 0.3.0), the CLI's
three commands (seven), the create-evestack README ordering a blocking `npm run dev`
before the compose bring-up. `eve dev` does not auto-increment a busy port — the
scaffolder always passes `--port`, and `retryOnAddressInUse` is set only when none is.

create-evestack goes to 0.9.2 for the new pin. Third bump today; the scaffolder pins a
tested tag on purpose, and a bump per image is the price. The version job is what keeps
that price visible rather than silent.

  22 contracts, 508 assertions; 1,035 tests across nine packages; typecheck clean.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@SammyTourani

Copy link
Copy Markdown
Owner Author

The blocker is resolved, and it was never eve 0.31.3. Recording it here so this PR stops reading as an open upstream problem.

What agent_v31 actually held

One row, found with the invariant query now in docs/troubleshooting.mdx:

wrun_01KZJSGF6CFDWJGVPV7B92CQZV | running | completed_at ✓ | output_cbor ✓

status='running' carrying a terminal payload. WorkflowRunSchema is a discriminated union whose pending/running branch declares all three of output, error and completedAt as undefined, and world.start()reenqueueActiveRunsruns.list({status:'running'}) parses every row before filtering, outside the try that wraps the enqueue. One such row and the deployment never boots again — deterministically, on every subsequent start.

This repository's own runtime probe wrote it. contract/runtime/probes/12-fleet-live-classification.probe.mjs forced status='running' onto a row the engine had already settled, then restored a stale value over the top. Fixed in 1b63559 and 06274c4: the fixture now writes a whole union branch in one statement and restores only if the row is still the one it left.

Why this PR's 0.30.8 control did not catch it

The control tested a clean database. 0.30.8 bundles the identical union — the shape is rejected there too. The variable was never the eve version.

Verified against the real thing

Repaired the row, then drove the exact read path world.start() uses, with a negative control both directions:

poisoned   runs.list({status:'running'})   THREW    ← the reenqueue call
repaired   runs.list({limit:1000})         27 rows parsed, no throw

agent_v31 boots.

Two things before anyone rebases this

  1. origin/eve-0.31.3 still carries the unguarded probe. Following this PR's own "rebase and re-run the seam probes" would re-poison a database. Rebase onto main first — the fix is at 06274c4.
  2. The agent-client.ts change here was the more urgent half, and it has already landed on main via Dashboard 0.3.1 — the upgrade page walked users into a 502 #43, shipped in dashboard 0.3.1. 0.31.3 stopped returning continuationToken, createSession required it, and docs/upgrading.mdx was telling self-hosters to npm install eve@latest — so following our own upgrade instructions produced a 502 on every new chat and every fork. That is fixed and published independently of the version bump.

What remains here is the eve 0.30.8 → 0.31.3 bump itself, plus the ScheduleHandlerArgs.receiveto migration. main is green at 0.30.8, so this is now a decision about when to take the upgrade rather than a blocked investigation. Retitling accordingly.

@SammyTourani SammyTourani changed the title DO NOT MERGE — eve 0.31.3 fails to boot against one database, cause not isolated eve 0.31.3 bump — unblocked: the boot failure was our own probe, not eve Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant