…ed (#40)
* The database that would not boot was poisoned by our own probe, not by eve 0.31.3
PR #36 holds the eve 0.30.8 -> 0.31.3 bump with DO NOT MERGE, because 0.31.3
never boots against one database — 4 failures out of 4 — with two zod errors:
Invalid input: expected undefined, received Date path: completedAt
Invalid input: expected undefined, received Uint8Array
Both come from one schema and one row shape. `WorkflowRunSchema` (@workflow/world
runs.ts:130-160) is a discriminatedUnion on `status`, and its pending/running
branch declares `output`, `error` and `completedAt` as `z.undefined().optional()`.
Reproduced against the installed schema: a single row of
status='running', completed_at=<Date>, output_cbor=<Uint8Array>
emits exactly those two messages, together, from one parse. A clean running row
passes. Those are the only `z.undefined()` calls in the package, so nothing else
in the stack can produce that wording — which pins the failure to
`workflow.workflow_runs` and rules out steps, waits, events and stream chunks.
Why it is fatal rather than noisy: `world.start()` calls `reenqueueActiveRuns`,
which does `runs.list({status:'running'})`, and `runs.list` parses EVERY row
before it filters. The parse sits outside the try that wraps the enqueue, so one
bad row aborts recovery, rejects `start()`, and does it again on every boot.
── who wrote the row ────────────────────────────────────────────────────────
`contract/runtime/probes/12-fleet-live-classification.probe.mjs`, the only thing
in this repository that UPDATEs that table. It reads `status`, forces 'running',
then spends several seconds calling the fleet API — and its own header says the
turn fails at the model call "in well under a second" without a provider key. The
engine settles the run inside that window, writing status, completed_at and
error_cbor in one statement. The probe's `finally` then put the stale 'running'
back with `WHERE id = $1` and no guard.
Both restores now name the value the probe itself wrote —
`WHERE id = $1 AND status = 'running' AND completed_at IS NULL` — so a row the
engine has moved on from is left alone.
`restored` came from having reached the line, after two `.catch(() => {})`. It
now comes from rowCount, and a new assertion checks the post-condition directly:
no pending/running run may carry completed_at, output_cbor or error_cbor. A probe
that had just bricked the agent's database reported "the fixture leaves nothing
behind", which is why this was silent for four days.
── the bump is not the cause ────────────────────────────────────────────────
PR #36 rules out 0.30.8 by watching it resume a DIFFERENT, clean database. eve
0.30.8 bundles the identical union (dist/src/compiled/@workflow/world/runs.d.ts),
so that control tests nothing about this failure; 0.30.8 against the same database
would fail the same way. The upgrade is being held by a fault that predates it.
── it can also happen without us ────────────────────────────────────────────
Upstream, `@workflow/world-postgres` guards run_completed, run_failed and
run_cancelled with `notInArray(status, TERMINAL_WORKFLOW_RUN_STATUSES)` and does
NOT guard run_started (verified by reading all four transitions in dist/storage.js).
Two workers racing one run can therefore leave a completed row marked 'running'
with no probe involved. So docs/troubleshooting.mdx gains the symptom, the query
that finds the row — it is invisible to the obvious one, because the poisoned row
HAS a completed_at and the dashboard defines an open turn as completed_at IS NULL,
so a database in this state reports zero open runs while refusing to boot — and
the repair, behind a back-up-first warning.
22 contracts / 499 assertions. The probe still skips cleanly with no stack and
still exits 1 under --require.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* A .env written on Windows read as empty, so attach gave the project a second database
`readEnvFiles` in attach.mjs split on "\n" and matched each line against a regex
ending in `$`. Without the `m` flag `$` anchors to end of input, and `.` does not
match CR, so against a CRLF file every line fails and the function returns an
empty Map. Measured: 0 of 2 lines parsed on CRLF, 2 of 2 on LF. A file written on
Windows, or checked out with `core.autocrlf=true`, is enough.
An empty Map is not a cosmetic loss. It is read in two places:
attach.mjs:420 `existingUrl` is undefined, so attach takes the else branch and
adds its own Postgres — the case the comment three lines below
spells out as the thing that must not happen, because "a second
Postgres would split the session history in half and neither
half would be complete".
attach.mjs:703 every key looks absent, so the block appended to the env file
re-declares keys already in it. Later wins, so the new
container's WORKFLOW_POSTGRES_URL silently overrides the
project's real one and the agent starts writing its sessions to
an empty database while the operator's own sits untouched.
`attach` is the command that edits a project someone already has, which is the
worst possible place for this.
The sibling readers — evestack-cli/src/project.mjs:108 and
templates/default/scripts/checks.mjs:41 — `.trim()` each line and were never
affected. This one matched the raw line. Fixed by splitting on /\r?\n/.
The regression test attaches to a project whose .env.local is CRLF and asserts
both consequences: WORKFLOW_POSTGRES_URL appears once and still points at the
project's own database, and no second Postgres service was added. Negative
control run: with the fix reverted the test fails, with it restored it passes.
82 scaffolder tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* "Most expensive sessions" was empty on every install, and the catch that hid it
/costs asks for the ten most expensive SESSIONS. `session_id` is declared
`groupable: false` in the catalog (metrics.ts:319), `compileMetricQuery` throws a
MetricQueryError for a non-groupable dimension, `ranked()`'s caller caught
everything and returned `[]`, and TopList rendered "Nothing to rank". The panel
has therefore been empty since it shipped, on every install, with the UI carrying
a sentence that made it look like an answer.
Two separate faults, both fixed.
── the dimension ────────────────────────────────────────────────────────────
`groupable: false` on session_id and run_id is right for its stated purpose:
keeping a hundred thousand distinct values off a chart axis. It is wrong for a
top-N list, which is exactly what `ranked()` builds.
Rather than making the dimension groupable — which would let it back onto an axis
everywhere — the query gains an explicit `topN: true`, and `ranked()` is the only
caller that sets it. A caller wanting a bounded ordered list says so; a caller
that forgets still cannot chart it. Verified by compiling the real query: refused
without the flag, `GROUP BY "d_session_id"` with it, and the unrelated guard
against ordering a bucketed query by a measure still fires.
── the catch ────────────────────────────────────────────────────────────────
`try { … } catch { return [] }` in overview.ts turned a malformed query into an
empty chart. A query this file built wrong is a bug in this file, and rendering it
as "Nothing to rank" is indistinguishable from a quiet month — which is how a
broken panel survived. MetricQueryError is deterministic and reachable in CI, so
both `ranked` and `stacked` now re-throw it and keep degrading only for runtime
failures, where an unreachable database should not take a page down.
516 dashboard tests, typecheck clean, 22 contracts / 499 assertions.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Nothing asked the running dashboard what it was, which is why nobody noticed it was four days old
The published image sat at 0.1.0 for four days while packages/dashboard took 51
commits and +25,621 lines, and no check anywhere could tell. `evestack verify`
printed
dashboard answering at http://127.0.0.1:4000, database connected
against the current image and against the four-day-old one, identically. That is
the whole blind spot in one line: four verification tiers, all of them validating
the checkout, none of them asking the running artifact what it is.
The Dockerfile does set OCI labels. A label is visible to `docker inspect`, not
to the person reading a green verify and being told everything is fine.
So the artifact says. `/api/health` gains a `version`, on every branch including
the failures — a stale image that cannot reach Postgres still needs to be
identifiable. `lib/version.ts` reads it from the package.json the Dockerfile
already copies in (Dockerfile:151), by the same walk-up `lib/facts.ts` uses for
sql/facts.sql, because the working directory differs between `next dev`,
`next start` and the container and that shape already survives all three. It
matches on `name === "@evestack/dashboard"` rather than taking the first manifest
it finds, since walking up from a nested cwd reaches the monorepo root first. It
never throws: a health endpoint that 500s because it could not name itself is
worse than one answering "unknown", and "unknown" is still not "yes".
`npm run verify` then compares that to the tag the project's own compose file
pins — the file `docker compose up` actually obeys, not an env var that could
disagree with it — and says so on its own line. A warn and never a fail: running
a newer or locally built image is legitimate, and a verify that refused to go
green over it would be wrong more often than right. It just has to be said.
An image published before this change reports no version at all, and that is
reported as its own warning rather than as a match, because "cannot tell" and
"correct" must not print the same. Which is the same rule this commit exists to
enforce.
516 dashboard tests, 6 template tests, typecheck clean, 22 contracts / 499
assertions. Pin reader checked against a real generated project: reads 0.2.0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* An undercharged cap, an open redirect, and three surfaces disagreeing about "wedged"
Three findings from a seven-hunter sweep, each adversarially refuted before being
acted on, each reproduced here before being changed.
── the cap did not count cache writes ───────────────────────────────────────
`costUsd(model, input, output, cacheRead, cacheWrite)` takes five arguments and
hook.ts called it with four, so `cacheWriteTokens` fell to its `= 0` default on
every step. eve reports the number — its own ZERO_TOKEN_USAGE is
`{cacheReadTokens, cacheWriteTokens, inputTokens, outputTokens}` — and
pricing.ts has always accepted it. Only the call site dropped it.
Reads and writes are PARTS of inputTokens, not additions, so the miss is the gap
between two rates rather than a whole extra charge. Measured against the shipped
catalog, 1M input tokens of which 400k were writes:
anthropic/claude-sonnet-5 charged $2.000, correct $2.200 (-10%)
openai/gpt-5-mini charged $0.250, correct $0.250 (none)
Anthropic prices a write at 1.25x input; gpt-5-mini publishes no write rate, so
pricing.ts falls back to input and the buckets cost the same. The bug therefore
undercharges on Anthropic and is a no-op on the DEFAULT provider, which is why it
survived. It still matters: `cost` is what the cap is measured against, so an
Anthropic prompt-caching workload passes its limit before anything trips.
`budget_steps` has no cache_write_tokens column, so the count is still not stored
per step — only its cost is now correct. That column needs a migration against
tables created with CREATE TABLE IF NOT EXISTS, which is more than this bug needs.
Guarded by a source-text assertion, the technique composio-identity.test.mjs
already uses here, because the hook needs Postgres and a live session to run and
a four-argument call is exactly the shape that regressed.
── the sign-in redirect could leave the origin ──────────────────────────────
`safeNextPath` rejected the `//` and `/\` prefixes. The WHATWG URL parser strips
tab, LF and CR BEFORE parsing, so the string a browser resolves is not the string
that was validated. Measured with Node's own implementation:
given guard returned browser resolved to
"/\t/evil.example" "/\t/evil.example" https://evil.example
"/\n/evil.example" "/\n/evil.example" https://evil.example
"/\r/evil.example" "/\r/evil.example" https://evil.example
A link to /signin?next=/%09/evil.example sent the operator off-site immediately
after they typed the deployment password — the one moment a convincing clone is
worth most. Stripping those three characters first means the string validated is
the string emitted; every existing check then sees the normalised value.
── "wedged" meant two different things ──────────────────────────────────────
The alert hardcoded 15 minutes while facts.sql, lib/fleet.ts and the `wedged`
outcome all use STUCK_TURN_MS = 1 hour. So this alert counted turns the fleet
banner called healthy — and after 95ba65a wired its link to filter the sessions
list on outcome=wedged, it would report a number and then link to a shorter list.
It now takes the shared constant as a bound parameter and describes it in words
derived from the same value, so the prose cannot drift from the query.
While editing it I put backticks in a SQL comment inside a template literal and
broke the parse — the trap the comment four lines above warns about, in as many
words. Caught by typecheck immediately. Left the warning where it is.
843 tests across nine packages (5 new), 22 contracts / 499 assertions, typecheck
clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* The probe wrote half a row at a time, and half a row is a row eve cannot read
CI caught the first fix being incomplete: probe 12 went red on a real race,
with the session row left `running` while carrying completed_at and output_cbor.
The guard added yesterday protected the exit and not the entry — the statement
that FORCED status='running' onto a row the engine had already settled was
itself the poisoning act, and the guarded restore then correctly declined to
touch what it had just broken.
The root cause is that status, completed_at, output_cbor and error_cbor are not
four columns. They are one value, and WorkflowRunSchema's discriminated union
(runs.ts:131-160) constrains them jointly. The old fixture wrote them in two
separate statements, each a valid half of a write that is only valid whole:
SET completed_at = NULL on the turn row, status untouched
SET status = 'running' on the session row, payload untouched
Both were measured against the installed schema; both are rejected. The second
shape is the subtle one and was missed entirely by the check added yesterday: a
terminal row with a NULL completed_at reaches the schema as `undefined`, not
null, because world-postgres maps null to undefined in compact() (util.js:8-19).
Fed null, z.coerce.date() would quietly coerce to 1970-01-01 and boot. Fed
undefined it builds an Invalid Date and throws. A helper three files away is
what makes it fatal rather than merely wrong.
So reopen() now writes the whole branch in one statement and returns the prior
values through a CTE, and release() puts them back only if the row is still the
one it left — a run the engine settled in the meantime stays settled rather than
being resurrected into the next reenqueue sweep. Verified against Postgres on a
scratch table: the guarded restore matches 1 row when the fixture still owns it
and 0 when the engine took it.
The "was it restored" assertion is gone, replaced by a note. It was the reason
CI went red for doing the right thing: the turn is EXPECTED to finish inside the
window the probe holds it open, and when it does, keeping the engine's row is
correct. What matters is not who won the race but whether the surviving row is
readable, which is now asserted directly — both poison shapes, over the whole
table, plus a check that the fixture's backdating never leaks.
Confirmed end to end against the real thing. agent_v31 held exactly one bad row,
`running` with completed_at and output_cbor. Repaired, then driven through the
actual read path world.start() uses:
poisoned runs.list({status:'running'}) THREW (the reenqueue call)
repaired runs.list({limit:1000}) 27 rows parsed, no throw
That is PR #36's blocker, and it was never eve 0.31.3.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Nineteen agents, two adversarial rounds, and the fixes that needed fixing
The remaining audit findings, plus the release engineering that was the real
answer to "what is stopping this from being finished". Every lane was reviewed
by an agent whose job was to refute it, and five of nine did not survive the
first pass — the dominant failure being fixes that introduced NEW defects. Those
are fixed here too, and named, because the pattern is the point.
WHAT WAS ACTUALLY BROKEN
A filter matching nothing exited green. `--only=seem` instead of `--only=seam`
turned a whole verification tier into a no-op that reported success. Fixing it
then introduced an unhandled EPIPE: waiting for the flush keeps the process
alive long enough to receive the error the old fire-and-forget code outran, and
an 'error' event with no listener is rethrown. `node contract/run.mjs
--format=json | head` printed a stack trace.
The same vacuity one level up: `node --test` exits 0 when its glob matches
nothing (measured), and the new runner-regression suite was wired into CI
through exactly such a glob while sitting untracked. It would have gone green
running nothing, under a step name promising otherwise. Counted first now.
The release gate was in the job that runs after publication, then moved to the
right job but carrying `if: github.ref_type == 'tag'` — exempting manual
dispatch, which the workflow's own header calls "the first publish, and
re-running one that failed", and which still pushes `:<version>` and moves
`:latest`. It gates the image, so it runs always.
/schedules projected a DST fall-back an entire day early and flagged it
`pinned`, which SUPPRESSES the hedge — a confident wrong answer where an unsure
one had been. Verified against the real runner, not against a description of
it: `30 1 * * *` in America/New_York standing on 01:30 EDT projected the 01:30
EST repeat; the runner fires the next day. Fixing it initially broke spring
forward in the other direction, because the resume line sat outside the branch
it belonged to — caught by a differential harness against
@evestack/schedules' own nextFire(), which is now the regression test.
MCP's floor path let the accounting crowd out the answer. With enough
shrinkable nodes the `cuts` list sank a document that fitted comfortably: 80
arrays whose floored data measured 43,832 bytes returned 914 bytes and zero
rows, under a notice claiming the data was too big. It was not. A caller can
act on one real row and can do nothing with the 200th entry of a list of paths,
so the cuts list yields and `cutsOmitted` counts what it stopped naming.
A three-state SQL column read as a boolean, on the page users read. `priced` is
TRUE / FALSE / NULL — sql/facts.sql:155 says so — and `!== true` folded the last
two together, so every turn that never reached a provider was reported as
unpriced spend. The empty parenthetical in "N turns ran a model with no catalog
price ()" was the visible tell: a turn that called no model has no model to name.
An ingest token interpolated unquoted into a printed shell command. Cosmetic
while the token was always minted hex; the same change that made attach reuse
the operator's token made `$(...)` in it executable.
A security claim that was false in ten places, one of which wrote it into every
generated docker-compose.yml. The dashboard does not answer 503 on every route
including sign-in; PUBLIC_PATHS holds three paths and proxy.ts gates on method.
The repair SQL shipped yesterday for the unbootable database DID NOT RUN —
`status` is an enum and the statement fed it text. Both statements now dry-run
clean against the real schema, and the find-query covers the mirror-image shape
it never looked for.
RELEASE ENGINEERING
A CHANGELOG reconstructed from real history for all 28 released versions, one
tag convention, and a support statement that says Windows is untested rather
than implying otherwise. @evestack/budget 0.2.1 so its undercharge fix can
reach anyone; @evestack/sandbox-opensandbox 0.4.0 for a compatibility break.
@evestack/mcp re-cut 0.2.1 -> 0.3.0: it was a patch while the change was
README-only, and an output cap with a new environment variable is not a patch.
CLAIMS THAT WERE FALSE, INVESTIGATED AND DROPPED
No SQL or shell injection in MCP — the package has neither. No
`initialNetworkPolicy` was being dropped, because no such option existed. The
memory page already showed a real total. Four tags cover 2 of 28 published
versions, not 2 of 10.
22 contracts, 508 assertions; 1,034 tests across nine packages; typecheck clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* The third job to trip over a gitignored dist, with the note about the first two right there
`dashboard-build` runs the dashboard's unit tests before the Next build on
purpose — a 300ms signal ahead of a two-minute compile. Its comment justified
that with "their whole import graph is node builtins plus pg (checked: no
workspace package among them)", and the new schedules test broke that
precondition: it checks the page's projection against @evestack/schedules'
own nextFire(), the function the runner actually fires on, which is the only
oracle that can say whether the two agree.
That import resolves through package exports to a gitignored dist/, so on a
cold checkout the whole suite died with ERR_MODULE_NOT_FOUND before running a
single test. It passed locally only because an earlier build had left dist/
lying around — the test was reading the machine, not the repo.
The comment eleven lines below already describes this exact trap for the
typecheck and Next-build steps, and ends "fixing only the typecheck job left
this one red for exactly one commit". This is the third.
Building the one package it needs rather than the workspace keeps the fast
signal: tsc on @evestack/schedules is seconds. The stale claim above it is
corrected rather than deleted, since it is the reason the step is ordered
this way.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* A schedule named __proto__ lost its next-fire silently, and took the prototype with it
`nextFires[schedule.name] = …` on a plain object literal, where the name is
whatever the operator typed. For every name but one that is an ordinary
assignment; for `__proto__` it sets the prototype instead of creating a
property. Measured:
const a = {}; a["__proto__"] = { at: "x" };
Object.hasOwn(a, "__proto__") -> false
JSON.stringify(a) -> "{}"
So the entry does not exist, the row renders with no projection and no error,
and the object handed across the server/client boundary has had its prototype
replaced by a NextFire.
`Object.fromEntries` uses CreateDataProperty, which makes it an ordinary own
key. Preferred over `Object.create(null)` — that also fixes the key but hands
RSC a null-prototype object, trading a wrong answer for a serialisation
failure.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Unblocked. The boot failure was ours, not eve's, and the fix is on
main.What the failure actually was
agent_v31refused to boot under 0.31.3, four times out of four, with two Zod errors oncompletedAtand aUint8Array. The database was not evidence about eve. Our own runtimeprobe had poisoned it.
WorkflowRunSchema's pending/running branch declaresoutput,errorandcompletedAtasz.undefined().optional(). A single row carryingstatus='running'and acompleted_atemits both of those messages together, from one parse. Probe 12 is the only thing in the
repository that UPDATEs that table: it forced
status='running', spent seconds calling thefleet API, then wrote the stale status back with no guard.
world.start()parses every rowbefore filtering and outside the try, so one bad row bricks the deployment permanently — which
is why a restart never recovered it.
eve 0.30.8 fails identically on the same row. This was never a regression.
What that retires
Three claims this PR used to rest on, all withdrawn:
databases boot fine, and that was already corrected once before the real cause was found.
was not produced by eve. It was produced by our probe, writing a shape eve does not allow.
The migrations, unchanged and still correct
ScheduleHandlerArgs.receive→to—receive(channel, {target, message, auth})becomesto(channel, target).send(message, {auth}). The template's heartbeat is the only caller.createSessionno longer demands a continuation token — 0.31.3 returns{ok, sessionId, status}and publishes the token onsession.waiting. Requiring it madeevery session start a
502 "returned no handles". Contracts pin routes and module exports,not response bodies, and this is JSON parsed at runtime — so typecheck saw nothing and only
the live seam probes caught it.
Measured against 0.31.3 with both: contracts 22/495 green, the Postgres probe tier
13 probes / 239 checks green, the agent tier green, and 5 of 6 seam probes green — the
sixth being the fleet sweep's own anti-vacuity guard reporting 0 candidates, which is the
harness, not a 0.31.3 behaviour.
Before merging
Rebase onto current
mainand re-run the seam tier. Those numbers were taken before #40, #43and #45, and a measurement worth quoting is worth retaking.