Skip to content

fix(sync): say which check returned 13 - #911

Merged
fujibee merged 2 commits into
mainfrom
fix/identify-driver-exit-13
Aug 21, 2026
Merged

fix(sync): say which check returned 13#911
fujibee merged 2 commits into
mainfrom
fix/identify-driver-exit-13

Conversation

@fujibee

@fujibee fujibee commented Aug 21, 2026

Copy link
Copy Markdown
Owner

Closes #835.

The problem

Exit 13 is the sqlite sync driver's "a check failed". It was also the only thing a caller received:

storage sync reprocess failed for team 'agmsg' (exit 13):
  driver returned a non-zero exit without diagnostics; inspect that team's storage and its binding at .../teams/agmsg/config.json

That names no check and no line. There are 113 return 13 sites across 13 functions in scripts/drivers/storage/sqlite-sync.sh, so working out which one fired meant reading the file against the failing machine's state.

Nothing above the driver needed changing

scripts/internal/remote-sync.mjs:1973 already does the right thing:

const diagnostic = stderr.trim() ||
  "driver returned a non-zero exit without diagnostics; inspect that team's storage" + bindingNote(bindingPath);

It prefers the driver's stderr and falls back to that sentence only when stderr is empty; :1936-1942 also forwards stderr to the terminal as it arrives. So the message above was never "we cannot tell you" — it was an accurate report that the driver said nothing. After this change it means only that, which is what it was always trying to say.

I checked this before designing anything, because a diagnostic written to a channel that gets swallowed is worth nothing.

The shape

The location is derived, not written. A hand-authored reason at each of 113 sites is 113 more sentences that can drift from the condition beside them — the drift #781 removed by folding an authority-file fault and its wording into one function. FUNCNAME and BASH_LINENO cannot disagree with where the return is, because the shell computes them from it.

_sqlite_sync_why() {
  printf 'agmsg: sqlite-sync: %s failed at %s:%s\n' \
    "${FUNCNAME[1]:-?}" "${BASH_SOURCE[1]##*/}" "${BASH_LINENO[0]:-?}" >&2
}

The transform is mechanical — two shapes, zero exceptions

shape count
A || return 13|| { _sqlite_sync_why; return 13; } 76
B already inside a brace group, an if body, or a case body → prefixed in place 37
total 113

Derived, not enumerated, so the claim is checkable in one command each:

return 13 sites in the file      113
_sqlite_sync_why calls           113
sites with NO helper before them    0

Reviewing one instance of A and one of B reviews the whole diff. If any site had a third shape it would appear in that last count as non-zero.

The brace on shape A is load-bearing. cmd || _sqlite_sync_why; return 13 ends the || list at the semicolon and returns unconditionally — every check in the file would stop guarding anything. This is also why the helper only reports: a function cannot return on behalf of its caller, so each site keeps its own return. A naive sed 's/return 13/_sqlite_sync_why/' would have silently done exactly that.

Behaviour is unchanged

The exit code is still 13 on every path. What is added is a line on stderr.

Measured against main, same call, same arguments:

stderr exit
main (nothing) 13
this branch agmsg: sqlite-sync: storage_sync_reprocess failed at sqlite-sync.sh:1151 13

1151 is the _sqlite_sync_valid_binding gate, which is the check that call actually fails. The main row is the control — without it, "the diagnostic appears" would not distinguish a new line from one that was always there.

A diagnostic that has never fired is indistinguishable from one that does not work

So one test drives a real site and asserts the string arrives.

tests/test_remote_sync.bats — "resync status is read-only and absent bindings fail closed" — already existed and went red on this change: it asserted $output was empty, and bats merges stderr into $output, so it had been pinning "says nothing about why", which is the defect being fixed.

Rather than loosen it, it now runs with --separate-stderr and asserts both halves:

  • $output (stdout) stays empty — "fail closed" is about not emitting a status record
  • $stderr names the function and a line number

Neither can be lost by the other changing. --separate-stderr is new to this suite; bats has had it since 1.5.0 and CI installs bats unpinned via npm.

Mutations

# mutation result
A drop the braces on one shape-A site (|| _sqlite_sync_why; return 13) 2 tests red — "explicit reprocess imports quarantine without rewinding transport", "reprocess candidate body and trailer share one keyset page"
B strip the helper from every site in storage_sync_resync_status 1 test red — the diagnostic assertion above

A is the row that matters most: it is the failure mode unique to this transform, and it shows the existing suite catches it rather than only my own grep.

B strips the whole function on purpose. Removing the helper from a single site would have been an inert mutation — the test fails at whichever of that function's 9 gates trips first, so a one-site edit could leave the test green for a reason unrelated to the code.

Ran

Every suite that exercises the changed file, derived by grepping the test tree for storage_sync_ / sqlite-sync rather than picked by hand:

test_jsonl_remote_sync     25/25      test_sync_cipher       7/7
test_migrate_team_store    26/26      test_storage          24/24
test_remote_sync           32/32      test_storage_contract 34/34
test_sqlite_sync_jq_binary  8/8
                                      total 156, 0 failures

Not run locally: the full matrix. CI's job.

Scope, and what this does not do

  • scripts/drivers/storage/{jsonl,sqlite}.sh also return 13. They are not on the reported path and are not touched here.
  • Not Applying a pull page spawns 34 processes per message, and the quoting helper is 32 of them #908. That issue is about _sqlite_lit spawning 32 of the 34 processes per applied message — a throughput problem in the same functions, not a diagnosability one. An earlier revision of this body cited it by mistake; the commit message on 0d63275 still carries that wrong reference and cannot be rewritten without discarding a review already in flight against this head. This is the correction of record.
  • This makes the failure identifiable. It does not fix whatever the underlying failure on a large-history join turns out to be — that is the point: the next report will name a line.

Exit 13 is the sqlite sync driver's "a check failed", and it was the only thing
a caller received. 113 sites across 13 functions returned it with nothing on
stderr, so

  storage sync reprocess failed for team 'agmsg' (exit 13):
    driver returned a non-zero exit without diagnostics

named no check and no line. Working out which of the 113 had fired meant reading
the file against the failing machine's state.

Nothing above the driver needed changing. remote-sync.mjs already prefers the
driver's stderr and falls back to that sentence only when it is empty, and it
forwards stderr to the terminal as it arrives. So the message quoted above was
never "we cannot tell you" -- it was an accurate report that the driver had said
nothing. It now means only that.

The location is derived, not written. A hand-authored reason at each site is one
more sentence that can drift from the condition beside it, which is the drift
#781 removed by folding an authority-file fault and its wording into a single
function. FUNCNAME and BASH_LINENO cannot disagree with where the return is,
because the shell computes them from it.

One transform, two shapes, no exceptions:

  76   `|| return 13`  ->  `|| { _sqlite_sync_why; return 13; }`
  37   already inside a brace group, an if body, or a case body -> prefixed
  ---
  113  sites, and 113 `return 13` still in the file afterwards

The brace on the first shape is load-bearing. `cmd || _sqlite_sync_why; return
13` ends the `||` list at the semicolon and returns unconditionally -- every
check in the file would stop guarding anything. That is also why the helper only
reports: a function cannot return on behalf of its caller, so each site keeps
its own `return`.

Behaviour is unchanged. The exit code is still 13 on every path; what is added
is a line on stderr.

Measured against main, same call, same argument:

  main         (silent)                                              exit 13
  this branch  agmsg: sqlite-sync: storage_sync_reprocess failed
               at sqlite-sync.sh:1151                                exit 13

1151 is the `_sqlite_sync_valid_binding` gate, which is the check that call
actually fails.

Refs #908
A diagnostic that has never been emitted is indistinguishable from one that does not work. 113 sites were added; this drives one of them for real and asserts the string arrives.

The test already existed and already went red on this change: it asserted $output was empty for an absent binding, and bats merges stderr into $output, so it had been pinning 'says nothing about why' -- the exact defect being fixed. Rather than loosen it, it now runs with --separate-stderr and asserts BOTH halves: stdout stays empty, because 'fail closed' is about not emitting a status record, AND stderr names the function and line. Neither can be lost by the other changing.

--separate-stderr is new to this suite; bats has had it since 1.5.0 and CI installs bats unpinned via npm.
@fujibee
fujibee merged commit ac30d1a into main Aug 21, 2026
42 of 43 checks passed
joelmitz added a commit to joelmitz/agmsg that referenced this pull request Aug 23, 2026
`_sqlite_sync_apply_fail` closed fd 3 with `exec 3<&- 2>/dev/null || true`.
`exec` with no command word runs nothing: it applies its redirections to the
shell itself, and they outlive the statement. Every later write to stderr in
that shell therefore went to /dev/null.

The first thing written after it is `_sqlite_sync_why` -- the line fujibee#911 added so
a 13 says which check produced it. On a malformed pull page the operator saw

    agmsg: sqlite-sync: jq: error (at <stdin>:1): record 1: not one JSON value on its line
    agmsg: sqlite-sync: record 1 ended mid-frame at field type

and nothing else. The two messages the helper emits before the close still
arrived, so the output looked almost right; only the line naming the check was
missing. With the brace group the same input also prints

    agmsg: sqlite-sync: storage_sync_apply_pull failed at sqlite-sync.sh:1062

This is the single cleanup path for every failure after the stream opens, so it
covered every apply failure on the pull path, not one unlucky site.

The regression test drives a malformed page through `storage_sync_apply_pull`
and asserts both the 13 and the presence of the "failed at" line. It fails on
the previous line and passes on this one.

Found while mirroring this function on the read-state path, and only because a
reviewer insisted stderr be part of an equivalence check.
fujibee added a commit that referenced this pull request Aug 25, 2026
…derr

`_sqlite_sync_apply_fail` closed fd 3 as `exec 3<&- 2>/dev/null`. With no
command word, bash does not scope that redirection to anything — it applies it
to the shell, permanently. Every `>&2` after the first apply failure went to
/dev/null.

It sits on the failure path, so what it silences is the output a failing apply
is about to produce. The first thing lost is the message #911 added this
morning, naming which check returned 13 — the diagnostic that made today's work
possible, disabled a few hundred lines away in the same file.

`{ exec 3<&-; } 2>/dev/null` scopes it to the block.

Two tests, because either alone passes while the other's failure ships: one
lifts the driver's own function body out of the file by line range and calls it
(a copy of the shape would keep passing while the driver regressed), one greps
the file for a bare `exec` carrying a redirection. Both go red on the old form.

Reported by @joelmitz, who also identified #911 as the first casualty.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

storage_sync_prepare_push returns 13 with nothing on stdout or stderr

1 participant