Skip to content

fix(tools): a failed input read must not exit 0 - #1499

Closed
lusoris wants to merge 1 commit into
masterfrom
fix/cli-read-error-exit-code
Closed

lusoris wants to merge 1 commit into
masterfrom
fix/cli-read-error-exit-code

Conversation

@lusoris

@lusoris lusoris commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

vmaf exited 0 on every input read failure, and wrote a full report over whatever prefix had arrived. Measured on master (ef1c16071) with two y4m clips truncated mid-frame: exit 0 and a 697-byte JSON file.

Two defects, one behind the other.

1. The frame loop threw its status away. run_frame_loop() returned a single unsigned — the frame count. Every reason it could stop (both streams ending, one ending early, a reader error, vmaf_read_pictures() failing) collapsed into that number, so main() had no way to tell a completed run from a failed one.

2. Two failed reads were classified as a clean end of stream. fetch_picture() returns 1 at EOF and -1 on error, and the chain tested ret1 && ret2 before ret1 < 0 || ret2 < 0. Both -1 values satisfy the first test, so when both inputs failed the loop took the "both streams ended" branch and printed no diagnostic at all. Upstream has the same ordering and recorded it as knowingly unfixed in Netflix/vmaf#1604. (The related !ret mapping that PR fixes was already correct here, via finish_unread_picture().)

This is worse than a crash: the exit status is the entire interface for automation, and docs/usage/cli.md already promised that code 1 meant "any parse / I/O / runtime error" — a promise the binary did not keep.

What changes

Before After
Both inputs truncated exit 0, no diagnostic, report written exit 102, problem while reading pictures, no report
One input truncated exit 0, report written exit 102, no report
One input legitimately shorter exit 0, ended before warning, report unchanged
Two clean inputs exit 0, report unchanged

run_frame_loop() returns FrameLoopResult { frames, exit_code }; classification moves into classify_frame_fetch() with the error test first; a read failure exits with the new dedicated VMAF_EXIT_INPUT_READ_ERROR (102), following the ADR-0543 (100) and no-frames (101) precedent.

A stream that merely ends earlier than its partner is deliberately not an error. Scoring the common prefix of a legitimately shorter clip is a supported use, and silently turning it into a failure would break callers that rely on it. Exit 102 is for a read that failed. ADR-1262 records that line and the reasoning; folding the two together is listed there as a rejected alternative.

The two failure exits share the existing jump to the cleanup spine, so the file's goto count is unchanged at 16.

Verification

Check Result
New test_vmaf_read_error_exit.sh, 4 cases, fast suite passes
Same test against unpatched master fails: two truncated streams exited 0, expected 102
meson test --suite=fast 140 of 140
praetorctl audit passes; total infractions 1433 → 1433 (no new debt)
cppcheck, CI's flags, findings in the touched file byte-identical to master (53 findings, all in the untouched cli_parse.h)
mkdocs build --strict exit 0
Netflix golden pairs at --precision max vs a binary built from master byte-identical, all three

Golden values unchanged: 76.66783086300072 / 35.0686714193046 / 7.985899011514694. This touches only the status a run reports, never a score.

Two follow-ups this PR absorbs

The two clang-tidy warnings the ratchet caught are fixed rather than baselined, as ADR-1142 requires: enum FrameFetchOutcome now has a std::uint8_t base (performance-enum-size) and the loop returns {.frames = …, .exit_code = …} (modernize-use-designated-initializers). core/tools/vmaf.cpp measures 0 clang-tidy warnings again.

.standards-baseline.json drops 19 entries that are not this PR's doing, and that needs saying. The edit shifts lines in a file whose debt is line-keyed, so a re-record is forced; a re-record then captures the tree as it actually is, and origin/master's committed baseline turns out to be 19 findings looser than its own tree. Measured on 551d35a63 with both praetor engines (e4b35cb3c7fe and cf5338982708, which agree exactly): committed 1433, measured 1414. The entire delta is two files cleaned without tightening the baseline in the same PR:

File Baseline says Tree has Cleaned by
core/src/svm.cpp 14 0 #1498
core/test/test_ciede_neon.c 5 0 an earlier PR

A too-loose baseline fails nothing — the audit errors only on an increase or on unmatched fingerprints in a touched file — so those 19 stale entries were quietly acting as headroom for 19 new findings. Correcting them here is a side effect of a re-record this PR cannot avoid, not a change it set out to make. Filed as ledger L-80. Recorded from a clean clone, branch verified at fix/cli-read-error-exit-code.

Type

  • fix — bug fix

Checklist

  • Commits follow Conventional Commits.
  • make format is green; pre-push hooks pass.
  • Unit tests pass: 140 of 140 fast.
  • I did not modify any assertAlmostEqual(...) score in the Netflix golden Python tests.
  • docs/state.md: row T-CLI-READ-ERROR-EXIT-ZERO-2026-09-19 added as closed.

Deep-dive deliverables (ADR-0108)

  • Research digest — no digest needed: trivial. The defect, the measurement and the fix are in this description and in ADR-1262; the diagnosis was one reproducer, not an investigation.
  • Decision matrix — ADR-1262 ## Alternatives considered, four options including the rejected "also fail on a length mismatch".
  • AGENTS.md invariant note — core/tools/AGENTS.md, plus the index entry in docs/rebase-notes.md.
  • Reproducer / smoke-test command — below.
  • CHANGELOG fragment — changelog.d/fixed/cli-read-error-exit-code.md, flagged as a behaviour change.
  • Rebase note — docs/rebase-notes.md.

Reproducer

meson setup build core -Denable_cuda=false -Denable_sycl=false && ninja -C build
meson test -C build test_vmaf_read_error_exit

By hand, against any build:

python3 - <<'PY'
from pathlib import Path
W = H = 24; F = W*H + 2*(W//2)*(H//2)
hdr = b"YUV4MPEG2 W24 H24 F25:1 Ip A1:1 C420jpeg\n"
f0 = bytes((i*7+16) % 256 for i in range(F)); f1 = bytes((i*11+96) % 256 for i in range(F))
for t in ("ref", "dis"):
    Path(f"trunc_{t}.y4m").write_bytes(hdr + b"FRAME\n" + f0 + b"FRAME\n" + f1[:F//2])
PY
vmaf -r trunc_ref.y4m -d trunc_dis.y4m --feature psnr --no_prediction --json -o out.json
echo "exit=$?"   # master: 0, and out.json exists. This branch: 102, and it does not.

Rebase hazard

Upstream still has both shapes, so a sync will offer them as "theirs": the ret1 && ret2-first ordering, and the count-only return. docs/rebase-notes.md and core/tools/AGENTS.md name both; case 2 of the new test is what catches a bad resolution.

@github-actions github-actions Bot added the type:bug Something isn't working label Sep 19, 2026
@lusoris
lusoris force-pushed the fix/cli-read-error-exit-code branch from 927c33a to 0786beb Compare September 19, 2026 18:56
run_frame_loop() reported only a frame count, so every reason the loop
could stop looked identical to main(): a truncated or corrupt input was
indistinguishable from a clean end of stream, and vmaf exited 0 while
writing a full report over whatever prefix had arrived.

A second defect compounded it. fetch_picture() returns 1 at end of stream
and -1 on a read error, and the branch chain tested ret1 && ret2 before
ret1 < 0 || ret2 < 0 -- both -1 values satisfy the first test, so when
both inputs failed the loop classified it as a clean end of stream and
printed no diagnostic at all. Upstream carries the same ordering and
recorded it as knowingly unfixed in Netflix/vmaf#1604.

The loop now returns FrameLoopResult { frames, exit_code }, classification
moves to classify_frame_fetch() with the error test first, and a read
failure exits with the dedicated code VMAF_EXIT_INPUT_READ_ERROR (102)
and writes no output file. A stream that legitimately ends earlier than
its partner keeps its warning and exit 0 -- scoring the common prefix of
a shorter clip stays supported.

The two failure exits share the existing jump to the cleanup spine, so
the file's goto count is unchanged.

The three Netflix golden pairs are byte-identical before and after.
@lusoris
lusoris force-pushed the fix/cli-read-error-exit-code branch from 0786beb to 35bc51d Compare September 19, 2026 19:46
lusoris added a commit that referenced this pull request Sep 19, 2026
…aking

Three unrelated papercuts from the bug ledger, all measured.

A path glob written inside a block comment opens a nested comment, so
GCC and clang both warn under -Wcomment and the zero-warning gate fails.
Fifteen files had one: fourteen HIP parity tests and one CUDA ADM test
all said "see core/src/feature/hip/*.c", and core/src/metal/state_priv.h
said "feature/metal/*.mm" while also still naming the pre-ADR-0700
libvmaf/src/metal/ path. They now name their file sets in prose. A full
HIP-lane build goes from 53 warnings to 0.

__HIP_PLATFORM_AMD__ was #define'd at the top of eight HIP host sources.
It is a reserved identifier, so cert-dcl37-c made every PR that touched
one of those files responsible for it, and the build-side -D sat inside
the hip_runtime_dep fallback branch only -- a ROCm shipping hip-lang.pc
relied entirely on the in-file copies. Both definitions are textually
identical, which C permits silently, so nothing ever failed. hip_deps
now declares it once, outside the branch (ADR-1263).

The 10-second subprocess caps in four scripts/ci/ test modules are hang
detectors, not timing assertions. A loaded runner blew one on #1497 and
the TimeoutExpired read as a deliverables failure, costing an
investigation. All six sites share SUBPROCESS_TIMEOUT_S = 120.

.gitignore also matches .workingdir / .workingdir2 as symlinks now. The
trailing-slash form only matches a real directory, and an agent
worktree's state symlink was committed into PR #1499 that way.
@lusoris

lusoris commented Sep 19, 2026

Copy link
Copy Markdown
Contributor Author

Absorbed into the integration train #1506, folded unchanged except where that PR's description says otherwise. This PR was green on its own; it and its three siblings kept knocking each other into conflict on docs/state.md and the generated ADR indexes every time one of them merged, so they land together.

@lusoris lusoris closed this Sep 19, 2026
@lusoris
lusoris deleted the fix/cli-read-error-exit-code branch October 6, 2026 08:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type:bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant