Skip to content

flake: classify-paths large-input case fails with want == got #559

Description

@EricAndrechek

What happened

make ci failed once on the classify-paths test step:

✗ classify-paths test
      ok   grep-failure       aborts instead of answering
      FAIL large-input        want exit 0 + code=false/docs=true, got exit 0: code=false docs=true
    1 case(s) failed
make[1]: *** [Makefile:438: test-classify-paths] Error 1

Note the reported want and got are identical. The exit code was 0 and
the classifier's output was exactly what the case wants.

Why the message can't be trusted to explain it

scripts/classify-paths.test.sh (the large-input case) asserts three things
in one && chain:

if [ "$big_rc" -eq 0 ] && printf '%s' "$big_out" | grep -qx 'docs=true' && printf '%s' "$big_out" | grep -qx 'code=false'; then

but the failure branch reports only $big_rc and $big_out. So when the chain
fails for any reason other than those two values — either grep pipeline
returning non-zero under set -o pipefail, for instance — the message prints
a want/got pair that looks like a match and names nothing that differs.
That is exactly what happened here, and it is why this report cannot say what
the underlying cause was.

Reproduction attempts — none succeeded

  • 40 serial runs of scripts/classify-paths.test.sh: 0 failures.
  • 42 runs at 14-way parallelism (matching make verify's -j 14): 0 failures.
  • A pipefail + SIGPIPE interaction was the obvious first hypothesis, since
    that is the bug fix(ci): make classify-paths fail loudly when grep fails #549 fixed inside classify-paths.sh itself. It does not
    apply here: grep -qx 'docs=true' matches only the last line, so grep must
    consume all input and never exits early. Verified directly at 200 KB and at
    the real ~20-byte payload — both pipelines return 0.

So the mechanism is unknown, and the flake is rare.

Ask

Make the assertion self-diagnosing, so the next occurrence explains itself
rather than costing another investigation:

  1. Split the three conditions and report which one failed.
  2. Print $big_out unambiguously (delimited or through od -c), not through
    tr '\n' ' ' — the current transform makes a single-line output and a
    correct two-line output print identically, which is one of the few
    explanations the evidence cannot rule out.
  3. Consider grep -qx … <<<"$big_out" instead of the printf | grep pipeline,
    which removes the pipeline exit status from the assertion entirely. Cheap,
    and it eliminates a whole class of cause whether or not it was this one.

Provenance

Introduced with the classify-paths behavioral test in #549. Observed while
gating stack/3-content-type (#552); a re-run passed. Filing rather than
guessing at a fix, since the cause is not established.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions