Report each failure against master and against the branch point

[cfsmp3]

Builds on #1175. That PR makes the verdict trustworthy; this one makes it readable, which is the half a reviewer actually consumes.

What a reviewer needs that the verdict cannot give

A verdict says a test does not match the approved output. It cannot say who made that true — and that is the only part a reviewer needs.

The comment tried to answer it by checking whether each test last passed in the newest master run:

if last_test_master and getattr(test['test'], platform_column) != last_test_master.id:
    common_failed_tests.append(...)   # fails on master too
else:
    extra_failed_tests.append(...)    # "Your PR breaks these cases"

Two problems. The newest master run is not where the branch was cut from, so a branch is charged for everything master did while it was open. And the classification rests on a last_passed_on_<platform> column, which the alternating baseline of #1173 keeps setting on runs that only "passed" by comparing against themselves.

The result: ccextractor PR #2322, whose whole diff was in file_buffer.h, was reported as breaking 45 tests — teletext, Options, DVB — that master fails too.

What this does

Each failure is described against two references as well as the approved output:

reference	question it answers
the approved output	does this match what we signed off? — the verdict, unchanged
the tip of master	is this broken where everyone else is working?
the closest ancestor with a run	did this branch change it?

Where both sides fail, the recorded hashes split the two cases that matter:

• failing identically — same bytes on both sides. Behaviour did not move; the baseline has gone stale. Not this branch's doing.
• failing differently — both fail, but the output changed. Something moved even though the verdict did not, and nothing in the old comment could show that.

A reference with no completed run is reported as no reference, never as agreement.

The ancestor is resolved by walking back from the PR's base (ANCESTOR_SEARCH_DEPTH = 50, one API page) to the newest completed run on the same platform. Any GitHub failure degrades to "no ancestor" — one missing comparison beats no comment.

Pass/fail is untouched

Still decided against the approved output alone; comment_pr returns SUCCESS/FAILURE from that and nothing else. A comparison explains a failure, it never excuses one — a baseline that stopped describing reality is a thing to fix, not a thing to pass.

Notes

The per-test verdict is taken from get_test_results, which already handles exit codes, absent outputs, and the alternative hashes an output may legitimately produce (multiple_files). Re-deriving it in the comparison would have created a second definition of "passed", free to drift from the first — test_the_platform_verdict_is_taken_as_given pins that.

Also removes a loop that shadowed get_info_for_pr_comment's own test parameter with each entry it iterated.

Testing

720 tests, OK (18 new). isort, pycodestyle, pydocstyle, dodgy clean; mypy back to master's single pre-existing yaml stub error.

• tests/test_ci/test_comparison.py — 14 tests over the pure logic, including the case this exists for: 45 tests failing identically to the reference must read as not this change
• ancestor resolution — nearest-first, ignores runs that never completed, survives GitHub being unavailable, and returns None without a repository
• test_comment_pr_gives_the_report_a_repository_to_walk — I built the report before opening the GitHub client at first, which silently dropped the ancestor comparison in production while every test still passed, because the fixtures have no ancestor run either way. Verified this test fails when the hand-off is reverted.
• the pre-existing "passed PR" test asserted only that the word passed appeared somewhere in the comment, which the failure copy also contained — it would have held whatever the comment said. It now asserts the verdict and the returned status.

Sequencing

Independent of #1175 textually, but they are worth landing together: while the baseline alternates, the per-test verdicts feeding these comparisons are themselves unstable, and a comparison built on unstable verdicts inherits the noise.

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud