Repository navigation
verification: finish report bounds and integration follow-ups #125
Description
Follow-ups from PR #13 at 1a0433e, still present at 77ae847. These do not block the current convergence round.
verification/Makefile: rootmake checkomits this suite. PR build: include available experiment suites in root checks #90 integrates other components, not verification. Add the locked verification tests and lint to root dispatch without changing other teams targets. Codex 4114226413.scenes/eth3d-facade/build_scene.py:48,456-457: the overhead probe fixes 6.5 ft rather than taking the tested rule. It currently matches the public rule. Make the dependency explicit and test a changed rule without loading a dataset. Codex 4114226410. The separate finite-coverage overclaim remains a current-round repair, not deferred here.hsverify/e2e.py:381-387,668-678: both successful and HTTP-error bodies use unrestricted read(), and the total budget is checked only after the read. TreeLimit covers the server, not the verifier. Tiny fake responses confirmed both paths use an unbounded read and can return after fake time exceeds the deadline. No real exhaustion or P1 outage was reproduced. Cap actual received bytes and enforce an overall deadline; test both paths with tiny caps and a fake clock. Codex 4114226421.hsverify/replaycheck.py:246-250: with ground_truth.json present, accuracy runs despite format errors. A matched pose of length one raises ValueError in reshape rather than returning the format report. Skip invalid accuracy inputs and test main, not only check_format. Codex 4114226423.hsverify/resultcheck.py:469-490: the independent ground check uses a conservative lower bound past the battery ends, not the full circular clearance region. A straight-wall example leaves an unseen point 0.71 ft from a battery corner under a 1 ft radius without a verifier finding. This is a verifier limitation, not a reproduced server false PASS. Strengthen shape-independent necessary bounds and test without copying the server solver. Codex 4114226425.hsverify/gitref.py:26-27: fetch does not explicitly prune remote refs; absent pruning configuration, deleted branches remain resolvable as if current. Test remote branch deletion and make stale provenance explicit or prune during refresh. Codex 4114226429.hsverify/scoreboard.py:751-759: gh pr list omits --limit, so its default 30 can silently omit open PRs. Fetch all required rows and test more than 30. Codex 4114226433.
Keep dataset frames and generated scenes outside tracked files. Do not regenerate datasets or start a Simulator for these unit-level checks.
Activity
SamGu-NRX commentedon Sep 27, 2026
Further verified nonblocking findings from Codex on36ac02f, still present at64d70d1:
-4114352255, hsverify/e2e.py:493-502. The transformed response calls invariant_problems without the sent bytes, unlike the initial request at466. A reused input hash on transformed answers escapes provenance validation. Preserve variant.raw and test stale hashes on changed-input responses.
-4114352258, hsverify/a11yaudit.py:127-129,243-247. The subprocess return code is discarded; TEST FAILED counts as harness_ran. A tiny nine-screen fixture with that marker yields no audit_problems, so an unexplained failed XCTest can receive ok=true. Require successful completion or explicitly account for every expected failure. No Simulator was run for this reproduction.
-4114352263, hsverify/simrun.py:226. The regex misses line-leading warning: while matching file-prefixed diagnostics. A tiny string probe confirms the omission. Include project-level warnings and test zero-warning classification.
These concern provenance and audit reporting, not a reproduced placement safety falsePASS. Keep failed or partial evidence visible rather than treating old metrics as current proof.
SamGu-NRX commentedon Sep 27, 2026
4114352260 is a verified conditional coverage-credit defect, not a failure of the current #11 run. e2e.py:658-666 skips a bundled hostile probe for a JSON-only endpoint without recording a contract problem; report aggregation at711-727 can still mark contract_ok with a passing ordinary scene. A tiny fake probe confirmed skipped, no post call, and contract_ok=true. The current9176125 report actually exercised the400MB-expanded ZIP probe and received HTTP413 in5.6ms. Make the unexercised large-input guarantee explicit or fail that coverage claim for JSON-only endpoints. Do not run a large allocation to test it.
SamGu-NRX commentedon Sep 27, 2026
4114352256 is confirmed at64d70d1: with_ground_short_of accepts one ground entry at5ft and another without out_ft through its eligibility check, then raises KeyError on the omitted value. A tiny in-memory probe reproduced it. This aborts the verifier rather than silently certifying a result, so it is a convergence follow-up. Apply the documented omitted-ground default consistently and test mixed entries.
SamGu-NRX commentedon Sep 27, 2026
Final Codex follow-ups at verifier 371c512:
The two separately confirmed false-certification findings, 4114676165 and 4114676173, are not deferred here.
SamGu-NRX commentedon Sep 27, 2026
Added from Codex on #13 at 255bc84: 4114865931 (and Greptile 4115043215). The accessibility audit keys a screen by its static-text and button labels only, so two distinct screens with the same labels but different controls count as one, and the second is never audited.
SamGu-NRX commentedon Sep 27, 2026
Final bot-review follow-ups, checked at 255bc84 and source-checked where unchanged at b9f1601. These are separate from the currently blocking result-certification repairs.
fixtures/a11y-audit/AccessibilityAudit/AccessibilityAuditTests.swift: screen identity uses static text and button labels, so changes confined to fields, toggles or layout can skip an audit. No missed application defect was reproduced. The manager deferred this evidence-coverage limit here. Codex 4114865931 and Greptile 4115043215.hsverify/simrun.py:581-585,639-641: same-ref and same-label runs started in one second can share a report directory and replace its built-app copy before installation, which occurs outside the build lock. The runner also shares a Simulator. No concurrent wrong-build result was reproduced; current serial manual ownership avoids the race. Codex 4114865933 and CodeRabbit 4115325486.hsverify/simrun.py:629-641,656-663: an install exception bypasses discard_app and leaves the copied app. A bounded mock of the actual main function confirmed report finalization without cleanup; current source retains that control flow. CodeRabbit 4115325493.hsverify/replaycheck.py:43-49: numbers([10**400,1,1,1],4) raises OverflowError in math.isfinite instead of returning a format failure. Independently reproduced at b9f1601. This precedes the malformed-pose accuracy failure already tracked here. CodeRabbit 4115325475; Greptile 4115043212 duplicates the earlier accuracy finding.verification/Makefile: sim-app requires a result and export but does not assert the ordered capture journey. Existing manual evidence covers 12 states; the scoreboard intentionally remains partial. Add a machine-checked flow before claiming automatic journey coverage. Codex 4114865922.hsverify/resultcheck.py: less-coverage comparison ignores newly appearing starts. Coverage changes can also change candidate boundaries, so this alone did not demonstrate an unsafe placement. Test the ordering claim at matched candidate positions. Codex 4114865928.hsverify/e2e.py: bundle selection can choose old-scene.json before the canonical scene.json. No ordinary app export containing both was found. Select the canonical entry explicitly. Codex 4114865938.hsverify/e2e.py: a missing ETH3D input can be skipped while all_passed remains true. real_scene_passed remains false, and the current pinned run actually includes ETH3D. Make required real-scene evidence a separate explicit gate rather than interpreting overall success as proof it ran. Greptile 4115043208.scenes/eth3d-facade/build_scene.py:484-511: 0.25 ft probe cells are not clipped to wall length. A dataset-free production-function probe returned observed span [0,2.25] for a 2.2 ft wall and [0,2.0] for a 2.01 ft wall. No incorrect ETH3D placement decision was demonstrated. CodeRabbit 4115325497.Greptile 4115043204 is not a defect: contract_ok and expectation success answer different questions. An expectation failure still makes all_passed false and the command fail.
No Simulator, dataset regeneration, real captures or private rules were used for these review checks.