Skip to content

ios: bring the t3/ios-mvf UI test flake fixes to beta - #169

Merged
AidenJohnston merged 4 commits into
betafrom
t3/ci-flake-fixes-beta
Sep 27, 2026
Merged

AidenJohnston merged 4 commits into
betafrom
t3/ci-flake-fixes-beta

Conversation

@AidenJohnston

@AidenJohnston AidenJohnston commented Sep 27, 2026 •

Copy link
Copy Markdown
Collaborator

Beta carries the same full-flow harness as t3/ios-mvf, so it has the same intermittent UI test failures. This brings the three t3/ios-mvf fixes for them to beta. Beta is 298 commits ahead of t3/ios-mvf, so this is a separate branch rather than a merge. #165 explains each failure in full.

Commits

  • f45fef0, cherry-picked from 397b6d6: a solid fill on the instruction card's reply button. Beta's reply capsule is still 14% white over the camera behind the card. The audit failed it on gapRequest-followUp-AX5: Contrast failed (id 'action.skipGap' label 'I can't get there') in runs 36299913974 (codex/beta-qa-next) and 36276179058, and on the walk's "Can't get there". The solid colour (#2D333F) is what the translucent one looked like over ink, and chalk text on it is about 11:1 whatever is behind the card. This is a real contrast fix in the app, not a test change.
  • 3011831, cherry-picked from ad942d8 (ios tests: audit the walk strictly once held and in motion to report; triage of the UI test failures #165): audit the walk once the autopilot holds it. The full-flow tests audited the walk while the autopilot was still marking, ending and tilting up over a playing replay. Each audit pass caught a different crossfade, and transient issues from two passes matched on a label or on the "no element" key. The app now leaves <phase>.held in the gate folder while it waits on a gate, and the walk is audited once that file appears. README conflict resolved by keeping beta's text and adding the .held sentence to the -autopilotGate row. 9fc6019 (from ee8cdcc) corrects the comment on what the held walk shows.
  • 029050e, cherry-picked from Sam's 4461f13: the held-back gap the Can't-get-there flow asks for, and the replay's upper walk. testCantGetThereEndsWhereThePhoneIs sometimes went from the review straight to the upload, because the held-back window's gap wasn't one that flow asks for, and failed on screen.gapRequest never appeared (run 36310614834 on t3/practice-meter, and five runs on t3/ios-mvf). It cherry-picked cleanly; XcodeGen 2.46.0 regenerates the committed project unchanged. If beta would rather take it with the next merge of t3/ios-mvf, drop this commit.

Still red on beta, not touched here

  • testPracticeMeterFromReplay: "the reader never offered the sample meter's number" on the Mac mini, "action.startOver" not found in CI. This is Sam's practice-meter work, as ios: make beta green: aim legend layout and review-flow test waits #155 says.
  • onboarding-practice: Dynamic Type font sizes are partially unsupported ... 'Practice meter on' in the every-state audit. codex/beta-qa-practice-label-20260927 works on it.

Testing

On the Mac mini (Xcode 26.6, iPhone 17 on iOS 26.5, the workflow's flags):

  • swift test of HouseScanKit with HOUSESCAN_REQUIRE_UPSTREAM=1 and warnings as errors: passes. The XcodeGen drift check: no change.
  • FullFlowUITests, 3 iterations: every test passes every time (testCantGetThereEndsWhereThePhoneIs, testFullFlowFromLidarReplay, testFullFlowFromReplay, testLiveFogWalkScreenshots, testSomethingThereChecksTheWallAgain), except testPracticeMeterFromReplay, as above.
  • ScreenStatesUITests with the every-state audit: every test passes. The audit's one finding is onboarding-practice ... 'Practice meter on', listed above; gapRequest-followUp-AX5 and the other reply buttons pass with the solid fill.

🤖 Generated with Claude Code

SamGu-NRX and others added 3 commits September 27, 2026 06:04
Its capsule was 14% white, composited over the camera behind the card, and
CI's accessibility audit measured its contrast as nearly passing on the
LiDAR replay. It is now the same colour as a solid (#2D333F), so chalk text
on it is about 11:1 whatever is behind the card.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 397b6d6)
The full-flow tests audited the walk the moment it appeared, while the
autopilot was still marking features, setting the ends and tilting up over
a playing replay. Each audit pass caught a different crossfade; a copy
fading out has no identifier, so transient issues from two passes matched
on a label or on the "no element" key and failed as one lasting issue
(runs 36304552823, 36305050676, 36307476187).

The app now leaves <phase>.held in the gate folder while it waits for the
test to finish with a screen, and the walk is audited once that file
appears, on the settled "Done with this wall" state. Every state the walk
passes through is still audited frozen by ScreenStatesUITests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit ad942d8)
…Can't-get-there flow asks for

The synthetic replay saw the wall only to 6.5 ft: its walk looks 20 degrees
down, and its three tilt-up views share one position, so the 7 ft row never
got a second. The server's wall request (out_ft 6.500001) could not settle.

- make-synthetic-replay.swift appends 20 frames (k00042-k00061) after the
  existing 41, which keep their numbers and bytes: a walk along the wall 22
  degrees up, 0.5 m apart. Both fixtures regenerated; additions only. The
  project lists the new files among those kept out of the UI test bundle.
- A wall request above the walk's band plays the tilt-up run, as an overhead
  request does (GapPlan.asksAboveTheWalk); the autopilot lets the whole run
  play before it ends the walk.
- The held-back window is planned over the walk's frames, and for the
  Can't-get-there flow against the ends that flow sets (EndPolicy.walked), so
  its gap is one that flow asks for. testCantGetThereEndsWhereThePhoneIs sometimes
  never reached the gap screen.
- Tests: the first 41 frames still see 6.5 ft; all 61 see 7.5 ft over the
  request's span and settle it; the Can't-get-there held-back gap lies inside
  its ends.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 4461f13)
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Repo admins can enable using credits for code reviews in their settings.

@coderabbitai

coderabbitai Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 01c5a45b-4bba-4a93-ad45-668aced2605d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added the size:XXL 1,000+ effective changed lines. label Sep 27, 2026
@codemetrics-ai

Copy link
Copy Markdown

Note

Review skipped

Codemetrics is installed on this repository, but it hasn't been added to your team yet.

To enable AI code reviews for this repo, go to your dashboard and add house-scanning-master to your team's repositories.

@greptile-apps

greptile-apps Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 4/5

[Medium risk] Adds test fixture frames and adjusts replay logic for UI test stability.

The PR should not merge until walked-end planning matches the ends the app can actually set; the gate timeout margin also warrants attention.

Fix All in Claude CodeFindings

  1. P1 Planned ends can differ ▶
  2. P2 Held gate may expire ▶
Fix with agent prompt
### Issue 1
ios/HouseScanKit/Sources/HouseScanKit/Replay/ReplayPlanning.swift:237-239
With `-autopilotCantGetThere`, this code plans each end at the farthest position of any retained frame. The actual walk uses `WalkedEnd.choose`, which can place an end closer when that frame has limited tracking, is too far from the wall, or is more than 2 m past the kept-view evidence. In those cases, the held-back frames are chosen to close a gap the app may not request; it can ask for a different gap or none at all.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

### Issue 2
ios/HouseScanUITests/FullFlowUITests.swift:380-384
After finding `wallWalk.held`, the test still takes a screenshot, runs a multipass audit, and performs its `beforeLeaving` check. The app stops waiting 120 seconds after writing the marker, while the “Can’t get there” wall-tape check alone can wait 90 seconds. If those checks run slowly in CI, the autopilot can leave `wallWalk` before the test finishes, making the settled-screen audit unreliable. The gate timeout should cover the work performed after the marker appears.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Summary

The PR adds a solid instruction-card reply fill, holds the walk for a settled accessibility audit, and extends synthetic replays and held-back planning to exercise the “Can’t get there” and upper-wall flows.

  • The walked-end planner can disagree with the ends the app actually sets on some replay inputs.
  • The held-screen test barrier has a narrow timeout margin after the marker appears.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Load replay] --> B[Plan held-back walk frames]
  B --> C[Play walk without held-back window]
  C --> D[Set ends and play tilt-up run]
  D --> E[Write wallWalk.held]
  E --> F[UI test screenshots and audits]
  F --> G[Write wallWalk gate file]
  G --> H[Finish walk and handle gaps]
Loading

Reviews (1) · Last reviewed commit: "Replay: an upper walk that sees the wall..."

Comment on lines +237 to +239
let s = stood.map { wall.wallPoint($0.camera.position).s }
guard let low = s.min(), let high = s.max(), low < 0, high > 0 else { return nil }
return low...high

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Planned ends can differ

With -autopilotCantGetThere, this code plans each end at the farthest position of any retained frame. The actual walk uses WalkedEnd.choose, which can place an end closer when that frame has limited tracking, is too far from the wall, or is more than 2 m past the kept-view evidence. In those cases, the held-back frames are chosen to close a gap the app may not request; it can ask for a different gap or none at all.

Prompt To Fix With AI
This is a comment left during a code review.
Path: ios/HouseScanKit/Sources/HouseScanKit/Replay/ReplayPlanning.swift
Line: 237-239

Comment:
**Planned ends can differ**

With `-autopilotCantGetThere`, this code plans each end at the farthest position of any retained frame. The actual walk uses `WalkedEnd.choose`, which can place an end closer when that frame has limited tracking, is too far from the wall, or is more than 2 m past the kept-view evidence. In those cases, the held-back frames are chosen to close a gap the app may not request; it can ask for a different gap or none at all.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Claude Code Fix in Codex

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is Sam's code from 4461f13 on t3/ios-mvf; this PR only cherry-picked it, so I'm leaving the fix to him. The point stands as far as I can tell: .walked plans each end at the farthest retained frame, and WalkedEnd.choose can stop short of it. On the synthetic fixture the planned gap was asked for in all 10 of 10 Mac mini runs of testCantGetThereEndsWhereThePhoneIs, so it doesn't bite there. It would on a replay whose farthest frame has limited tracking or stands far from the wall.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes sense, and the synthetic fixture result explains why this cherry-pick is reliably green. The concern is still valid for other replays: .walked models the ends as the extrema of retained phone positions, while WalkedEnd.choose can clamp an end based on tracking, wall distance, or the 2 m evidence limit. That can make the planned gap differ from the one the app requests. Since this PR intentionally only cherry-picks the fix and the correction belongs to Sam’s 4461f13 work, I’m leaving it as a follow-up rather than treating it as a regression introduced here.

Tip: You can customize Greptile's behavior for this repo with .greptile/rules.md and .greptile/config.json.

Comment on lines +380 to +384
if Self.auditedOnceHeld.contains(phase) {
let held = gate.appending(path: "\(phase).held")
let deadline = Date().addingTimeInterval(150)
while !FileManager.default.fileExists(atPath: held.path), Date() < deadline { Thread.sleep(forTimeInterval: 0.2) }
XCTAssertTrue(FileManager.default.fileExists(atPath: held.path), "the autopilot never finished with screen.\(phase)")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Held gate may expire

After finding wallWalk.held, the test still takes a screenshot, runs a multipass audit, and performs its beforeLeaving check. The app stops waiting 120 seconds after writing the marker, while the “Can’t get there” wall-tape check alone can wait 90 seconds. If those checks run slowly in CI, the autopilot can leave wallWalk before the test finishes, making the settled-screen audit unreliable. The gate timeout should cover the work performed after the marker appears.

Prompt To Fix With AI
This is a comment left during a code review.
Path: ios/HouseScanUITests/FullFlowUITests.swift
Line: 380-384

Comment:
**Held gate may expire**

After finding `wallWalk.held`, the test still takes a screenshot, runs a multipass audit, and performs its `beforeLeaving` check. The app stops waiting 120 seconds after writing the marker, while the “Can’t get there” wall-tape check alone can wait 90 seconds. If those checks run slowly in CI, the autopilot can leave `wallWalk` before the test finishes, making the settled-screen audit unreliable. The gate timeout should cover the work performed after the marker appears.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Claude Code Fix in Codex

@AidenJohnston AidenJohnston Sep 27, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid. The test does its audit and map check only after .held appears, and the Can't-get-there map check can wait up to 90 s, so a slow runner could have used up the two-minute gate. c3aeabe raises the gate to four minutes. It's in #172 (beta) and #165 (t3/ios-mvf, as 4925f37). In practice the map check returns at once, because both ends are already set when the walk is held.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit ee8cdcc)
@AidenJohnston
AidenJohnston merged commit a1c2bcf into beta Sep 27, 2026
5 checks passed
AidenJohnston added a commit that referenced this pull request Sep 27, 2026
… a held screen

A test audits a held screen only once <phase>.held appears, and the walk's
map check after it can wait 90 s; the two-minute gate could expire first on
a slow runner and let the app leave mid-audit (Greptile on #169).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AidenJohnston added a commit that referenced this pull request Sep 27, 2026
… a held screen

A test audits a held screen only once <phase>.held appears, and the walk's
map check after it can wait 90 s; the two-minute gate could expire first on
a slow runner and let the app leave mid-audit (Greptile on #169).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit c3aeabe)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL 1,000+ effective changed lines.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants