Repository navigation
AR view draws the battery over people and objects in front of it (R-4) #41
Description
Seen in device field test run 2: TestFlight build 3.1, beta at ad91620, capture code t3/ios-mvf e9e405d, server e106acb. The exported scan was lost before it was copied off the phone, so this comes from the screen recording and the code at ad91620. Times are in the recording. b7f51ba and 49a676e on t3/ios-mvf don't change this.
This is the first time the AR view was reached on a device.
What happens. "See it on your wall" showed "The spot an installer will check / 1 ft 3 in left of your meter, 1 ft
of cable". The drawing stayed in place relative to the meter while the tester walked around for about 11 s, which is
good. But:
- at 4:14–4:16 the battery is drawn over a person standing at the wall (runbook R-4, predicted);
- at 4:17–4:24, seen from beside the end of the mock wall, it is drawn through the wooden post at the wall's end.
The tester: "see it on the wall, that's not correct, yeah, that's very not correct".
Cause.
- The overlay is a SwiftUI Canvas projection,
BatteryOverlay, drawn on top of the camera image
(ResultARScreen.swift#L15-L19). - Nothing in
ios/HouseScanusespersonSegmentationWithDepthor scene depth for occlusion. The session has
sceneDepthand the mesh when the phone supports them (LiveCapture.swift#L61-L73), so
both are already available.
Placement itself follows the meter's anchor with .gravity alignment, as intended. R-5 (hide while relocalizing) is
handled in code at the same ResultARScreen.swift lines but wasn't tried on the device.
Possible fixes (your call):
- add
.personSegmentationWithDepthon the AR screen and mask the canvas with it; - or render the box in RealityKit with scene-understanding occlusion from the mesh, which also hides it behind posts
and equipment.
Activity
Merge pull request #51 from SamGu-NRX/t3/ar-occlusion
AidenJohnston commentedon Sep 27, 2026
Build 4.1 field test (TestFlight 4.1,
beta45017ba, iPhone 14 without LiDAR, three runs on 2026-09-26): the RealityKit move draws nothing at all now; see #67.Analyses: PR #60 (run 1, run 2).
Merge remote-tracking branch 'origin/t3/ios-mvf' into t3/ios-map3d
AidenJohnston commentedon Sep 27, 2026
Decision needed: how far AR occlusion should go for the demo (#41)
Where the code is now (
t3/ios-mvf): on the live camera the result is drawn in RealityKit. Mesh occlusion is turned on for LiDAR phones, and on every A12+ phone people occlusion is turned on by re-running the session just after the battery is added (LiveCapture.setOcclusion). On build 4.1 (iPhone 14, no LiDAR) that path drew nothing (#67), so R-4 couldn't be tested.The planned fix for #67 (not in a pull request yet) has phones without LiDAR draw the result with the 2D Canvas overlay (
BatteryOverlay), which has no occlusion. LiDAR phones switch to RealityKit only once it is confirmed to draw. The fix also removes the mid-screen session re-run, because that re-run is the main suspect for #67. So the question is what to add after that:segmentationBuffer) as a mask on the Canvas. Objects still aren't occluded. This needs a per-frame mask pipeline, and the mask has to be rotated and scaled with the landscape camera image into the portrait view (only that part can be unit-tested).Also from build 4.1: the walk's gap patches and outlines (
CameraOverlays) are also 2D and were drawn through an existing battery and office furniture. Under A or B that stays as it is on phones without LiDAR; only depth would fix it.What unblocks it: a pick, and a device check (runbook R-4) on a LiDAR phone once the #67 fix is on TestFlight. That check also shows whether #67 happens on LiDAR phones at all.
🤖 Generated with Claude Code
AidenJohnston commentedon Sep 27, 2026
Team decision (2026-09-27): option A. LiDAR phones get mesh occlusion, and phones without LiDAR draw over people and objects (see #100). No further occlusion work before the deadline.
🤖 Generated with Claude Code