A hardware keyboard and trackpad on the device: keys without the Keyboard button, Esc, scrolling (#67, #68)

[Saffsanity]

Why

• ESC not passed through? #67, Esc never reached the Mac. The reporter's own log (iPad Pro 13" M4, Magic Keyboard) counted 22 in.text and no in.key while he pressed Esc: the iPad never sent it. The device forwarded Esc only from pressesBegan, and iPadOS can keep a hardware Escape from there (UIKit's header: a key command needs wantsPriorityOverSystemBehavior to win over "focus or text-editing system commands"). Sideways there is no on-screen esc at all.
• Two finger scrolling on Magic Keyboard trackpad, and allowing typing into text fields without needing to hit the keyboard button #68, typing needed the Keyboard button. Keys and text reach the Mac only while the stream's input view is first responder, which only the Keyboard button (or a restore) gave; a new session, a rotation and Pair This iPad… all lost it, and with nothing first responder iPadOS's own ⌘W closed Sill's window.
• Two finger scrolling on Magic Keyboard trackpad, and allowing typing into text fields without needing to hit the keyboard button #68, two-finger trackpad scrolling did nothing. Trackpad and wheel scrolls are scroll events, which only a pan with allowedScrollTypesMask receives, and the stream's pan was fingers-only.

What

Device only: no wire change, nothing on the Mac changed. The plan, its eleven open questions with the defaults taken, and every result: docs/hardware-keyboard-plan.md.

• Esc (ESC not passed through? #67). InputOverlayView has key commands for Esc with each of the sixteen combinations of ⌃, ⌥, ⇧ and ⌘ (wantsPriorityOverSystemBehavior, no title), only while it holds the keyboard and no EscapeKey is on screen in its own window, so the tour's, the pairing overlay's and the Settings panel's Esc still win while they show. Each Sill window is its own: Settings open in one window leaves Esc to the stream in another, and an EscapeKey's Esc goes to the newest one in the window it was pressed in (EscapeKey.onScreen(in:), fire(in:)). One key press is sent once whichever channel brings it (ForwardedKeys.commanded, pressed, released: the press first, the command sends nothing; the command first, the press no later than it is swallowed with its up). The Settings panel's Done now answers Esc through EscapeKey too (it keeps .cancelAction for ⌘.). Debug builds print a line for every hardware key and the channel it came by (hwkey: usage 41 down via keyCommand (shift): to the Mac); -SillKeyProbe 1 adds GameController's own view.
• Keys without the Keyboard button (Two finger scrolling on Magic Keyboard trackpad, and allowing typing into text fields without needing to hit the keyboard button #68). HardwareKeyboardPolicy (new, pure, with its check): while a hardware keyboard is attached (GameController's coalesced keyboard) and nothing open takes the keyboard (Settings, pairing, the tour, the Apps list, the Menus pull-down), the input view holds the keyboard by itself: at a session's start, after a rotation or a reconnect, and when the last of those closes; it lets go while one is open and when the keyboard goes. That hold is never counted as the person's own: the Keyboard button stays dark, and the tour's busy and activity tests ignore it, so keyboard users still get the tour. Nothing is drawn during it (an empty input view, no shortcuts bar), so no on-screen keyboard comes up over the stream even where UIKit and GameController disagree. The Keyboard button keeps its meaning for the software keyboard: pressed over the hold it becomes the person's (iPadOS's own keyboard views come back), pressed again the software keyboard goes and keys still reach the Mac.
• Trackpad and wheel scrolling (Two finger scrolling on Magic Keyboard trackpad, and allowing typing into text fields without needing to hit the keyboard button #68). Two scroll-only pans on the input view, a continuous one (a trackpad's two fingers: a phased gesture, then Sill's own coast, as a finger's) and a discrete one (a wheel: plain deltas), sharing the finger pan's path, in every layout that shows the picture.
• Tests/keyrig (TEST ONLY, as Tests/duorig): a virtual HID keyboard and mouse inside a simulator of your own, so UIKit routes real key and scroll events. Its README has what it showed.
• Docs: CLAUDE.md, docs/DEVELOPMENT.md, the checks' README, README's How it works, the support page (a keyboard with your iPhone or iPad), the App Store description's keyboard bullet and the next version's What's New. The public words say only what a device can be expected to do from the simulator's evidence: a keyboard types on the Mac as soon as the picture shows, without the Keyboard button. Esc and scrolling with a trackpad or a mouse are built, but no device has shown them, so their words wait for Noah's steps 1, 7 and 9 (the plan's "Words that wait for a device" has each line).

The review, fixed

An adversarial review of 2fc53ed re-ran every claim. Each finding was checked here against the code first, then fixed in a commit of its own:

1. Esc and EscapeKey across Sill's windows (should-fix; bdc121d). EscapeKey.onScreen counted EscapeKeys in every window, so Settings, the tour, the pairing overlay or the Add a Mac card open in one window removed the stream's Esc key commands in every other, and fire() answered with the newest EscapeKey in any window. Now onScreen(in:) and fire(in:) take the window. Reproduced on a private iPad simulator (-SillSecondWindow 1 -SillSettings 1 -SillEscViaCommand 1, the key rig's Esc then "a"): before, Esc was handed on and the host got only the "a"; after, via keyCommand (none): to the Mac and the host got key 53 down and up.
2. Public words beyond what a device has shown (nit; af9f13c). README.md (public on merge), the support page, the App Store bullet and the next What's New promised Esc and trackpad scrolling; they now keep to typing without the Keyboard button, and the fuller lines wait in the plan. CLAUDE.md's "a pan gets no momentum events" is now a simulator observation.
3. The support page's Keyboard button (nit; af9f13c): "It brings up your device’s keyboard bar, and from there the on-screen keyboard."
4. Full Keyboard Access and Switch Control (nit; 7fdc902): with a keyboard attached the hold puts Tab, Space and the arrows on the Mac and the Keyboard button never lets go for good (toggled()), so the plan has open question 11 (default: the hold as for anyone; UIKit can tell Switch Control, not Full Keyboard Access) and step 13 for Noah.

And the two claims that did not hold: this pull request now says Towards #68 (no real trackpad's events reached the continuous pan); and run-all.sh's protocol check has a p = 0.001 statistical case ("digits uniform", StreamProtocol's pairing-code generator, not touched here) that failed once for the reviewer and passed alone. Here, on 7fdc902, run-all.sh passed all 41 at the first run.

Verified

Before the review (2fc53ed and earlier; the plan's Results):

• Tests/checks/hardware-keyboard (new): 51 checks and 20,000 random sessions (690,353 events) against six invariants; 31 of 31 mutants caught; added to CI's mutants matrix.
• Tests/checks/key-strokes: 347 checks (329 before, 18 new, Esc's key command among the random sessions' keys); 81 of 81 mutants (9 new).
• The iOS app, Debug and Release for the generic simulator and Debug for a generic device (CODE_SIGNING_ALLOWED=NO), and swift build -c release: main's warnings only.
• Harness photos, this branch against main's build on the same private simulators: 19 of 19 identical pixel for pixel (iPad 1000×710 with the drawer, Settings, the lit Keyboard button, the Menus pull-down open; 710×1000; 500×710; the Duo's book and laptop poses and cover; the iPhone's arrangement upright and sideways; the connect screen).
• Live on private simulators against SillHost --synthetic on loopback (software encoder, its input log), with keys and wheels from Tests/keyrig: Esc, typed text, ⌘W (key 13 with command; Sill's window stayed) and F5 reached the Mac with no Keyboard button; on an iPhone, after a turn sideways and back too; Esc, ⇧Esc, ⌘Esc and ⌥⌘Esc through UIKit's key-command dispatch with their flags; with Settings open Esc closed it and nothing reached the Mac, then keys flowed again; the tour showed with a keyboard attached, a key during it reached nothing, Esc skipped it; a keyboard going let the hold go and its return took it back; on a fresh iPhone the Keyboard button brought the software keyboard up and took it down as main's does; one, two and one wheel notches gave 1, 2 and 1 scrolls.
• The touch rig: 72 of 72 against main's surfaces. CI's Build and check passed on 807c8ef and 2fc53ed.

After the review (on 7fdc902; the commit after it is docs only):

• The iOS app (Xcode 27.1, 27A9275) Debug and Release for the generic simulator and Debug for a generic device, CODE_SIGNING_ALLOWED=NO, from new derived data: each succeeded (22, 38 and 18 s) with only the StreamClient.swift:3430 capture warning and appintentsmetadataprocessor's note. The Release binary holds none of the DEBUG strings (the new escape: lines included) and links GameController.
• swift build -c release from git archive of 7fdc902: complete in 43.4 s, only the CaptureProbe warning.
• Tests/checks/run-all.sh: all 41 passed at the first run (347 s; hardware-keyboard 51 checks, key-strokes 347, protocol ALL PASS). No check compiles a file the review changed (AddMacCard.swift, InputOverlay.swift, ContentView.swift), so the mutants were not run again: 31 of 31 and 81 of 81 stand.
• The touch rig, whose stub follows onScreen(in:): 72 scenarios on main's surfaces and on this branch's, compare.py 72 checks, 0 failed.
• CI's Build and check passed on 7fdc902 (run 38498602356: swift build -c release in 51 s, run-all.sh all passed with hardware-keyboard's 51 and key-strokes' 347, the iOS app on Xcode 27.1 27A9269 with only the StreamClient capture warning) and on the head 46b208c (run 38499812932).
• Live on a private iPad Pro 13-inch (M5) simulator (iOS 27.0) against loopback SillHost --synthetic: the two-window case above, before and after; a tour in each of two windows, then Settings opening in the one behind: Esc in the front window closed its own tour and left the other window's EscapeKey alone (1 in other windows, left alone); in one window, Esc closed Settings with nothing at the host, then Esc and "a" reached it; the tour showed 1.10 s after the picture with the keyboard attached, a key during it reached nothing, Esc skipped it, then keys reached the host; Esc, ⇧Esc and ⌥⌘Esc through the key command (-SillEscViaCommand 1), key 53 with each one's flags; -SillInputTest 1's Esc, ⇧Esc and scrolls (104 scroll events); the harness's Add a Mac card closed on Esc. No screen changed, so no harness photos were taken again. The simulators are deleted.

Untested, for Noah

On the iPad with a Magic Keyboard, a Debug build (the console's hwkey:, escape: and keyboard: lines), streaming a window; the plan's "Untested, for Noah" has each step:

1. Esc reaches the Mac, and which channel brought it (hwkey: … via pressesBegan or via keyCommand; with -SillKeyProbe 1, if only via GameController comes, the next fix is a GameController path).
2. Esc in iTerm2's vim, zsh's vi mode and Zed's vim mode; Control-[ in each.
3. Typing right after connecting, after a turn, after Settings, Menus, the Apps list and Pair This iPad… close, and after a reconnect, with no Keyboard button.
4. ⌘W closes the Mac's window, not Sill's.
5. No on-screen keyboard or shortcuts bar over the stream, and none popping up when the keyboard is folded away or taken off.
6. The Keyboard button with the keyboard attached: lit, iPadOS's own keyboard views; again, dark, typing still reaches the Mac.
7. Esc closes Settings, skips the tour, cancels pairing, with nothing sent to the Mac.
8. The tour on a fresh install (or -SillTourState fresh) with the keyboard attached.
9. Two-finger trackpad scrolling over the picture (any double coast), and a mouse's wheel, one notch included.
10. VoiceOver unchanged.
11. An input method's candidates (none drawn during the hold; the Keyboard button brings iPadOS's views back).
12. Two Sill windows side by side (Split View or Stage Manager), each streaming: Settings open in one, Esc in the other reaches the Mac and Settings stays open; Settings open in both, Esc closes only the window's own.
13. Full Keyboard Access, then Switch Control with a keyboard key as its switch: can Tab, Space and Esc (or the switch) still move through Sill's bar, Settings and Disconnect while the stream holds the keyboard? If not, open question 11's alternative is no automatic hold while one runs.

When steps 1, 7 and 9 pass, the words waiting in the plan go into README, the support page and the App Store text.

For the reporter (#67): Control-[ reaches the Mac as Control plus the [ key, which iTerm2, zsh's vi mode and Zed's vim mode take as Escape (untested on a device).

Towards #67 (a device decides which channel brings Esc, and the hwkey: lines say so). Towards #68 (a real trackpad's two-finger scroll, and whether iPadOS adds a second coast, need a device).

🤖 Generated with Claude Code

[addhumal]

Heads-up on the Xcode project file: this PR adds HardwareKeyboardPolicy.swift as A1000001000000000000AC01 / FC01, and #66 (Sill Viewer) adds ClientPlatform.swift under the same two IDs (with HEVCSampleFeed.swift at AC02 / FC02). Git merges both without a conflict, but whichever lands second leaves two files under one object ID, which breaks the project.

I've moved my own #93's KeyframeRecovery.swift, which used the same pair, to AC11 / FC11. As far as I can see, the IDs the other open PRs use are AA01–04, AB21–23, AC01–02, AC11, AC64, AD01, AD64, AE01 and AF01 (and their F… twins). So one of #80 and #66 needs a free pair, for example AC21 / FC21.