Skip to content

The Mac's sound on the device: Send Audio, kind 43, AAC-ELD (#83) - #94

Open
Code-Tomato wants to merge 60 commits into
Saffsanity:mainfrom
Code-Tomato:audio
Open

Code-Tomato wants to merge 60 commits into
Saffsanity:mainfrom
Code-Tomato:audio

Conversation

@Code-Tomato

@Code-Tomato Code-Tomato commented Oct 14, 2026 •

Copy link
Copy Markdown

Closes #83.

This finishes Noah's audio branch, which stopped at 9288aa6 partway through the device step. Main at 985a733 is merged in with one merge commit, not a rebase. Everything follows docs/audio-plan.md. Its Results sections record each gate, each departure from the plan and the review.

What it does

The device plays the sound of what it streams: the picked window's app, or every app for the Desktop. The sound stays in step with the picture and never runs ahead of it, and nothing in the picture waits for it.

  • Wire, additive. Kind 43 audio (host to device; it was 29 until Sill Viewer: a Mac controlling another Mac #66's 29–42 turned up, see 6b0911c) carries a JSON AudioFormat and binary AudioPackets. It goes only to a device whose hello lists aac-eld. Kinds 16 and 17 gain sendAudio, kind 16 gains audioNote, and ClientStats gains audioBehindMs and audioLate. SillProtocol stays 1. Older devices and older Macs see nothing new.
  • Host.
    • Send Audio (HostConfig.sendAudio) is off by default and isn't in restartNeeded, so turning it on never restarts the picture.
    • A second, audio-only SCStream follows the streamed app at 48 kHz stereo, with excludesCurrentProcessAudio. --synthetic uses a test tone instead.
    • Encoding is AAC-ELD at 128 kbps, with AudioToolbox only.
    • The sound has its own send path, never in inflight and never paced. Its bytes count in pendingBytes and takenThisSecond, so the link judge and remote pacing see it.
    • The CLI takes --audio.
  • Sill.app. Send Audio sits in the status menu and Settings › Streaming. The card says "sound" while a device gets it.
  • Device.
    • AudioPlayout, a pure model: the floor, the jitter cover, the picture's lag and the guard, dedupe, placement, drift corrected by single frames, and jumps over 40 ms.
    • AudioDecoder (AudioToolbox).
    • AudioOutput: AVAudioEngine with an AVAudioPlayerNode, session .playback + .mixWithOthers, interruptions, route changes, configuration changes, and an idle stop after 10 s.
    • A Sound button in every bar that holds it, a Sound switch in the panel's group for this device, and the panel's Send Audio row.
    • Mute is per device and remembered (Sill.soundMuted). Sound continues across every hand-over and the move home.

Defaults taken (the plan's open questions; each is one place to change)

  • Q1 Send Audio off by default on the Mac and in the CLI.
  • Q2 and Q3 AAC-ELD at 128 kbps. Q16: AudioToolbox's default quality, Medium.
  • Q4 .playback with .mixWithOthers: the device's own music is never stopped.
  • Q5 Mute is per device, local and remembered.
  • Q8 Devices may turn Send Audio on, like the other stream settings.
  • Q10 Sound goes after Desktop in every bar that still holds a whole thumbnail, with Menus counted whether or not it shows, so the bar never reflows. On a phone held upright it goes at the end of the thumbnails' row. In the Duo's book pose there is only the panel switch.
  • Q11 Headphones unplugged: mute and say so.
  • Q15 The device's engine stops after 10 s without a packet.

Verified here

Nothing ran on a device, and nothing captured or played this Mac's sound. Every host ran on loopback with the software encoder (SILL_TEST_LOOPBACK=1 SILL_TEST_SOFTWARE_ENCODER=1 SILL_TEST_NO_ROUTER=1), and every simulator run used -SillSoundSink manual -SillSoundVolume 0.

  • Builds. swift build -c release: only the CaptureProbe warning. iOS Debug and Release for the simulator and Debug for a generic device: only the known StreamClient capture warning. These were built with Xcode 27.0 against a scratch copy that stubs main's 27.1 Duo APIs; see Not verified.
  • Checks. Tests/checks/run-all.sh: all 43 pass, including the new audio-playout (218), audio-packetizer and audio-codec.
    • Mutants: all 31 checks whose sources this branch changes, 925 caught out of 925.
    • Plus the review's two new mutants (audio-playout 38 of 38).
    • ledger has mutants again (12; it had none).
  • Harness, Scripts/audio (no encoder in the picture's path).
    • H5–H8: 60 of 60 clicks within 1 ms.
    • Away at 8 and 4 Mbit/s: no packet dropped.
    • A dip recovers in the same second as the picture.
    • Through the relay: no eviction.
  • Pacing harness, Scripts/pacing --sound. real24, kf25m32, restartkf, stillend and the link verdicts pass. over8 reads 4.8 against 4.9 fps: the sound is 1.9 % of that overfilled 8 Mbit/s link, and drops are unchanged.
  • H2. Main's CLI against this branch's, with Send Audio off: identical after masking, apart from timing noise. No Audio line, no aud. key, no kind 43.
  • H10. Sill.app's previews differ from main's only in menu.txt, the Streaming and Permissions panes, and the two new sound cards.
  • Simulator.
    • S1: 273 photos. Every photo without sound is pixel for pixel main's (90 of 90).
    • S2: live against SillHost --synthetic --audio. The click finder placed clicks within 0.2 ms, with no late packets. Mute, the panel switch, background and return, and Send Audio turned off from a second client all behave as specified.
    • S3: a move hands over with one duplicate dropped, no gap and no jump.
  • Review. Three reviewers (the sync model, the host's threads and the picture's isolation, the device's session/lifecycle/UI), each followed by a verifier trying to disprove their findings. 7 findings were confirmed and fixed, one commit each (e6bc0a0 to 3760b01). 1 was refuted.

Not verified, for Noah

  • The iOS build with Xcode 27.1. This Mac only has 27.0, so this PR's CI run is the first real 27.1 build.
  • Device tests P1–P16 (the plan's §11): Music on the iPad, AirPods and AirPlay latency, away over Tailscale, interruptions, VoiceOver, battery.
  • H12, the host's CPU (open question 16): +3.9 to +4.4 points of a core with Send Audio on in the harness. The gate is not met.
  • Left open by the review:
    • Packets that arrive while the app is in the background still feed the jitter cover, so it can sit at its bound for 10 s after returning. Changing that means changing §7.5.
    • The host's first AAC-ELD encoder can block sill.audio for seconds on a loaded Mac, giving a burst of late packets at the start.
  • Decision note. CLAUDE.md listed audio as out of scope for v1. The plan records Noah's word of 2026-09-27 bringing it forward, and this PR updates BRIEF.md and CLAUDE.md to match. The public copy is left for the release that carries sound: README's Good to know, docs/app-store-metadata.md and site/privacy.html.

Saffsanity and others added 30 commits September 26, 2026 03:02
Noah's request of 2026-09-25 ("have Sill send audio back with the
video"), planned against origin/main at 8b0d418 and the remote-pacing
branch (3126821). Design only; audio stays out of v1 (BRIEF.md).

- Capture: an audio-only ScreenCaptureKit stream of its own, per app
  (the window's app, every app but Sill for the Desktop), so rotation,
  Aa and settings restarts never cut the sound. Apple (WWDC22 10155)
  filters audio per app, and a single-window filter does carry its
  app's sound. Screen Recording covers it; no new permission.
- Codec: AAC-ELD at 128 kbps, 480 or 512 frames a packet, chosen from
  an in-memory AudioToolbox probe on macOS 27 and the iOS 27 simulator
  runtime: 15.0 ms end to end against AAC-LC's 74.7 ms and Opus's
  16.5 ms, at half Opus's encode CPU.
- Wire: kind 28 (23 hello, 24/25/27 menu bar, 26 pointer), a JSON
  format then binary packets stamped on the video's clock; the hello
  lists the codecs a device plays; Send Audio in kinds 16/17.
- Pacing: never in the home delta-drop count; in the remote byte
  backlog, never dropped by it, so video drops first.
- Device: a pure playout model (a floor from the packets' own stamps,
  a jitter need, a guard one frame behind the picture, drift by single
  frames, jumps over 40 ms) and AVAudioPlayerNode playback (no Swift
  render block), .playback mixing with others, a Sound button to mute.
- Send Audio off by default; CLI --audio; stdout byte for byte without.
- Gates H0-H12, S1-S3, P1-P16; 15 open questions with defaults;
  about 8-9 days.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Checked against Apple's documentation (SCStreamConfiguration's audio
properties, SCContentFilter, WWDC22 10155 and 10156, AVAudioSession,
AVAudioPlayerNode and AVAudioNode, the engine's configuration change,
kAudioFormatOpus), main at 150f781, every branch and worktree, and an
in-memory AAC-ELD probe (a reset, a late join). Fixed in place:
- Kind 29: trackpad-gestures-plan.md reserves 28 (Tier 2). pbxproj
  A401-A403, clear of remote-pacing's A020 and home-pairing's renumbering.
- The need adds the output latency (AirPods got no sound at all at home:
  every packet late, the 150 ms cap kept it so); "late" and the held
  packet's deadline count the IO buffer, read back after activation.
- Only the Mac's segment flag starts a decoder clean: a packet decoded
  on a reset decoder mid-stream is noise (-5 dB; the fourth is exact);
  a segment's first packet is placed by T + P / rate.
- No zero fill and no flush at a gap: packets made late raised the need
  as if the link had jittered. A hand-over resets nothing.
- The budget recomputed (the capture chunk and the IO buffer were counted
  twice, the keyframe's cover was low): 60-155 ms, 20-115 ms behind the
  picture. "Never leads" bounded: the frames' stamp is taken at
  encode-out, so the guard cannot see the picture's time on the Mac.
- Remote wait behind a keyframe 1-4 s at 4 Mbps; 100 small messages cost
  about 25 KB/s on a still link; AirPlay, activation failure, the
  configuration change, a growing pool, recovery at the next select.
- Tests: silent simulator gates against the harness, the tone stamped by
  sample count, SILL_TEST_AUDIO_PAUSE, the model in audiocheck, new
  invariants and mutants; step 8 updates the README, the App Store text
  and the privacy policy.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…o audio

The plan comes in as written on 2026-09-26, from main at 8b0d418; the
next commit refreshes it against main at 643af6b.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Noah moved the sound into scope on 2026-09-27 ("Audio: the plan is done
and parked as a v2 feature by your earlier decision", first in what he
wants worked on): Q13 is answered, now, and every other open question
takes its default. Read again against main (the pointer Saffsanity#31, the menus
Saffsanity#36, remote pacing Saffsanity#34, the iPhone's portrait layout Saffsanity#30, the tour Saffsanity#35)
and the open Saffsanity#37 (pairing at home), Saffsanity#38 (gestures, kind 28) and Saffsanity#39
(away from home), and every branch and worktree:

- Kind 29 is still free; the checks that name the free kinds flip with
  it and the mutants that renumber onto 29 move to 250-254.
- The send path on main's StreamServer: the sound counts in remote
  pacing's pendingBytes (budget, slack, hold, idle mark) and is never
  dropped by it; a send of its own like the tick and kind 26; lastSentAt
  set, so TLS clients (remote now, at home with Saffsanity#37) skip their ticks.
- The sound follows the source from active's didSet, where the pointer
  and the menus already do, not from select's defer.
- The Sound button on every layout: after Desktop in the bars, Menus
  kept first where only one fits; on a phone upright at the end of the
  thumbnails' row, row 1's approved five untouched; and a Sound switch
  in the panel's group for this device. The tour needs nothing.
- pbxproj AA01-AA03 (A401-A402 went to the tour, A801 to away).
- Test rules: loopback and software-encoder hosts, no-device.sh, the
  simulator's engine in manual rendering so nothing reaches a speaker.

The new last section, "Refresh (2026-09-27)", lists every change.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The wire of docs/audio-plan.md §3, additive only (Compatibility floor):
kind 29 (host → device) carries a format (type 1, JSON AudioFormat:
codec, rate, channels, the packet size, the priming, the bitrate, the
epoch, the cookie, what it is of) and packets (type 2, binary: epoch,
seq, the segment flag, 1–255 length-prefixed packets). A reader skips
a type it does not know, and AudioMessage.parse never traps: nil for
an empty payload, an unknown type, a count of 0, a length past the
end, bytes left over, a format over 4 KB or packets over 16 KB.

The hello gains `audio`, the codecs a device plays best first, and
AudioCodec.choose is the host's pick: the first of the device's list
it makes ("aac-eld" only). A device without the list (every one before
today) is never sent kind 29. Kinds 16 and 17 gain `sendAudio`, the
state `audioNote`, ClientStats `audioBehindMs` and `audioLate`, every
one optional. StreamSettings takes sendAudio with no default, as it
takes directWireless: the host reports nil until its sound exists (the
next commits), the DEBUG mock and the ledger check nil too.

Checks: compatibility 112 (the sound's JSON, the older types reading
the newer JSON, the pick) and 25 of 25 mutants; protocol 218 (the
plan's example bytes, every malformed payload, 2,000 random and
damaged ones) and 30 of 30; the kind checks say 29 is audio and 28 and
30 unknown, and the mutants that renumber a kind move to 250-254, which
no plan will take (menus 43 of 43, pointer-control 156 and 33 of 33).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AudioPacketizer (pure, Foundation only; docs/audio-plan.md §4.3): the
chunks a source delivers become blocks of one packet's frames, 512
when the first chunk is a whole number of 512-frame packets, else 480.
Each block is stamped with the host time of the input frame its packet
decodes to first, n·F − P, from the anchor of the chunk it falls in, so
time stamp jitter never moves a sample. A chunk more than a quarter of
a chunk from where the one before ended, later or earlier, ends the
segment: what was short of a block is dropped, never flushed, padded
or filled with zeros, and the next block starts a segment (the encoder
is reset before it). A chunk of another format is another epoch.

Beside it: AudioSourceRule (the same app running kept, one that ended
started again at most every 10 s, anything else a new stream, none a
stop), PCMLayout (a buffer's layout read from its description, never
assumed, made interleaved stereo: planar, mono, 16- and 32-bit
integers) and AudioBufferTally (the first minute's line P2 records).

Tests/checks/audio-packetizer: 89 checks (the stamps ±1 µs at 480,
1024, 441 and 960 frames, a jitter oracle, the tolerance at ±0.1 ms of
a quarter chunk both ways, gaps, the rule's table, the conversions,
the tally) and 27 of 27 mutants.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AudioEncoder (AudioToolbox only; docs/audio-plan.md §4.4): interleaved
stereo Float32 at 48 kHz to AAC-ELD at 128 kbps, 480 or 512 frames a
packet, one packet per block from the first; the packet size, the
priming (240, 256), the cookie and the bitrate read back; reset() for
a new segment. AudioDecoder (iOSClient, AudioToolbox only, not yet in
the Xcode project): one packet at a time to a buffer per channel, the
decoder's reset for a segment's first packet. TestTone: the synthetic
host's 440 Hz at -30 dBFS with a 4 ms 2 kHz click at each whole second
of the wall clock, by frame index.

Tests/checks/audio-codec (H4): the tone through the packetizer, the
encoder and the decoder in memory, every decoded sample placed by its
packet's stamp: each click at its whole second, 0 frames off, at 480
and 512 frames a packet and across gaps; a segment's first packet
within -28 dB of the tone (both ends reset); a late join's first packet
noise and fourth exact; 5 s of a busy signal at 130.2 kbps; nothing but
AudioToolbox linked. 21 checks, 9 of 9 mutants. (Without its cookie the
ELD decoder reads the frame length from the format, and with it from
the cookie: removing either alone is an equivalent mutant.)

H0 again on this Mac (macOS 27.0, Xcode 27.0): ELD 480 and 512 give
priming 240 and 256, the same after AudioConverterReset, 15.0 and 25.3
ms end to end with 480-frame chunks, and a late join's first packet at
-5 to -6 dB, its fourth exact.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
docs/audio-plan.md §4–5. Send Audio (HostConfig.sendAudio, off in
`standard`, so in the CLI and a fresh Sill.app) goes through kinds 16
and 17 from either door (DeviceSettings), and is not in restartNeeded:
the sound has its own stream and never restarts the picture.

- AudioCapture: a second, audio-only SCStream per app (the Desktop:
  every app but Sill's), 48 kHz stereo, Sill's own sound excluded, its
  2×2 picture thrown away, start and stop bounded to 2 s, each buffer's
  layout read (PCMLayout), the first buffer's line and the first
  minute's (P2). Not run by any test here: it would capture sound.
- SyntheticAudio: a synthetic host's test tone, chunked by sample
  count, SILL_TEST_AUDIO_CHUNK=1024 and SILL_TEST_AUDIO_PAUSE.
- AudioPipeline: follows the source from `active`'s didSet (the same
  app keeps its stream through the picture's restarts; another source
  is a new epoch; one that ended is tried again after 10 s), one
  transition at a time; on sill.audio the packetizer, the encoder and
  the stamps (host time to the wall clock as each packet goes); the
  format before each epoch's first packet; failures set a note and
  never touch the picture.
- StreamServer: each device's codec from its hello; kind 29 only to
  those that play aac-eld, the epoch's format first, on a send path of
  its own: never inflight, the drain eviction or the keyframes'
  bookkeeping; away its bytes count in pendingBytes (paceRemote still
  drops only frames) and lastSentAt skips the TLS ticks. A cap skips a
  packet only past a second untaken at home, or the drain backstop's 15
  s away (3 s, the plan's, cut a fifth of the sound at 4 Mbit/s with
  1.5 MB keyframes: the picture itself was 5 s behind).
- StreamCoordinator: the key (an app by the staged placement's pid or
  the catalog's owner, the Desktop, a synthetic host's tone, never a
  synthetic host's real window, never Sill), the sources' filters, the
  status (HostStatus.audio) and kind 16's audioNote; "Audio stopped:
  Sill is quitting" at quit.
- The CLI's --audio (one more startup line; without it nothing new);
  Sill.app's `sendAudio` default (off), for the menu in the next commit.
- sillclient.py --audio (a hello with the codec, kind 29 counted, an
  AUDIO line) and sendAudio in --set and --expect; the encoder-free
  harness, Scripts/audio (the real StreamServer and sound files with the
  tone and fake frames, both doors on 127.0.0.1) and its device,
  Scripts/audiocheck.swift (every packet decoded, each click found at
  its stamp); the pacing harness's --audio and --sound (H6).

Verified before main's merge: the harness at home, 100.03 packets a
second, no seq gap, 60 of 60 clicks within 1 ms of their second (0.0 to
0.3 ms), the picture's net.sent, net.dropped and frame age as without
sound; 1024-frame chunks 93.84 a second; the pauses and a source change
on two devices (segments at their stamps, each epoch's format first,
nothing over 40 ms late); away through 8 Mbit/s with 1.5 MB keyframes
no packet skipped while frames were, packets no later than the frames;
the dip late together and back together. The rest of the gates run on
the merge with main.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
docs/audio-plan.md §6: the status menu's Send Audio after Resolution
(checked, "The Mac keeps playing it too", setting the value it showed),
a Send audio section in Settings › Streaming after encoding speed with
its footer and, while the sound fails, the orange line ("Sill couldn’t
capture the sound of Safari: …. The picture is unaffected."), the
Permissions pane's Screen Recording words (and their sound when Send
Audio is on), " · sound" ending the card's source row while a device
gets it, -SillSetAfter sendAudio=1, and a `sound` preview sample. The
key, its registered default (off) and its save came with the host.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Kind 28 is the gesture's, 29 the sound's, 30 and up unknown. Conflicts:
StreamMessage.swift and Compatibility.swift (both kinds, in order);
the kind checks (protocol, menus, pointer-control and compatibility say
28 gesture, 29 audio, 30 unknown; compatibility keeps both sides'
cases, its "29 not this build's" now "29 the Mac's sound"); the
mutants that renumbered a kind onto 29 or 30 stay on 250-254, and
main's "kind 28 as 29" is "kind 28 as 250"; Tests/checks/README.md's
open-branch paragraph now the sound's. StreamCoordinator (the
gestures' handler beside the sound's calls), sillclient.py, ci.yml and
MockCatalog merged by themselves.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A source that could not start said so with Swift's description of the
error, which for ScreenCaptureKit's NSErrors is a multi-line dump of
the domain, code and user info. AudioCapture's own failures (its
bounded start and stop) now give their words (AudioSourceFailure), any
other error its localized description with its domain and code (-3801
for a missing permission, -3818 for sound that failed to start), in the
log's "Audio: could not capture …" line and in the note devices show.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
docs/audio-plan.md gains its Results for steps 1 to 5: the commits,
where the build departs from the plan and why (the away cap at 15 s,
the dip judged against the picture's own recovery, the sound's own
time in "no later than the frames", both tones pausing, 48 kHz stereo
only, no real window's sound on a test host, the log lines' order,
the decoder's shape, and H12 not met, with open question 16), and every
gate's numbers.

Noah's decision (2026-09-27: "Audio: the plan is done and parked as a
v2 feature by your earlier decision") in BRIEF.md and CLAUDE.md's
decisions: the sound is in scope. CLAUDE.md: the current step, the
floor (kind 29 and its fields if the sound ships in the first public
builds), the Layout (Audio.swift, the host's sound files, AudioDecoder
outside the project for now, Scripts/audio, the two checks) and Build
and run (--audio, the harness, sendAudio among the keys).
DEVELOPMENT.md: --audio, "The Mac's sound", the sound harness.
Tests/checks/README.md: audio-packetizer and audio-codec, the counts
the sound changed, the kinds paragraph. ci.yml: both checks in the
mutants matrix.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The plan's §4.10: the host's `client …` line gains ", sound 42 ms
behind" when the device's report carries audioBehindMs of 0 or more,
and ", 3 late" for the packets too late to play. The late packets are
summed over the reports since the line before, as the worst frame age
and rtt are, so a report the 1.5 s rate limit skips still counts.
Nothing changes for a device that sends neither: every device today.

Checked with a stand-in device sending the fields: "client statsclient:
60 fps, frame age 9 ms, rtt 7 ms, sound 42 ms behind, 3 late".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Scripts/audio/run.sh's cases as their gates judge them (the segments'
pauses in both tones; the dip late together and back together), its
header within its width. docs/audio-plan.md's Results name the commits
after the five steps.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
iOSClient/AudioPlayout.swift (docs/audio-plan.md §7.2), pure: which of
the Mac's sound packets are decoded and played, and where each goes on
the player's timeline. The floor, the need and its cover (40 ms at home
and 150 away until 2 s of packets have come; home 20–150, away 60–600),
the picture's lag and the guard a frame behind it, placement by time
and packet after packet, late packets, duplicates, gaps and joins on a
reset decoder, drift by single frames, jumps over 40 ms, steps, fades
and a frame gained or lost at a packet's quietest point. Not yet in the
Xcode project (the next commit).

Where it departs from the plan, each found by the check:
- Only a packet's last 5 ms (its tail, all a fade needs) waits for the
  next packet; the body goes to the player at once. Held whole, a
  packet at the window's largest excess raced its predecessor's
  deadline with no room at all, and a link sitting there (a path a few
  ms slower, a step's first second, repeated keyframes) faded every
  packet out and in. Now the next one has 5 ms more, for no latency.
- A packet's (arrival − stamp) is taken when the playout takes it, so
  the hop between the two queues is part of what the cover covers.
- The output latency moves the need and the ear alike, so it moves no
  packet; a route change still owes a jump for the IO buffer's sake.
- "A second of packets" above the floor is an unbroken run lasting a
  second (buckets of the wall's seconds saw a step up to 2 s late), and
  a step starts the picture's window again with the floor's: frames
  from both sides of a clock step put the guard a second off.
- A jump behind drops packets until one is due within half a packet of
  what is scheduled; the rest is left to single frames.

Tests/checks/audio-playout: a simulated Mac (a drifting, stepping
clock, pauses, skipped packets, a new epoch), link (bursts, a stall,
away, a move's two connections), decoder (its state after each reset)
and player (a drifting output clock read in IO-sized renders): 205
checks with 5,000 random runs holding every invariant, and 36 of 36
mutants caught (the plan's 14 and 22 more). In CI's mutants matrix.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The device plays what kind 29 carries (docs/audio-plan.md §7.3–7.5).
Its hello lists "aac-eld", so a Mac with Send Audio on sends it the
sound.

iOSClient/AudioOutput.swift, on its own queue (sill.audio): each format
makes the epoch's decoder, and each packet goes through the playout
model, then the decoder (its priming dropped, or dropped whole after a
reset), then fades and a frame more or less, then an
AVAudioPlayerNode → the main mixer → the output. Sill writes no render
block. The session is .playback mixing with other audio, 48 kHz and
5 ms asked. It is active only while the engine runs: a format has come,
not muted, in the foreground, a packet within 10 s. Mute turns the
player down and stops 20 ms later. Interruptions, route changes
(headphones out mute and say so), a media services reset and the
engine's configuration change are handled.
The output latency the model takes is the session's, or the player's
presentation latency if more: that one already includes the output's
own latency, and adding the two counted the device twice.

In the simulator, in every build, the engine renders in manual
(offline) mode: no output device is opened, so nothing can reach a
speaker. The stand-ins are an IO buffer of 5 ms and an output latency
of 10 ms. DEBUG -SillSoundSink manual|device overrides it, and
-SillSoundVolume 0 turns the player down besides. In manual rendering a
tap on the player (before its volume) finds each click of a synthetic
host's test tone and prints where it was heard.

StreamClient (+Audio): kind 29 handed to the sound's queue as it
comes; each frame's stamp for the picture's lag; the mute (local,
remembered as Sill.soundMuted); the session's start (at home or away)
and end; the stream's rate and the rtt; the second's numbers into
ClientStats (audioBehindMs, audioLate) and the HUD's " · sound 62". A
move's hand-over resets nothing. The four new files are in the Xcode
project as AA01–AA04, a block no branch uses.

The decoder imports CoreAudio on iOS for its buffer list, which
AudioToolbox brings with it on macOS only.

Checked: iOS Debug for the simulator builds, with only the known
StreamClient warning. Live, against SillHost --synthetic --audio on
loopback and the software encoder, in a private simulator: every click
was heard 0.0–0.2 ms from where it was placed, and within 0.1 ms of its
due time once settled. The host's line reads "sound 53 ms behind".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The Mac's sound on every layout (docs/audio-plan.md §7.6, §7.7), only
while the Mac sends it (its settings on this connection say sendAudio):
- Sideways and in the halves, the Sound button comes after Desktop,
  before Settings, wherever the bar still holds a whole thumbnail with
  it. The check is MacMenuButton.fits with one more button, after Menus
  where Menus shows, so where only one of them fits, Menus keeps its
  place. The Aa ruler fades it with the buttons after Aa.
- On a phone upright (and the Duo's outer display), row 1 keeps its
  approved five. Sound ends row 2 under Settings; while the Mac's menus
  show too, Menus moves one place left, under Desktop, and the strip
  ends 8 pt before the first of them (PhonePortraitLayout's sound,
  menusBesideSound, stripBesideBoth).
- The button is a speaker over "Sound", crossed out when muted, never
  in the accent colour. VoiceOver reads "Sound from ‹Mac›", On or
  Muted, as a toggle. On the phone its symbol sits in row 1's box and
  shows large on a long press.
- The panel's This iPad group starts with "Sound from ‹Mac›", the same
  mute as a switch: the only way to it where no bar holds the button (a
  Slide Over).
- Send Audio is a row after the stream rows' closing footer, before
  Direct Wireless. It restarts nothing, it carries the Mac's note when
  sound fails, and its footers are the plan's. The ledger gains the
  field, and rule 9 keeps it from a Mac that never reported it.

The mock: -SillSound on|muted; -SillSettingsCase sound, soundoff and
soundnote; -SillSettingsScroll sound. Without them the mock Mac has no
sound, so every other photo is as it was. ContentView's contract names
them and the device's sound arguments.

Checks:
- phone-portrait: 156, with every width from 300 to 599 pt for row 2
  with Sound and Menus, and 35 of 35 mutants (four new).
- tour: 43,786. No step lights the Sound button: a phone at every
  width, and ten sizes sideways and in the halves.
- ledger: 99, with Send Audio's cases and random runs against hosts
  without sound (rule 9).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…left them

The workflow was stopped to save usage (2026-09-27); the review and finalize
steps never ran. Not reviewed; see docs/audio-plan.md for what remains.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… release) into audio

Both sides kept whole, not rebased. Where they meet:
- Kind 16 is per connection on main (Saffsanity#39): every connection's state carries
  sendAudio (streamSettings(away:)) and the Mac's audioNote beside its own
  away and link. A device away sets Send Audio through applyingAway.
- HostConfig's away pair and requirePairing beside sendAudio, which stays out
  of restartNeeded.
- StreamServer: the sound's codecs are taken from the hello on every door,
  the TLS ones (Saffsanity#37's Door) included. The sound's bytes now count in every
  client's pendingBytes and takenThisSecond, as main counts every message
  for the link judge (Saffsanity#39); it still never counts in inflight, is never
  paced and never touches the keyframes' bookkeeping.
- The sound harness (Scripts/audio) builds against main's StreamServer: the
  Door's files and LinkJudge in its host set, serve(_:route:hello:).
- Tests: away-wire's StreamSettings gains sendAudio; the kinds read 28
  gesture, 29 audio, 30 unknown; the mutants matrix is both sides' lists.
- The Duo's book pose (Saffsanity#47): at 951 pt the bar has no room for Sound beside
  the fold, so Sound shows there only in the panel's switch.
- CLAUDE.md: the sound's entry first in Current step, both sides' Layout and
  Build and run; docs/audio-plan.md gains "Merged with main at 985a733".

Verified: swift build -c release (only the CaptureProbe warning);
Tests/checks/run-all.sh, all 43; the iOS app with Xcode 27.0 compiles
everything but main's iOS 27.1 calls in DuoPostureReader.swift (a scratch
copy with those stubbed builds Debug and Release for the simulator); this
Mac has no Xcode 27.1, so CI builds the real one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The move home (PR Saffsanity#39) gives the playout home's jitter bounds and resets
nothing else (docs/audio-plan.md §7.2, rules 2 and 7): AudioOutput.setAway
was never called. And the Sound button and switch no longer vanish for up to
2 s at each hand-over while the new connection's first kind 16 is on its way:
soundAvailable keeps what the last connection said until then.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oded while silent

AVAudioEngineConfigurationChange was never observed, though docs/audio-plan.md
§7.4 asks for it: an output at another rate (AirPods, Bluetooth's call
profile, another app changing the session's rate) stops and uninitializes the
engine, while `engine` stayed set and packets kept the idle stop away, so the
sound stayed silent until a mute and unmute. Now the change stops it on
`sill.audio` (never in the handler), and the next packet builds an engine in
the output's new format and reads its latencies again.

While the sound cannot be heard (muted, in the background, the session
refused during a call) the model is muted (§7.5): each packet counts for the
floor and the cover and none is decoded. Before, only mute did that, so the
background decoded every packet into the model's waiting list, and back in
front they were placed late and reported to the Mac as late. Coming back
from any of the three resets the playback, as unmuting did.

Also: a start's `.playback` reset now discards what waited for the last
timeline (it threw the discards away, keeping their samples until the next
stop); unmuting tries a session refused during a call again (the person's
tap is the resume signal), where only the app coming back to the foreground
did; and the DEBUG click finder no longer holds its engine in a closure the
engine keeps (a leak of every stopped engine, devices' Debug builds too),
and is set only in manual rendering.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd at a session's end too

docs/audio-plan.md §7.4: the engine stops after a fade on mute, on
didEnterBackground and at the end of the session. A mute turned the player
to volume 0 and stopped the engine 20 ms later; the background and the end
of a session stopped it at once, a cut. Measured in manual rendering (the
engine's offline mode: no output device, nothing heard): the main mixer
ramps a change of the player's volume linearly over 1,200 of its frames,
25 ms at 48 kHz, whatever the render's size (240 or 1,024 frames), so the
stop at 20 ms cut the sound at a fifth of its level. The fade is that ramp
now, and the engine stops 50 ms after it starts, on mute, in the background
and at a session's end (a reconnect's start too). An interruption and a
configuration change find the engine stopped already, and the idle stop
follows rule 10's faded tail. The plan's 5 ms fade is the mixer's 25 ms:
the player node offers nothing finer without a render block of Sill's own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/audio-plan.md §7.8 has the line "audio: jump +52 ms (clock step)"; the
jumps were only counted in the once-a-second line. AudioPlayout now keeps
why a jump was owed (rule 13's step, rule 12's error over 40 ms or a sound
ahead of the picture, a route's new latency) while it counts, and the next
placement by time measures it: where that packet goes against where it
would have followed (positive, placed later with silence before it;
negative, packets dropped). `takeJump` hands it over once, and AudioOutput
prints it as the outputs are applied (DEBUG).

The audio-playout check takes every report and holds the clock steps to
them: each jump reported once; the Mac's clock a second forward, one clock
step of nothing; a second back, the second dropped, then the clock step.
M33's pattern follows `owe(.step)`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`adopt` started every remote session's sound away (docs/audio-plan.md §7.2,
rule 2: a 150 ms cover to start, 60–600 ms bounds), including one whose way
in is `.address`: a LAN address through the remote door, which the Mac's
AwayPolicy counts as home. Such a session started 110 ms later than it
needed to and kept away's floor of 60 ms. Now only a route through a VPN or
over the internet is away.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/audio-plan.md §7.6: Sound goes where the bar holds it after Menus, so
where only one fits (an iPhone SE sideways, 639 pt of row; iPad windows
under 532 pt upright) Menus keeps its place and Sound is the Settings
panel's switch alone. The bars counted Menus only while the Mac's menus
showed, so on those bars Sound showed until the menus' top level came and
then went (a reflow after each connect, and with every app without menus).
A Mac that sends its sound takes menus, so Menus is counted always, as the
book pose's bar already counts it (BarMetrics.clearing). In the book pose
the buttons past the fold hold Menus and four more: Sound is the panel's
switch alone there, whatever the menus.

The tour check's Sound case gains the iPhone Duo's sizes: its inner display
both ways and its cover sideways (the cover upright is the phone grid's).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ndom runs' line

audio-playout: the move home (PR Saffsanity#39) was a scenario that only changed the
bounds; now also as it happens, a hand-over with both connections carrying
the stream for a moment, away (80 ms) on the old one and home (5 ms) on the
new: every copy dropped as a duplicate and counted nowhere, no join, at most
one jump, nothing late, home's bounds after. And the random runs' summary
said "every invariant held" whenever any run happened; it now passes only
when none of them failed.

phone-portrait: the grid holds row 2 with Sound and Menus to two whole
thumbnails at every width from 375 pt (§11 H3; only the SE was checked).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mes H12's known failure

audiocheck exited 0 when its connection never became ready (a refused or
failing connection printed a summary of zeros), so run.sh's count of runs
that could not run missed it, and a case gated on nothing passed silently.
It exits 3 now, and run.py fails the run when a device did. run.sh's --help
printed its header but the last case's description. The header names H12's
CPU gate as the known failure it is until Noah answers open question 16
(+2.1 to +3.1 points with codec quality Medium, docs/audio-plan.md's
Results), so a full run's nonzero status is not read as a regression.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
S2's live run found every click of the test tone until the first mute, and
none after: a new engine's timeline starts at sample 0, while the finder
kept the last engine's rendered samples and their start, so it looked for
the clicks at the old timeline's places. It is made anew at each engine's
start, as at a session's. DEBUG only (manual rendering).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/audio-plan.md's "Results: the device": the device's steps (6878c11,
aefe477, 5f7b1ac, 9288aa6), the review against the plan and its fixes
(012c5de to c2a1802), where the build departs from the plan and why (rule
10's held tail, the output latency's max, packets stamped on `sill.audio`,
AA04, the harness's arguments, the mixer's 25 ms fade, a configuration
change rebuilding the engine, Sound counting Menus and the book pose's
panel switch, the simulator's offline rendering, S2 and S3 against the
synthetic host), and the gates: H1 from a scratch copy with the iOS 27.1
names stubbed, H3, H5–H8 with H12 still failing (open question 16), H6
with sound on the merge and the pacing gates against main, H2, H10, H11,
S1 (273 photos, every photo without the sound pixel for pixel main's), S2
and S3. The merge's §7.6 note says what the review changed.

CLAUDE.md: the entry's state, the device's design in brief, the results
and the harness's sound arguments; docs/DEVELOPMENT.md: the device's side
of the Mac's sound; Tests/checks/README.md: audio-playout's 214 checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ledger had no mutants since its old ones (whole copies of an older ledger) were dropped, so
nothing held its check to the rules it states, the sound's seventh field among them. Twelve one-line
mutants of HostSettingsLedger.swift and of the merge it uses in HostSettings.swift: rule 9 for Send
Audio and for Direct Wireless, the field list, the single field and the merge without Send Audio,
rules 2, 4, 5, 6 (the timeout itself) and 7, and isEmpty and applied(to:) without Send Audio. All 12
caught. run.sh --mutants runs them, CI's mutants matrix lists ledger, and the README's table says 12.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Saffsanity and others added 14 commits October 20, 2026 00:45
…side Send Audio

Main gained Saffsanity#153 (a saved Mac kept across a force quit) and Saffsanity#142 (a typed
destination kept across a key change, DNS names told from IP ranges), and
Saffsanity#142's build.sh quoting for three checks. Only Tests/checks/README.md
conflicted: the protocol row keeps this branch's kind 43 counts (221 + 8,
34 mutants) and the remote-rules row takes main's (144, 76). The audio
checks have no build.sh, so main's quoting fix has nothing to follow here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Any device, from home or away (and at home any device on the network
while Require pairing is off), could turn Send Audio on for the whole
Mac: every app's sound for the Desktop, calls included, played unmuted on
every connected device. Share Clipboard is the Mac's alone for a like
reason, and the PR's own author proposed the same for Saffsanity#66's mute-host.
For the beta, only the Mac's menu and Settings, and the CLI's --audio,
set it:

- DeviceSettings.accepted refuses a device's sendAudio, on or off, from
  either door, and names it ("Settings from ‹device› refused: send audio
  (set on this Mac only)"); the rest of the change applies.
- The device's Send Audio row shows the Mac's value, read-only (a
  disabled switch, as Direct Wireless's from afar), its footnote saying
  where it is set: "Turn on Send Audio in Sill on ‹Mac› …".
- kind 17's sendAudio stays on the wire (never reused; a later decision
  may let devices set it again, as Saffsanity#66's Mac viewer asked).
- away-quality: 12 new cases (refused on and off, home and away, named,
  the rest applied; a device still sees it; its log words) and five
  mutants (AQ20-24), 24 of 24 caught.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e's row

Noah: "if so then merge as beta support". Nothing said beta, so a person
turning it on could not tell a feature still being tried from a finished
one. Now:
- the status menu's item is "Send Audio (Beta)" (in the title, so
  menu.txt and VoiceOver have it);
- Settings › Streaming's switch is "Send audio (beta)", its footnote
  starts "Beta." and says only this Mac turns it on;
- the CLI's --audio line reads "Audio on for this run (beta): …" (only
  with --audio, so the CLI's output without it is unchanged);
- the device's Send Audio row has a small "Beta" tag after its title,
  read by VoiceOver as "Send Audio, beta", and both footnotes start
  "Beta.".
The bars' Sound button and This iPad's "Sound from ‹Mac›" switch are the
device's own mute and carry no tag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… default)

The merge with main at 5c323e6 put Sound in Larger Picture's row 1, a
seventh button from 384 pt, which the plan's own §13 rule (row 1 stays as
Noah approved it) forbids, and which it left to Noah. For the beta, row 1
is main's again (Apps, Menus while the Mac sent menus, Aa, Keyboard,
Desktop, Settings), and with Larger Picture the Mac's sound is the
Settings panel's "Sound from ‹Mac›" switch alone, as on phones narrower
than 384 pt already. The standard arrangement keeps Sound at row 2's end.

- PhonePortraitLayout: Row1Beside and row1Beside go; in Larger Picture
  `sound` and `stripBesideBoth` are the strip's empty place and
  `menusBesideSound` Menus' own place in row 1.
- PortraitStreamScreen: phoneRow1 is main's, Menus only.
- phone-portrait: the row-1 Sound cases give way to "no Sound button"
  phone by phone and over the Larger Picture grid (495 checks); A1-A4
  give way to three mutants of the new lines.
- tour: the block for Sound in row 1 goes (111,191 checks).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`stream` was set and cleared on the main actor (start, stop) and read and
cleared by SCStreamDelegate's didStopWithError on ScreenCaptureKit's own
queue, with nothing between them: a stop racing the system's end of the
stream could both take it (a stopCapture of a stream already reported
ended, or an end reported for a stream being stopped), the same kind of
race the review fixed for onPCM and onStopped (1de43aa). Now the stream
is set under a lock and only ever taken through `take(_:)`, which hands
it to exactly one of the stop, a failed start and the delegate.

Not run by any test here (no test captures a person's sound); the
change is a lock around the three places that touched it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every video frame was handed to `sill.audio` for the sound's guard (rule
3), a hop and a sorted insert per frame, even when no sound ever comes:
Send Audio off (the default), or a Mac without sound, so every user paid
it. AudioOutput now drops a frame on the network queue until the
session's first kind 43 format (a flag under its own lock, set by
`format`, cleared at a session's start and end, kept across a move's
hand-over as the sound is). The guard reads the frames of the last 2 s,
which fill within the sound's first seconds, while the jitter cover
decides the delay at the start anyway.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AudioOutput took the panel's refresh from UIScreen.main, which the app
avoids everywhere else (a window's scene has its own screen) and which
iOS has marked for deprecation. It now reads StreamClient.screenMaximumFPS(),
the viewport's own reading (the foreground scene's screen, -SillScreenFPS
in DEBUG), once, as the client is made on the main thread.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each window's StreamClient has an AudioOutput, and each added seven
block-based NotificationCenter observers it never removed: a block
observer stays registered until removed, so every window made and closed
left seven behind (inert, through their weak self, but kept for good).
The tokens are kept now and removed in deinit, which also cancels the
deadline and once-a-second timers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SharedSession counts the engines that play on the device and lets the
session go when the last one stops. A SoundEngine freed while it played
(its window's client gone with its output, its scene discarded) never
called give(), so the count stayed up for good: no other window's last
stop could deactivate the session, and the app kept it active (and other
apps' audio unnotified) until it quit. SoundEngine's deinit now stops it
as `stop` does, which gives the session back. Holding the session implies
running (stop and abandon give it back as they end the run), so nothing
is given back twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The epoch's encoder was made on sill.audio from the first chunk, and the
first AAC-ELD converter of a process pays for loading the codec: 13 to
53 ms here (a fresh process, this Mac under heavy load; every later
converter 0.2 to 1.3 ms, either packet size), and 1.9 to 3.3 s on the
plan's loaded Mac, where the chunks waited behind it and the session's
first second came as a burst of late packets (docs/audio-plan.md, "The
review", left open there).

AudioPipeline.prepare() now makes one converter once a process on a
background queue and throws it away; the coordinator calls it at start
when Send Audio is on and when it turns on, so the codec is loaded
before a device and a source need it. Each epoch still makes its own
encoder on sill.audio at its first chunk's packet size, now quickly; a
failure is still the epoch's to say.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…re drafts

The PR left the public copy for the release that carries sound, but once
it merges main is that release, and README, the App Store description and
the privacy policy said Sill plays no sound. As the clipboard (Saffsanity#78) did:

- README: How it works gains "Sound from your Mac", a beta off until Send
  Audio (Beta) is turned on; Good to know says Sill plays sound from your
  Mac only as a beta, off by default (the trademark note's wording, not
  "your Mac's sound").
- site/privacy.html (dated October 20, 2026): what the Mac sends while
  Send Audio is on, the device's report gaining how far the sound trails
  and the late packets, the mute kept on the device, and what Screen
  Recording is also for; site/support.html: "Sound from your Mac (beta)",
  one paragraph (turning it on, the Sound button and the panel's switch,
  the silent switch, headphones out, AirPods' delay). Every line inside
  "<!-- Sound: for the release that ships it (PR Saffsanity#94); …" markers.
- docs/release-checklist.md part 1 §3: the Sound markers and their perl
  line. Tried on a copy: every page but privacy.html's date (and a line
  break in one list item, and a blank line in support.html) is main's.
- docs/app-store-metadata.md: for the next version, the description's
  bullet (and "Sill doesn't play sound" out of Good to know: 3,174
  characters), the What's New paragraph, and one review-notes line in
  WHAT TO TRY with a redundant sentence of PURPOSE AND AUDIENCE out (3,995
  bytes of 4,000); the App Privacy answers and Before you submit name the
  sound. "Beta" is always on the feature, never on the app (guideline 2.2).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- docs/audio-plan.md: a header paragraph for the beta (what says beta,
  the defaults it takes: Q8 the Mac's alone, Q10 no Sound in Larger
  Picture's row 1, Q4 and Q5 kept, Q16 with H12 accepted while off by
  default) and what ends it (P1-P16, H12 met or accepted with P13, P5
  answered, P6 measured); §4.1, §7.6 and §7.7 say what the beta changed;
  Q4, Q5, Q8, Q10 and Q16 record the choice; H7 and S2 say how they run
  now that a device's change is refused.
- CLAUDE.md: the audio entry ("in, as a beta, since 2026-10-20"), the
  host's and the Sound button's bullets, the merge's Q10 note, the
  Layout (prepare, the lock, no frame before a format), Build and run,
  the decision ("Audio is in scope since 2026-09-27 … in, as a beta,
  since 2026-10-20"), and the compatibility floor's kind 43 clause, which
  gains the reader's caps (a format at most 4 KB, a packets message at
  most 16 KB: a later host that batched past them would be dropped whole
  by 1.0 devices), also noted at Audio.swift's caps.
- docs/BRIEF.md: in, as a beta, since 2026-10-20.
- docs/DEVELOPMENT.md: "The Mac's sound (beta)", the Mac's alone, the
  device's read-only row, no Sound in Larger Picture's row 1, the CLI's
  --audio the run's only switch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…'s "no sound" line

- docs/DEVELOPMENT.md: the paragraph on a device's settings says Send
  Audio shows there read-only, the Mac's alone (the beta's Q8), and the
  sound's first paragraph is wrapped again (one line ran to 112 columns).
- docs/app-store-metadata.md: the description block's lead-in listed "no
  sound" among what the next version's block adds; the next version
  carries the sound as a beta, so the lead-in says the "no sound" line
  gives way to the sound's bullet there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…de Send Audio

Main gained Saffsanity#136 (the Apps list finds apps in folders of their own, at any
depth: InstalledApps and its check). Two conflicts, both lists:
- ci.yml's mutants matrix: both sides' checks, installed-apps among them,
  with the sound's audio-codec, audio-packetizer, audio-playout and ledger.
- Tests/checks/README.md's history: main's installed-apps sentence, then
  the audio branch's, which now also says what the beta changed in
  phone-portrait, tour and away-quality.
CLAUDE.md merged by itself (InstalledApps in the Layout and the checks).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HunterAhlquist pushed a commit to HunterAhlquist/sill that referenced this pull request Oct 19, 2026
Kind 29 was free on 985a733, where the pen was built; since, main's
comment and checks keep 29 to 31 free (29 the audio plan's, 30 and 31
Saffsanity#66's side streams), 32 is the clipboard, and open pull requests hold 33
to 44 (Saffsanity#66's 33 to 42, Saffsanity#94's 43, Saffsanity#93's keyframeRequest 44, checked against
every open head today). So `case pen = 45`, and everything that names the
number follows: SessionLink.isInput (8 or 45), the comments of Pen.swift,
Switcher, PenTablet, InputInjector and StreamServer (which match the enum,
not the number), sillclient.py's --raw45 and its sends, and the docs
(DEVELOPMENT.md, CLAUDE.md's entry, Layout and Build and run, the plan's
design and gates, with a note that its results before the rebase say 29).

The compatibility floor's kinds are 0–28, 32 and 45 again with 29 to 31
free as main has them; kind 26's `seen` counts kinds 8 and 45.

The checks: compatibility names kind 45 and 46 unknown (157 checks; its
mutant "kind 45 as 46"), protocol keeps main's two kind checks and has the
pen at 45 with 46 unknown (193 + 8), pointer-control's table and sweep say
45 (158), fence's input and its two C10 mutants say 45; menus' line and
mutant and pointer-control's two renumbering mutants are main's again, 29
being free once more. clipboard passes unchanged. Run here: compatibility
157, protocol 193 and the 8 cross-checks, pointer-control 158, menus 309,
clipboard 259, pen 197 and 5,000 random sessions, all passing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Saffsanity and others added 4 commits October 20, 2026 08:34
The beta's three dated lines (CLAUDE.md's decision, BRIEF.md, the plan's
header) said "Noah on PR Saffsanity#94", which reads as a comment on the pull
request; his "merge as beta support" was an instruction about it, and
the pull request has no such comment from him.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ffsanity#88 beside Send Audio

Main gained Saffsanity#110 (Shift+Return and Shift+Tab from an iPad's keyboard reach
the Mac with the shift: InputOverlay, KeyChords and the key-strokes check),
Saffsanity#87 (VirtualDisplay's BOOL return read as "B" or "c", with its new
virtual-display-api check) and Saffsanity#88 (Sill.app universal with
SILL_UNIVERSAL=1, release.sh checking every slice). Two conflicts, both
lists, both sides kept:
- ci.yml's mutants matrix: the sorted union, 49 checks, which are exactly
  the folders with a mutants.py (the sound's audio-codec,
  audio-packetizer, audio-playout and ledger, and main's
  virtual-display-api).
- Tests/checks/README.md: the opening list names both the sound's checks
  and the virtual display's encodings; the history keeps main's
  virtual-display-api sentence, then the audio branch's.
CLAUDE.md and docs/DEVELOPMENT.md merged by themselves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…as a beta

"Plays sound from your Mac only as a beta" read as a limit on where the
sound plays; the line now says what it does and that it is a beta, off
by default, in README's own words for the two ends (your Mac, your
device).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2e65f06 stopped handing a frame to the sound's queue until the session's
first kind 43 format, so a session without sound pays no hop and no
sorted insert per frame. But the guard (rule 3: the sound never ahead of
the picture) then had no frame when the first packets were placed, so
with a picture that trails by more than the need (away, a slow link) the
first packets could play ahead of it until frames came, and the delay
then jumped. The output now keeps the stamps of the last 32 frames
(about half a second at 60 fps) under the same lock until the first
format, which hands them to the model before itself: the guard holds
from the sound's first packet, as when every frame went to the model,
and a session without sound still hands its queue nothing (a lock and an
append per frame).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature request: Add audio streaming

4 participants