Depth behind the README's "What it handles" matrix: codec routing, HDR signaling, audio bridging, subtitles, frame extraction, disc playback, and the documented edge cases. For the pipeline shapes these route through, see docs/architecture.md.
Containers (demux side): MKV, MP4 (and 3GP / 3G2), WebM, MPEG-TS, MPEG-PS (DVD VOB), AVI, ASF / WMV, OGG, FLV, RealMedia (.rm / .rmvb).
That list is the supported set, not the compiled set. The FFmpeg build also carries the HLS and DASH demuxers, and neither is a supported source shape: the build ships no network protocols (file, pipe and data only, since bytes come from the engine's own AVIOReader over URLSession), so libavformat cannot fetch what a manifest points at. Remote HLS is handled a level up instead, by AVPlayer on the bypass or by the live ingest; a remote DASH manifest has no such path and is not playable.
Hardware decode (native AVPlayer path, VideoToolbox): H.264 (progressive), HEVC, HEVC Main10, on hardware whose VideoToolbox has a decoder for the specific profile. AV1 on devices with HW AV1 (M3+ Mac, iPhone 15 Pro+, future Apple TV chips) also routes natively.
Software decode (SoftwareVideoDecoder + AVSampleBufferDisplayLayer):
- AV1 (libavcodec / dav1d) on devices without HW AV1 (currently all Apple TVs, M1 / M2 Macs, pre-A17-Pro iPhones).
- VP9 and VP8 (libavcodec native) unconditionally, since AVPlayer's HLS pipeline rejects the
vp09/vp08CODECS attributes even where VideoToolbox can HW-decode them. - MPEG-4 Part 2 (XVID / DIVX / SP / ASP), MPEG-2 video, and VC-1, none of which AVPlayer's HLS-fMP4 pipeline accepts; libavcodec ships native decoders for all three.
- The legacy Microsoft tail (FFmpegBuild 2.4.3): MS-MPEG4 v1 / v2 / v3, the DivX 3.x that pre-2005 AVI rips carry, and WMV1 / WMV2 / WMV3 (WMV9). The routing was already correct before that release, so these reached
SoftwareVideoDecoderand failed the load withunsupportedCodecfor want of a compiled-in decoder. A native.wmv/.asffile plays whole since FFmpegBuild 3.1.0, which added theasfdemuxer and the entire WMA decoder family (Standard, Pro, Lossless, Voice). That chain is all-or-nothing on purpose: the demuxer without a matching WMA decoder opens the file and then has nothing to decode its audio with,AudioBridgeinit fails, the cascade ends indroppedNoPipeline, and the session serves video-only and plays the file silently, which is a worse failure than an honest one. WMA is not fMP4-legal in any flavour, so it always takes the AudioBridge, like MP2 and Blu-ray LPCM. The video of such a file is usually WMV3 or VC-1, both of which were already here for their Matroska and MPEG-TS shapes. None of the six supports frame threading, so they decode single-threaded; the content that carries them is SD. Reported by cmcpherson274 (FFmpegBuild#3), whose library carries WMV only inside Matroska and MPEG-TS, which is why the native form waited for a second field report. - The Flash tail (FFmpegBuild 3.2.0): FLV1 (Sorenson Spark, the H.263 variant every pre-2008
.flvcarries) and the On2 VP6 family (vp6,vp6f,vp6a) that Flash 8 brought. Theflvdemuxer has shipped from the start, so a modern Flash file (H.264 + AAC) always direct-played and only the legacy decoders were missing; that release also adds the era's audio, Nellymoser Asao, ADPCM-SWF, Speex and FLV's G.711 / big-endian PCM shapes, for the same all-or-nothing reason the WMA family shipped whole. Note where that reason lives: in the FFmpeg build, not in the routing table.AudioCodecCompatdecides whether a codec is offered a stream-copy; the bridge cascade below it asks libavcodec for a decoder by id and never reads the table, so a codec the table does not name still plays when the build carries its decoder (measured on Nellymoser-in-FLV before its entry existed), and a codec the build lacks is silent whatever the table says. Flash Screen Video (flashsv/flashsv2) is deliberately absent: it needs zlib, which the build does not link, and screen recordings are not what a media library holds. Requested in the Sodalite Discord. - The 2000s tail (FFmpegBuild 3.8.0): RealMedia, with the
rmdemuxer, RealVideo 1 to 4 (rv10/rv20/rv30/rv40; an.rmvbis almost always RV40) and every RealAudio such a file carries (Cook, ATRAC3, Sipro, RealAudio Lossless, the 14.4 / 28.8 speech codecs), routed as onerealAudiofamily; the feature-phone 3GP / 3G2, H.263 with AMR narrowband or wideband speech (amr), in themovdemuxer that was already here; Theora, the video of.ogv; and DVD LPCM (pcm_dvd), which routes as.pcm. The same all-or-nothing rule as WMA and Flash. None of the audio is fMP4-legal, so all of it bridges. Contributed by williamliu0516 (FFmpegBuild#7), checked against samples.ffmpeg.org files. - Every other video codec, by default. The native path is an allowlist of its own (HEVC, H.264, HW-decodable AV1) and
HLSVideoEnginerefuses the rest withunsupportedCodec, so a codec that is merely absent from the list above does not fall back to native, it fails the load. QuickTime RLE (qtrle, FFmpegBuild 2.4.1) is the case that surfaced it, and ProRes, MJPEG, Cinepak and rawvideo share the shape. Whether such a source then plays is a question for the FFmpeg build: with no decoder compiled in it fails atSoftwareVideoDecoderwithUnsupported video codec, which is at least the honest error.AV_CODEC_ID_NONEis the one default-native case, since an audio-only source probes as NONE. - Interlaced H.264 (declared field order TT / BB / TB / BT), so the deinterlacer below can run; tvOS AVPlayer does not deinterlace, so 1080i / 576i broadcast combs on the native path (#107). On a seekable VOD source the declaration is verified against decoded frames first (#232). A field order describes how pictures are carried, not that any picture is interlaced: progressive-in-interlaced-carriage (PsF, which is what European 25 fps Blu-ray masters are, Blu-ray having no 1080p25) took this detour and then never deinterlaced, because the filter engages on
AV_FRAME_FLAG_INTERLACEDand on nothing else.InterlaceProbedecodes a short sample and applies that same predicate, so it measures the runtime's own question rather than guessing at content: only a sample in which the flag never appears sends the stream back to the native path with hardware decode, an inconclusive one keeps the software route, and a flagged frame ends the sample at once (~29 ms on 1080i, ~100 ms for a full sample). Live sources are never probed. - H.264 High 4:2:2 / 4:4:4 / High-10 and HEVC Rext (4:2:2 / 4:4:4 / 12-bit) on hardware whose VideoToolbox has no decoder for the profile (Intel Macs, older Apple TV chips). AVPlayer accepts these at the HLS CODECS level and the item reaches
readyToPlay, but the native path then renders nothing, so a per-formatVTDecompressionSessionprobe at load (VTCapabilityProbe.canHardwareDecode) routes them to libavcodec, which decodes them (#2). Apple Silicon has HW decoders for all of these and keeps them native. The probe only judges a config record that actually carries out-of-band parameter sets: a source with in-band VPS/SPS/PPS (hev1,numOfArrays = 0) gives VideoToolbox no SPS to build a session from, which says nothing about hardware support, so those keep the native path (AetherPlayer#2). - H.264 that carries both stereo views inside one track, which is how a 3D Blu-ray MVC remux is muxed: Matroska StereoMode 13 / 14 (
block_lr/block_rl, both eyes in one block), reported by libavformat as stream-levelAV_PKT_DATA_STEREO3Dof typeAV_STEREO3D_FRAMESEQUENCEand spelledstereo_modein the stream metadata (#435). The dependent view's slices reference a subset SPS the base decoder never receives, so a plain H.264 decoder can only skip them; libavcodec does and decodes the base view, which is the left eye and the 2D fallback every non-3D player shows, while VideoToolbox is handed whole samples with both views' NALs inside and renders nothing. The decision is the container's declaration read at load, no probe. The frame-packed modes (side by side, top / bottom, checkerboard, row or column interleaved, anaglyph) are single self-contained pictures that keep the native path with hardware decode, and cropping an eye out of one of those is the host's call, not the engine's. MV-HEVC keeps the native path too: it is Apple's own spatial-video format and AVPlayer plays its base layer. Real MVC 3D output is not offered on any path.
Interlaced sources (DVD-rip MPEG-2, SD / HD broadcast H.264) are deinterlaced through a persistent bwdif graph (yadif fallback) that engages on the first interlaced frame and costs nothing on progressive content. The dispatch decision lives in AetherEngine.load (VideoRoutingPolicy), gated per source on VTCapabilityProbe, codec id, declared field order, and on VOD the decode sample that verifies it.
A length-prefixed H.264 file can retain its original avcC while later samples carry different
SPS/PPS, including a resolution change under the same parameter-set IDs. Native HLS can then play
audio without displaying video, including after seeking or reopening at that position. The producer
checks the parameter sets actually referenced by an IDR before writing it into the native segment;
repeated sets and announcements that no IDR uses do not change the route. Only the part of the SPS a
decoder session is built from counts (H264SPS.decoderSignature: profile, chroma format and bit
depth, frame numbering and picture order, reference frames, size and cropping). A new PPS, VUI or
level keeps the native route, since the native decoder takes those in band: on macOS such a change
played through natively in every arm, while a profile or resolution change froze the picture.
In automatic mode, with escalatesToSoftwarePath enabled, a confirmed mismatch uses the existing
native-to-software takeover. The software decoder accepts the changing frame dimensions, while the
rebuild preserves the requested seek/resume position and session transport. This can briefly buffer
and costs software decoding CPU; the native-only features described below are unavailable on that
route. Explicit decode-path choices and disabled escalation are honored, with an error when the native
initialization cannot serve the source. Live Annex-B configuration rotation is unchanged.
Some writers emit a sample table with no ctts while the H.264 or HEVC bitstream still reorders pictures.
Every sample then reports PTS == DTS, and since the native route stream-copies those timestamps
into fMP4, AVPlayer is handed decode order as presentation order: each future reference picture is
shown before the B pictures that precede it. Measured through AVFoundation's own decoder on a twin
pair (one encode muxed twice, composition offsets removed from one), 45 of 66 pictures landed at a
time belonging to a different picture, with the content order stepping backwards 30 times (#409).
The container lost the information, but the bitstream did not: every slice header carries a picture
order count, which is display order, and libavcodec's H.264 and HEVC parsers read it without decoding
a pixel and take MP4's length-prefixed payload directly. H264CompositionOffsetRepair samples the head
(twelve pictures at most, held rather than re-read, so no rewind and no second fetch) and repairs a
confirmed source at the demuxer boundary:
PTS = (decode time of the picture that opened this coded video sequence) + shift + rank * step
DTS = DTS + shift - reorderDelay * step
Pulling decode time back by the reorder delay is what keeps PTS >= DTS; a healthy file carries the
same negative head. shift is 0 or one reorder delay, depending on whether the writer left the
sample ladder on the presentation axis or kept the edit list that trims the reorder head, and is
clamped to that range so a malformed header cannot drag the picture off its audio. Because the
rewrite happens once, in the demuxer, the fMP4 producer, the segment plan, the software decoder and
the still extractor all read one axis, and the source keeps hardware decode: a missing table costs
no route change. The container's own index is folded onto the same ladder, since the segment plan is
built from index entries and then filled with these packets.
Detection is fail-closed and costs a healthy file almost nothing: the first real PTS-DTS offset ends the sample (usually on the first packet, since a reordered file's head sample sits one delay below zero). A source is only repaired when every sampled pair is equal, the decode ladder is uniform, the picture order regresses, and the ranks it produces are distinct and fill the sampled window (or a decode-order prefix of at least nine pictures fills its own range exactly, which covers a hierarchical mini-GOP longer than the reorder delay, #699). Anything short of that (a picture order that does not advance one rank per picture, a sample that starts nowhere it can be anchored) is delivered exactly as the container wrote it. Reported by @orut34iop; the HEVC case (#699) by @ijuniorfu.
For H.264 only, a changing decode cadence can instead use the original-slot policy described below. This fallback requires a progressive IDR head with equal PTS/DTS, a container start/index lead that exactly matches the observed reorder head, and a complete, distinct picture-order prefix whose timestamps pass the slot policy. It assigns existing slots plus that lead rather than fitting a frame rate; DTS, audio and the keyframe index remain unchanged. An ambiguous head stays untouched.
Matroska block timestamps are presentation timestamps by specification, and the format has no composition-offset table to lose. A writer that fills them packet by packet while the bitstream reorders pictures has therefore not lost anything: every presentation slot is still in the file, each one just arrived attached to the picture that was decoded at that position rather than the one that is displayed there. On the reporting asset the first slots are 0, 40, 73, 107, 140 and the decoder emitted them as 0, 73, 107, 140, 40, one stepped-back presentation clock per mini-GOP for the length of the file. Measured here through the engine's own software decoder on a generated twin, 15 of 30 frame times stepped backwards before the repair and 0 after (#511).
H264MatroskaSlotPermutation therefore permutes rather than reconstructs: a picture carries the slot
its own display rank owns, and the slot is read from the file instead of being computed. Nothing fits
a cadence, so a ladder quantized from a fractional frame rate is reproduced exactly rather than to
within a tick, and a slot the writer clamped onto its cluster origin (the reporting asset has one, 7
ticks below the 1001/30 lattice its other 59 slots sit on) survives as written. Nothing moves the
decode timestamps either: libavformat derives them from the rising slot ladder, which is the decode
order the stream really has, and a picture at most its own reorder delay behind its slot cannot
violate PTS >= DTS. The container index is untouched for the same reason, since it holds keyframe
slots and a keyframe is the first picture of its own sequence.
The slot a picture needs is a packet away, not a plan away. A picture coded ahead of the slot it owns waits for the packet carrying that slot, which is the mini-GOP reorder created: three video packets on the reporting asset, in a 60-picture sequence. Nothing waits for the end of a sequence.
Detection is fail-closed and costs a healthy file almost nothing: one stepped-back slot is the
container doing what the format says, and it ends the sample on what is normally the third packet.
PTS != DTS is not an eligibility test here, because libavformat synthesizes a decode ladder from a
rising presentation one just as readily as from a reordered one. A source is only repaired when the
sampled slots rise strictly, the picture order regresses, the ranks are distinct, fill the sampled
window and are a multiple of one measured step, and no picture sits further behind its own slot than
the reorder delay the container declares. A stream that later stops being that shape, or a wait no
mini-GOP explains, hands its packets back exactly as they arrived rather than permuting half a
sequence. Diagnosed by @orut34iop on PR #511, whose numeric ladder is the regression fixture.
A healthy head does not establish a healthy table for the whole file. Some mixed MP4s retain valid offsets at the head, then give later reordered pictures zero offsets. Seeking into that region can produce persistent judder despite normal aggregate FPS and sufficient network buffering.
When a healthy origin picture corroborates the container's edit/index lead, the demuxer keeps a zero-hold healthy path and watches for zero-offset IDRs. Progressive pictures are parsed as they arrive. Valid equal PTS/DTS and distinct even POC let the streaming slot ladder assign original DTS slots plus the corroborated presentation lead by display rank. Original DTS, audio and the already published keyframe index never move. Actual timestamp slots, rather than an average-FPS clock, preserve interval changes and quantization within a sequence.
The slot a picture needs is a packet away, not a plan away, exactly as in the Matroska policy above: a picture coded ahead of its own slot waits the mini-GOP the reorder created, so nothing waits for the end of a sequence and no sequence is too long to repair. Backward displacement remains bounded by the declared reorder delay. A forward-coded anchor farther ahead needs a bounded, complete decode-prefix proof, as in #700: distinct ranks fill a prefix, the backward delay still fits, and the observed slots plus lead pass a dry run. Only the measured forward distance is then allowed, limited to 16 pictures; that proof resets at each candidate IDR. The packets held behind it remain bounded by 1024 interleaved packets and 32 MiB, which is a ceiling on a container's interleaving rather than on its GOP. A candidate sequence has to show its reordering within 64 pictures before a single picture is rewritten.
Healthy nonzero offsets resume unchanged delivery. Fields, missing timestamps, incomplete POC, a rank claimed twice, arithmetic overflow and insufficient lead are not guessed at. Every refusal hands the held packets back exactly as they arrived and lets the rest of that sequence stream through, so a shape this policy does not own costs the repair and never the session; the next IDR is a fresh candidate. Seek and teardown release both unpublished input and pending output. No decoder-route or host-UI change is needed. The uniform whole-file policy above is unchanged; its strictly proven variable-cadence H.264 fallback reuses this same slot ladder.
See the partial-composition regression and reproduction for generated fixtures, original numeric evidence and verification limits.
A sample in an mp4 video track is a run of NAL units, each introduced by a big-endian
length of the width the avcC / hvcC record declares, and a parser walks that run by
addition. A damaged source can carry a length that reaches past the end of its own
sample, and the two consumers answer that differently: libavcodec logs Invalid NAL unit size, skips the frame and plays on, while Apple's fMP4 parser answers the whole SEGMENT
with CoreMediaErrorDomain -19602, which ends the session and every reload onto that
segment. A file like that therefore plays in mpv, plays after an MKVToolNix remux (which
drops the unparsable tail), and dies here at whichever position AVPlayer first has to
decode across the damaged sample.
The session muxer walks each video sample's chain before the write and cuts it at its
last complete NAL, which is the same bytes the remux would have written. A healthy sample
is left alone, an Annex B payload is refused rather than walked as lengths, and a sample
with no complete unit at all is dropped instead of written empty. The fixture generator
Scripts/nal-overrun-fixture.py forges the shape into any length-prefixed source by
rewriting four bytes, so the healthy original stands as the control arm (AE#561).
Under that sits a last rung for the cases the cut cannot reach, because every recovery
above it answers the same bytes again: the #93 revive reloads the item at the position
that died, and the stage-2 chain refills the same segment, so a segment AVPlayer refuses
on its merits ends the session with the replacement item dying milliseconds after the
first. A failure in the CoreMedia domain is therefore offered to SoftwarePlaybackHost
before it is made terminal: the session is rebuilt at its playhead with
preferredDecodePath = .software, which decodes with libavcodec (one skipped frame rather
than a dead session) and reads the demuxer directly instead of the loopback HLS. Once per
session, and only for a verdict on the MEDIA: a URL-loading failure is a verdict on the
SOURCE, which both paths read through the same reader. Whether the software path can serve
the source at all is the existing #461 decodePathRefusal, so a source it cannot serve
costs a refusal and the original failure, not a second dead session.
| Source | Wrapper signaling |
|---|---|
| H.264, HEVC (SDR) | BT.709 |
| HEVC Main10 (HDR10) | BT.2020 / PQ |
| HEVC Main10 (HDR10+) | BT.2020 / PQ + per-frame ST 2094-40 SEI stream-copied |
| HEVC Main10 (DV P5) | dvh1 track type (DV-only, IPT-PQ base; forced even on SDR panels) |
| HEVC Main10 (DV P8.1 / P8.4 / P7) | hvc1 primary + dvvC box (P7 converted to 8.1 first), DV engaged via SUPPLEMENTAL-CODECS on DV panels, RPU-driven tone map of the HDR10 / HLG base elsewhere |
| HEVC Main10 (HLG) | BT.2020 / HLG |
| AV1 HDR | BT.2020 / PQ |
HDR-to-SDR mapping is handled by AVPlayer and the system compositor according to the connected display. AetherEngine doesn't tonemap on the host; it tells the system "this is BT.2020 PQ" (or DV, or HLG) via the HLS-fMP4 sample description and lets tvOS / iOS pick the right path.
An HDR master playlist is only served when the panel is ready for it. On tvOS the external panel must already be in HDR mode or Match Dynamic Range must be on (an SDR-parked panel rejects an HDR master with -11848). On iOS and macOS the built-in panel engages EDR on demand with no display mode switch, so AVPlayer.eligibleForHDRPlayback counts as readiness there; SDR-only devices read ineligible and stay media-direct. DisplayCriteriaController issues the HDMI content-frame-rate and dynamic-range hint via AVDisplayManager before the first segment is fetched, so the receiver-side handshake is in flight by the time AVPlayer is ready to render. (For why this ordering is mandatory on tvOS, see the README's "Host setup on tvOS" section.) The per-mode capability split still comes from AVPlayer.availableHDRModes; the 26 SDKs deprecate it in favor of the eligibility Bool but ship no per-mode replacement, and the DV5 -11868 guard needs exactly that distinction, so the engine keeps the deprecated read until Apple obsoletes it. Since 6.82.0 that read may only ADD a mode, never subtract one: supportsHDR10 and supportsHLG take eligibleForHDRPlayback as their floor, because the table under-reports HLG over HDMI. Measured on a Samsung whose EDID advertises Hybrid Log-Gamma and which plays HLG in the TV's own player, connected straight to an Apple TV, while the table reported .hlg absent; a second Apple TV on a different Samsung agreed, and an iPhone 17 Pro on its built-in panel reported it present, so the absence is about the HDMI path rather than the panel (AE#459). What that term reaches is one source: effectiveVideoFormat guards on Dolby Vision, so supportsHLG is consulted only for a DV Profile 8.4, which previously resolved to SDR on such a display while the manifest served VIDEO-RANGE=HLG. A plain HLG title was never clamped. Dolby Vision is left on the table alone, where it is measured correct in both directions (false on a Samsung without it, true on an iPhone 17 Pro the same day). That read does not exist on macOS at all (API_UNAVAILABLE(macos)), so the Mac table is derived from eligibility for HDR10 and HLG and leaves Dolby Vision unclaimed: eligibility proves EDR, not that AVFoundation will accept a DV variant on this display. A host that knows the hardware claims it with LoadOptions.panelPresentsDolbyVision, and a wrong claim costs one in-place media-playlist fallback rather than the item (AE#493) for the failures AVPlayer reports as an item failure. The claim no longer picks the packaging of a Profile 5, 8.1 or 8.4 source: 6.72.0 gave the non-DV branch its dvcC back and 6.73.0 its SUPPLEMENTAL-CODECS, so those three serve byte-identical manifests and segments with and without it, and what moves is the published videoFormat, the criteria request and the HDR readiness that rides along (AV1 Dolby Vision is still gated on it; Profile 7 joined the three once it was converted on every display). On tvOS that leaves a gap in the net: the -15628 an HDR10-only panel showed on the DV packaging in May 2026 (AE#4) is a stall rather than an item failure, it now reaches that panel with or without the claim, and it did not reproduce on tvOS 26.6. DV composition on a panel without Dolby Vision is what forceDolbyVisionOnNonDVDisplay is for there (AE#455). An HDR master additionally requires a known source frame rate (#130): AVPlayer filters a VIDEO-RANGE=PQ/HLG variant that carries no FRAME-RATE attribute out of the master at parse time and fails the item with -1002 without ever fetching the media playlist (SDR variants are accepted without it). The manifest frame rate uses the probe's avg_frame_rate with an r_frame_rate fallback; a source where both are unset (some live MPEG-TS ingests) routes media-direct instead of serving a master AVPlayer provably rejects.
On a VOD load the dynamic-range switch and the session start run side by side (AE#348). After an HDR or Dolby Vision criteria write the pre-flight gate starts at the write, as before, but only the master-or-media choice waits for it: HLSVideoEngine.prepare() opens the source, plans the segments, starts the local server and the producer cutting the first segments while the panel is still switching, and serve(panelIsInHDRMode:) picks the playlist once the gate has released and the panel has been read. AVPlayer sees nothing before that, so the routing read is taken no earlier than with the serial order, and a Dolby Vision write is never followed by a decode mid-switch. The cold start becomes roughly the longer of the switch and the segment production instead of their sum: on an Apple TV 4K with a Dolby Vision title whose first two segments took 2.0 s to cut, the serial order started playback 1.5 s after the switch had ended. start() is still prepare() followed by serve with the panel state given at init. Live loads keep the serial order, since their join is timed against the edge and the overlap has not been measured there.
Non-DV HEVC derives its primary CODECS string from the source hvcC profile_tier_level (profile space, profile, tier, level, constraint bytes), so an 8-bit Main source is not mis-declared as Main10. The compatibility-flags element is the stored general_profile_compatibility_flags in REVERSE bit order per RFC 6381 / ISO 14496-15 Annex E: a real Main10 record stores 0x20000000 and prints hvc1.2.4..., matching MP4Box and Dolby's own reference manifests. The declaration is checked against the init segment on device, so it has to be exact.
Whatever the profile, the hvcC shipped in init.mp4 must carry the parameter sets out of band. Sources authored with in-band VPS/SPS/PPS only (hev1, numOfArrays = 0, what MP4Box ...:xps_inband and the common Dolby Vision MP4 recipes produce) are normalized at load by scanning the head of the stream for the parameter sets and rebuilding the record; a record that arrives with extra non-parameter-set arrays (libx265's user-data SEI) is stripped down to VPS/SPS/PPS instead. Shipping the source record verbatim in either case leaves AVPlayer with no usable format description: it buffers the whole forward window and never renders a frame.
For DV streams the demuxer surfaces the source's AVDOVIDecoderConfigurationRecord, and the route depends on the profile's base-layer compatibility:
-
Profile 5 (DV-only, IPT-PQ, no base layer) emits a bare
dvh1.05.<dvLevel>codec tag in the primaryCODECSattribute with thedvcCbox preserved.dvh1is forced even on non-DV panels (AVPlayer's system DV decoder tonemaps IPT-PQ internally; withoutdvh1the IPT chroma reads as YCbCr and shows a green / purple cast), so P5-on-non-DV is routed through a media playlist to dodge the-11868variant rejection. -
Profiles 8.1 / 8.4 (HDR10- / HLG-compatible base) emit
hvc1.2.4.L<level>as the primaryCODECStag. The muxer writes thedvvCbox and the variant carriesdvh1.08.<dvLevel>/db1p(8.1) or/db4h(8.4) inSUPPLEMENTAL-CODECS, which is what makes AVKit engage DV on a display that can present it. Both are emitted on every display, not only a DV-capable one: the pairing is what the authoring spec asks for, it is what lets a client that does not knowdvh1read the base layer instead of failing, and the loopback master reaches AirPlay receivers that are exactly that client. It used to be stripped, against an-11868measured on tvOS 26.0 that no longer reproduces on 26.6, and keeping it is what makes AVPlayer put the RPU on the pixels wherever it has to convert the base layer, which on tvOS is every HDR source while the panel is not in HDR mode. On a panel that is already in HDR the base layer plays as plain HDR10 / HLG either way (AE#493). -
Profile 8.1 on a non-DV display, with
LoadOptions.forceDolbyVisionOnNonDVDisplay(experimental, default off, AE#455) is served the way Profile 5 is served instead:dvh1sample entry, the containerdvcCrewritten to profile 5 / compatibility 0,CODECS="dvh1.05.<dvLevel>", no supplemental. AVPlayer then runs its own Dolby Vision composition and applies the per-frame RPU to the pixels before they leave the device, where the default route gives the panel one static HDR10 grade. The bitstream never changes; only the container's claim about it does, and what makes that hold together is that a Profile 8.1 RPU already carries the mapping out of its own HDR10 base layer, so the composer does not need the container to describe that layer. What makes it experimental is that this is not what the profile field means: a decoder that read the base layer's colorimetry from the profile rather than from the RPU would decode IPT out of YCbCr, the green / violet cast of #4 and #176. Profile 8.1, and Profile 7 once converted to it. Profile 8.4's base layer is HLG, and a profile-5dvcCover an HLGcolris a container that contradicts itself. -
Any profile with a presentable base layer, with
LoadOptions.dolbyVisionHandling = .baseLayerOnlyis served as that base layer alone, on every display: plainhvc1/av01sample entry, thedvcCstripped, no supplemental, no Profile 7 conversion, and HDR10 / HLG display criteria rather thandvh1. The RPU NAL units stay in the samples and are ignored. It is a host option and not a routing decision because the case it exists for cannot be told from the container: a remux whose container record claims Profile 5 over a bitstream whose VUI declares BT.2020 YCbCr PQ, whose RPU carries the NLQ and residual fields only Profile 7 has, whose mapping is the identity and whose base layer is plain HDR10 with its own static metadata. Served asdvh1.05the decoder reads YCbCr as IPT and the picture is green / violet; served as its base layer it is the HDR10 every player that ignores the record shows. The engine logs the contradiction on the default route (DV Profile 5 record over a BT.2020 YCbCr VUI) so a host knows to offer the option. What is admitted: HEVC Profile 7 / 8.1 / 8.4 and AV1 Profile 10.1 / 10.4, whose record names the base, and a Profile 5 / AV1 10.0 record whose VUI names one (matrix_coeffsBT.2020 with a PQ or HLG transfer; a genuine Profile 5 leaves both unspecified, IPT having no VUI code point). A Profile 5 whose VUI is unspecified has no base layer to present and keeps its route. -
A Profile 5 record its own RPU contradicts is served as what the RPU says, on every display and with no option to set (#532). A remux whose container record claims Profile 5 over a bitstream whose VUI declares BT.2020 YCbCr with a PQ or HLG transfer states its own contradiction: IPT-PQ-c2 has no VUI code point, so a genuine Profile 5 leaves
matrix_coeffsandtransfer_characteristicsunspecified. For that pairing, and only for it, the engine reads the firstunspec62RPU and takes libdovi'sguessed_profileover the record, which is proof rather than a heuristic because a Profile 5 RPU cannot carry a residual or an NLQ. An RPU that reads 7 takes the Profile 7 route above, an RPU that reads 8 takes the Profile 8.1 route and its compatibility rewrite, so the served container carries the record the source should have had. An RPU that agrees with the record, an RPU that cannot be read, and every source outside that pairing read no packets at all and keep their route, which is what makes the audit free for a genuine Profile 5. The read is a second open on the source, because the session's probe demuxer is handed to the software path as it stands.LoadOptions.dolbyVisionHandling = .baseLayerOnlytakes precedence over the correction, and the Profile 5 software-path refusal (#176) stands down for a corrected record, whose base layer is plain HEVC. AV1 Profile 10.0 is not covered: its RPU rides in an ITU-T T.35 metadata OBU rather than anunspec62NAL, so the base-layer option remains the answer for that class. -
A Profile 5 the container never recorded is given the record it should have carried (
AE#recordless). Some Matroska releases are Profile 5 with theBlockAdditionMappingdropped and a VUI that says nothing, so the source loaded as plain SDR and the IPT picture was decoded as YCbCr. For HEVC 10-bit 4:2:0 with no Dolby Vision record andcolor_trc,color_spaceandcolor_primariesall unspecified, and for that pairing only, the demuxer reads the first RPU from a second open at the end of its probe; if libdovi'sguessed_profilesays 5 it appends anAV_PKT_DATA_DOVI_CONF(profile 5, level 6, rpu 1, el 0, bl 1, compatibility 0) to the stream'scoded_side_data. Because it lives on the demuxed stream, the load, the HLS producer's own open, display criteria, the segment muxer'sdvcCand every reload see a genuine Profile 5 record:dvh1sample entry,CODECS="dvh1.05.06",VIDEO-RANGE=PQ, and the #176 software-path refusal. It is served the way a recorded Profile 5 is, thecolr nclxrepair above included: the gate is an unspecified VUI, which is the shape that repair exists for, and the same bytes cannot have two deliveries depending on what the container wrote down. A failed read, or an RPU that reads anything else, leaves the source as it was..urlsources only (a custom reader has no second open, and live is skipped);LoadOptions.dolbyVisionHandling = .baseLayerOnlyis unaffected because a record with compatibility 0 has no base layer to present.
AV1+DV emits a bare dav1.10.<dvLevel> primary for Profile 10.0 (DV-only, no base layer to fall back to), and an av01... primary plus a supplemental entry for the two cross-compatible profiles: dav1.10.<dvLevel>/db1p with VIDEO-RANGE=PQ for Profile 10.1 (HDR10-compat base) and dav1.10.<dvLevel>/db4h with VIDEO-RANGE=HLG for Profile 10.4 (HLG-compat base), on hardware-AV1 hosts. Same split as HEVC: the dav1 sample entry belongs to the profile without a base layer the way dvh1 belongs to Profile 5, and a cross-compatible profile carries its base layer's own sample entry so a client that cannot read Dolby Vision still plays it.
Profile 7 (dual-layer, the common UHD-Blu-ray remux profile) is handled according to LoadOptions.preferDualLayerDolbyVision (default false: dual-layer composition is opt-in) and LoadOptions.dolbyVisionHandling. The dual-layer composer is an optional component, compiled only with the DualLayerDolbyVision package trait (off by default, README › Optional components); built without it, preferDualLayerDolbyVision changes nothing and every Profile 7 source takes the single-layer conversion:
- Dual-layer composition (
DualLayerDolbyVisiontrait,preferDualLayerDolbyVision = true,dolbyVisionHandling.automatic,el_typeFEL, VideoToolbox HEVC hardware decode): Profile 7 routes throughSoftwarePlaybackHost+DualLayerVideoDecoder. Each packet is split into the 4K base layer (BL), the enhancement layer (EL, typically 1080p, carried inunspec63NAL units) and the RPU (unspec62). Both layers decode in their ownVTDecompressionSession;DoviFelComposerapplies the RPU's reshaping (polynomial luma, MMR chroma) and the inverse-NLQ residual from the upsampled EL at 12-bit precision, then an ordered 8×8 Bayer dither (inverted on odd frames) takes the result to 10-bit P010 forAVSampleBufferDisplayLayer. The composed frame also carries the RPU converted to Profile 8.1 as aDolbyVisionRPUDataattachment, withDolbyCompatibilityIDset to the string"1"; tvOS 27 reads it on every frame and applies Dolby Vision display management, so L1 / L2 dynamic metadata reaches the panel. A MEL source never reaches the dual-layer decoder: the route is FEL only, and MEL (nothing to compose) keeps the native route below. The route needs a seekable VOD and is logged once per Profile 7 load (DV P7 dual-layer route: taken/not taken (<reason>)). It does not consult the display: on a display that does not take Dolby Vision the route is still taken and the composed picture is presented with its attached metadata, tvOS deciding how; the native route would have played the converted Profile 8.1 there. The software host also plays an E-AC-3 Atmos track as its channel-bed core (no objects). The stream is classified byel_typealone; an RPU that setsdisable_residual_flagonly switches the residual off for that frame and never changes the classification. See dolby-vision-fel.md for the design. - Single-layer conversion (the default:
preferDualLayerDolbyVision = false, a build without theDualLayerDolbyVisiontrait, or no VideoToolbox HEVC hardware decode): No Apple platform has a Profile 7 decoder, so the engine converts the stream to single-layer Profile 8.1 live during muxing: the RPU of each video packet is rewritten with libdovi (dovi_convert_rpu_with_mode, mode 2, the same transform asdovi_tool -m 2), the enhancement-layer NALs are dropped, and the containerdvvCis set to Profile 8.1. The conversion runs on every display and the result is served exactly as a Profile 8.1 is: on a DV-capable display this means real Dolby Vision (dvh1.08/db1psupplemental) instead of the plain HDR10 base, and on a display without Dolby Vision the keptdvvClets AVPlayer tone-map from the RPU rather than from the base layer's static metadata, the way mpv plays Profile 7 without an EL decoder. It used to skip the conversion and strip the record there, which cost every Mac (no per-mode table) and every HDR10 / SDR panel the RPU and shipped the enhancement layer into segments nothing decodes. The conversion is loss-free relative to what Apple could show before (the enhancement layer was never decodable on Apple hardware). MEL and FEL sources are both handled; a Full Enhancement Layer (FEL) source is logged, since its enhancement layer, which a native Profile 7 player would fold in, is discarded on this route (the dual-layer route above keeps it) while a Minimal (MEL) source loses nothing. Any per-packet conversion failure drops that RPU so the frame degrades to the clean HDR10 base, rather than shipping a Profile 7 RPU inside a container already declared 8.1. A session that converts publishesdolbyVisionConversion = .profile7ToProfile81, so a host label can name the profile actually served next tosourceDVProfile.
SDR-compatible-base profiles (HEVC Profile 8.2, AV1 Profile 10.2) carry a Rec.709 base layer that no Apple platform has a DV decoder for. The engine strips the dvcC / DV config and plays the base as plain hvc1 / av01 (logging not DV-routable, playing Rec.709 base); there is no Dolby Vision on any display for these, on DV panels and SDR panels alike.
ST 2094-40 metadata stays attached to the HEVC bitstream as user-data-registered ITU-T T.35 SEI NALs. The HLS-fMP4 stream-copy preserves the SEI through to AVPlayer, which forwards it to the system compositor. HDR10+-capable TVs apply the per-scene tone-mapping curves; HDR10-only TVs fall back to the static HDR10 base.
The published videoFormat starts at .hdr10 for any BT.2020 / PQ source and flips to .hdr10Plus the first time the producer validates a packet's HDR10+ metadata. That includes a Dolby Vision source carrying an HDR10+ layer next to its RPU (Blu-ray Profile 7 does, and so does a Profile 8.1 remuxed from one) whenever the label resolved to its HDR10 base, while sourceVideoFormat keeps saying .dolbyVision; the evidence is latched per session, so a label republished later by the panel proof (AE#459) keeps it. Debounced across producer restarts so a scrub doesn't re-fire. Hosts can drive an HDR10+ badge or analytics hook off the $videoFormat transition.
The same structural validator serves the opt-in metadata probe: codec metadata framing, registered T.35
identifiers and the complete ST 2094-40 payload must be valid. A matching byte sequence inside compressed
picture data is not metadata. FFmpeg's parsed HDR10+ packet side data remains another supported carriage,
including VP9/AV1 Matroska. No video decoder is opened just to inspect HEVC metadata. Probe confirmation
sets SourceProbe.carriesHDR10PlusMetadata independently of the primary format, preserving .dolbyVision.
See whole-probe limits and cancellation for bounded source
reads; absence of confirmation is not proof of absence or a statement about the connected display.
HDR Vivid (CUVA T/UWA 005.1) is built to be backward compatible: the base layer is plain HLG or PQ in BT.2020, and its dynamic metadata is an optional registered T.35 SEI on top (country 0x26, provider 0x0004, oriented code 0x0005). The engine plays that base exactly like any HLG or HDR10 source and the SEI is stream-copied with the rest of the bitstream; no Apple platform applies it, and a display without Vivid support shows the static base, which is the format's intended fallback. There is no host-side tone mapping: the native route has no pixel stage, tvOS gives an app no EDR or panel-peak reading to map against, and the TV maps the HDR signal itself (#699).
probe(url:detecting: .hdrVivid) reports carriage as SourceProbe.carriesHDRVividMetadata, in the
same packet pass and budget as .hdr10Plus. libavcodec's CUVA parser is internal, so
HDRVividMetadataScan walks the body with the same field widths and only counts a message whose
system_start_code is one of the defined 1 to 7, whose fields are all present, and whose bits after
the last field are zero. HEVC only, as in libavcodec. videoFormat stays .hlg / .hdr10.
The label can also be taken back from the item itself, where the platform has no capability table to clamp it against (AE#515). A Dolby Vision source on macOS resolves to .hdr10, because supportsDolbyVision is unclaimable there without a host assertion, while AVFoundation goes on playing the dvh1 sample entry the engine served. Measured with the assertion off on a 16" XDR, a Profile 5 and a Profile 8.1 grade of Dolby's reference content both strobe, so the RPU reaches the pixels with no claim set anywhere and the clamp was moving nothing but the label. When the item's sample entry reads dvh1 / dvhe and the probe agrees the source is Dolby Vision, the label is upgraded from .hdr10 to .dolbyVision at readyToPlay. It is an upgrade and not a mirror of what AVFoundation parsed, for two reasons that both matter: an .sdr label is the clamp being right about a display presenting no HDR at all, and on tvOS and iOS the per-mode table answers the capability question, so the label follows it rather than a sample entry that a Profile 5 master carries on every panel. Profile 8.1 keeps .hdr10 on macOS: it reports hvc1 with the DV configuration alongside it, it composes on that display all the same, and nothing in the stack reports that.
| Stream-copy (lossless into fMP4) | AAC-LC, AC3, EAC3, FLAC, ALAC. HE-AAC / HE-AACv2 stream-copy when the source carries an AudioSpecificConfig (any movie container) and bridge only without one (live ADTS / MPEG-TS, where a synthesized ASC would mis-signal SBR). LATM/LOAS-framed AAC (DVB broadcast framing) always bridges. FLAC whose STREAMINFO min_blocksize is below 192 bridges too (AE#764): CoreMedia drops a track that declares less, and hangs on frames smaller than the field claims, so such a stream cannot be copied (the bridge is bit-perfect for FLAC). The same field sends an audio-only FLAC to the FFmpeg host instead of AVPlayer |
Bridged (AudioBridge) |
TrueHD, MLP, DTS, DTS-HD MA, MP3, MP2, Opus, Vorbis, PCM (raw shapes plus G.711 A-law / mu-law and DVD LPCM), WMA (Standard, Pro, Lossless, Voice), the Flash tail Nellymoser Asao / ADPCM-SWF / Speex, RealAudio (Cook, ATRAC3, Sipro, Lossless, 14.4 / 28.8) and AMR-NB / AMR-WB: decoded to PCM and re-encoded |
| Surround | 5.1 / 7.1 with correct AudioChannelLayout preserved through the wrapper |
Non-streamable codecs route through AudioBridge in one of two modes (LoadOptions.audioBridgeMode):
.surroundCompat(default): the encoder is chosen per source, not per mode. A source with more than two channels is re-encoded to lossy EAC3 at 128 kbps per channel (768 kbps 5.1). AVPlayer hands the encoded bitstream to HDMI and the sink decodes its own 5.1 mix, so surround works on essentially every AVR and soundbar, including one behind legacy ARC. Two channels or fewer have no surround to carry and take the FLAC encoder instead, which is lossless where EAC3 would have been 256 kbps lossy, and LPCM the route decodes on the routes that can only pass a Dolby bitstream through (AE#395: an AirPlay 2 optical adapter played one MPEG-TS program's stream-copied AC3 5.1 and was silent on the bridged EAC3 stereo of the same program, so which trackav_find_best_streamhappened to pick decided whether the viewer heard anything)..lossless(opt-in): FLAC up to 7.1 lossless, which AVPlayer decodes to LPCM. Needs a sink that takes multichannel LPCM: an AVR over HDMI, or an AVR or soundbar on eARC (a Sonos Arc on eARC takes 6-channel LPCM, so a 7.1 source arrives as lossless 5.1 there, folded by tvOS). Behind legacy ARC, which carries stereo LPCM or a Dolby bitstream only, the LPCM is downmixed to stereo, and.surroundCompatis the right mode.
The engine does not pick between the two from the route (AE#794): with tvOS's Continuous Audio Connection on, every HDMI route reports 32 unlabelled channels whatever the sink takes, so an eARC soundbar and a legacy-ARC one look the same, and a wrong FLAC pick on the second plays stereo. The mode stays the host's choice.
Either encoder opens at a sample rate it actually has, which is not always the source's (AE#548). E-AC-3 exists at 32 / 44.1 / 48 kHz only, so a 96 kHz TrueHD or DTS-HD master is resampled to 48 kHz on the .surroundCompat arm; opening the encoder at the source rate instead made avcodec_open2 refuse the context and took the whole session to silent video-only. The rates come from avcodec_get_supported_config, so an FFmpeg bump that widens them needs no change here, and an encoder that advertises no list (FLAC) keeps the source rate untouched, which is what keeps .lossless bit-perfect at 96 and 192 kHz. The resampler is in the path on every bridged packet either way, so the conversion costs nothing that was not already being paid.
.surroundCompat is the default because the soundbar / basic-AVR install base is the majority. Object metadata (TrueHD Atmos objects, DTS:X heights) is lost in either mode: FFmpeg's EAC3 encoder doesn't produce JOC, and FLAC has no object-channel concept. objectAudioRendering is the route that keeps it in a build with the TrueHDObjectAudio / DTSXObjectAudio package traits (off by default), README › Optional components), rendered into a speaker bed and delivered as APAC (TrueHD Atmos with heights). If a JOC source ever falls through to the bridge the engine logs a loud WARNING: Atmos downgrade, ....
Two bridge lifecycle invariants (issue #99): the encoder PTS counter re-bases onto the first fed packet's (gate-shifted) source PTS on every session start and producer restart, so bridged audio always shares the video's output timeline, including a load(startPosition:) resume that anchors mid-file (a 0-based bridge timeline puts the audio track a full resume-offset away from video inside the same fragments, which AVPlayer silently discards). And the EOF tail flush leaves the encoder in FFmpeg's terminal draining state, so the bridge latches that and rebuilds the encoder on the next restart; a VOD pump that still dies with muxerFailed gets a bounded producer rebuild instead of stranding the session.
AC-4 (ATSC 3.0 / NextGen TV) and MPEG-H 3D Audio have no decoder in the bundled FFmpeg, and there is
nothing to switch on: FFmpeg carries codec ids for both so a container can be demuxed, but
libavcodec/allcodecs.c names neither. The AC-4 decoder patches have sat out of tree for years and
the format is Dolby patent-encumbered; MPEG-H has an upstream wrapper around Fraunhofer's mpeghdec,
whose licence is not LGPL-redistributable. Apple's own stack does not fill the gap either: there is no
AC-4 format constant in CoreAudio.
Such a track is not merely silent, it is expensive. has_codec_parameters fails an audio stream with
no sample rate, and that value can only come from the container or from opening a decoder, so
find_stream_info reads to the full probe budget before failing open with the track missing anyway.
On a live source that budget is spent at the wire rate, which is where Sodalite#100's minute-long
tuning indicator came from. The demuxer therefore parks a stream whose codec has no decoder AND whose
parameters the container left unset out of the probe's way, and restores it immediately after, so the
open costs what it would have without the track and the caller still sees the track in
audioTracks. A host with its own metadata (Jellyfin names a live channel's audio codec in
PlaybackInfo without opening a tuner) can do better still and refuse the channel with a real sentence.
AVFoundation reads a track's language from the master playlist, not from the media, so the audio the
engine muxes into its single variant is also declared there: one URI-less
EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aud",LANGUAGE="<iso 639-2/T>",DEFAULT=YES,AUTOSELECT=YES plus
AUDIO="aud" on the variant (RFC 8216 4.3.4.2.1, the URI is absent precisely because the audio is
inside the variant). AVPlayer then exposes it as a one-option audible AVMediaSelection group and
labels it from LANGUAGE, which is what a system audio menu shows. Measured on macOS 26: the same
fMP4 whose audio mdhd reads deu reports AVAssetTrack.languageCode == nil and no audible group
at all when the master does not declare the rendition, while the identical mdhd read from a
progressive .mp4 reports deu. The mdhd is written too (the track is that language whoever
reads it), it just is not what the label comes from. The rendition is advertised only when the audio
actually reached the variant, so an audio cascade that fell through to video-only never names a
group its segments do not carry (AE#458).
Source labels are resolved to ISO 639-2/T through ICU, which covers every language it knows plus
BCP-47 subtags (pt-BR becomes por) and rejects free text such as English or a track title, in
front of a twenty-row table for the ISO 639-2/B bibliographic codes ICU does not resolve and
Matroska routinely writes (ger, fre, cze). Two ISO 639-3 classes need more than that lookup.
The members CLDR aliases to a macrolanguage (cmn, arb, pes, swh, uzn, kmr) have no alpha3
entry at all, so the tag is canonicalized where the direct route came back empty, which resolves the
alias without moving any tag that already resolved (no stays nor, not nob). The members CLDR
does NOT alias (cnr, prs, npi, ory, quz, crs and 50 more, 56 measured on macOS 26) have neither
an alpha3 entry nor a canonical form, and there is nothing to convert: the tag already IS the ISO 639
code, so it passes through as itself. That last step is gated on ICU having a display NAME for the
tag in a fixed reference locale, which is the validity signal canonicalization cannot give
(canonicalLanguageIdentifier echoes dub and xyz back unchanged exactly as it echoes cnr).
The locale is fixed rather than the device's, or a file would resolve on an English Apple TV and not
on a German one. A label that resolves to nothing writes nothing, so an untagged source keeps the
master it had before. The one exception is a stream-copied E-AC-3 JOC track (AE#726, next section),
which gets a rendition without LANGUAGE because its CHANNELS has nowhere else to go.
EAC3+JOC packets are stream-copied through the muxer untouched, on every output route. AVPlayer reads the segment and recognises JOC from the dec3 box's ETSI TS 103 420 extension (flag_ec3_extension_type_a = 1 followed by complexity_index_type_a, the object count), with the height channels a dependent substream adds named by that substream's chan_loc bits, and then lets the downstream renderer decide: over HDMI it tunnels out as Dolby MAT 2.0 and the AVR lights up the Atmos indicator; over AirPods it renders spatially; over plain Bluetooth A2DP / LE it downmixes the bed channels to stereo natively. The route never changes the engine's decision (a JOC track is signaled in the playlist as ec-3, the same CODECS string as a non-JOC EAC3 5.1 track, so AVPlayer accepts it everywhere and the bitstream is never re-encoded for a route reason). The engine emits an explicit [HLSVideoEngine] EAC3+JOC Atmos: stream-copy engaged; ... diagnostic on every Atmos session.
CHANNELS on the audio rendition. CODECS cannot carry the distinction. A JOC track is signaled ec-3, the same string a non-JOC E-AC-3 5.1 track gets (#34: never ec+3, which tvOS 26.5 refuses), so that attribute says nothing about objects either way, and the dec3 box sits a layer below the playlist. The master's EXT-X-MEDIA:TYPE=AUDIO tag therefore carries CHANNELS, which Apple's HLS Authoring Specification makes required on every audio rendition and which Dolby's DD+ Online Delivery Kit defines for object audio as the count of decodable objects, a slash, then JOC. The engine writes CHANNELS="16/JOC" for a stream-copied E-AC-3 JOC rendition and the plain served channel count otherwise (CHANNELS="6" for a 5.1 bed); a source whose audio never reached the variant advertises no rendition and so gets no attribute. An untagged JOC stream copy still gets a rendition (AE#726): NAME="Dolby Atmos", no LANGUAGE, CHANNELS="16/JOC", and that rendition forces the master on an SDR source the same way a language does. AVFoundation lists it as one audible option named "Unknown" (measured on macOS against Dolby's untagged JOC test signal), where the tag-less master before built no audible group at all. An untagged track that is not object audio still gets no rendition, so no other untagged source changes route. Two things decide the value rather than one. The JOC form is gated on the delivery being a stream copy as well as on the source probe, because a JOC source whose stream-copy probe was rejected and which fell back to the audio bridge has no objects left in its segments and must claim none; and the count is read from the codec parameters the muxer was configured with, not the source's, so a bridged track reports the encoder's layout. The literal 16 is the object count Dolby's delivery kit and Apple's own Atmos masters carry: strictly it is complexity_index_type_a from the EC3SpecificBox, which is not available at the playlist layer.
Matroska CodecPrivate doesn't usually carry the pre-parsed dec3 / dac3 box content the mov muxer needs at avformat_write_header time, so the muxer is configured with +delay_moov (alongside +empty_moov+default_base_moof+frag_custom). The moov atom is deferred until the first fragment-cut flush, by which point packets have flowed through mov_write_packet and libavformat's handle_eac3 / handle_ac3 have populated the sample-entry boxes from the actual packet bitstream. The first cut emits the deferred ftyp+moov (routed by FragmentSplitter to init.mp4); subsequent cuts emit normal moof+mdat. Net effect: EAC3 / AC3 from matroska direct-play stream-copies cleanly with valid sample-entries, no manual bitstream parsing on the host side.
A dependent substream needs FFmpegBuild 3.7.0 or later. A Blu-ray style DD+ track is an AC-3 core syncframe (bsid 6) followed by an E-AC-3 dependent syncframe that carries the height channels and the objects. Stock handle_eac3() gets two dec3 fields wrong for that shape: chan_loc is taken from the dependent chanmap with a shift and mask that do not match ETSI TS 102 366 F.6.2.3 (fixed upstream as f10fdd6310, master only), and complexity_index_type_a is read from the independent substream alone, so the TS 103 420 extension is dropped. The box then describes a bare 5.1 bed and an Atmos receiver reports multichannel PCM (AE#728). FFmpegBuild's patch_ffmpeg_eac3_dec3 fixes both in the muxer; measured on a 5.1.2 source with chanmap = 0xA010, the payload goes from 14 00 0C 0F 02 00 to 14 00 0C 0F 02 40 01 10 (chan_loc 0x040 is Lvh/Rvh, 01 10 states the extension and 16 objects). A track whose objects sit in the independent substream already got a correct box and is unchanged.
Apple TV cannot bitstream TrueHD, and no Apple platform hands a third-party app a compressed bitstream output, so the audio bridge decodes a TrueHD track and re-encodes its 7.1 channel presentation (audioBridgeMode). For TrueHD Atmos that loses every height and every object position before the audio leaves the engine. LoadOptions.objectAudioRendering = .apac(layout) keeps them (AetherEngine#731): the engine decodes the Atmos object presentation (the bed, up to 15 dynamic objects and their object audio metadata, through LibTrueHD) together with the stream's own channel-based presentation (the 7.1, or 5.1, bed Dolby renders the same mix into), builds the SpatialSpeakerLayout bed the host names (5.1.2, 5.1.4, 7.1.2, 7.1.4 or 9.1.6, as SpatialSpeaker positions in CoreAudio's channel order) out of that channel presentation, and encodes it as Apple Positional Audio. tvOS sends APAC to an Atmos receiver or soundbar as Dolby MAT, the same way it delivers its own spatial content, and iOS plays it as Spatial Audio.
How the bed is built. Every output is a share of the stream's channel presentation (BedSplitter). Per STFT bin (2048-frame Hann windows, 512-frame hop), each of its channels is divided between the floor speakers it covers and the heights, in proportion to how much of that bin's energy comes from objects at each place: an object's part is its spectral energy times its ear-level gain into the channel, and it goes up in proportion to its layout gains above ear level. The objects only say where energy lies; no object signal reaches an output, so they are decoded without the stream's matrix interpolation values (LibTrueHD's set_ignore_interpolation, the channel presentation comes from the same access units through set_bed_companion) and taken as plain points at their positions, without size, snap, zones, screen anchoring or the elevation switch. Two corrections keep the estimate honest. Per band, an object holds no more than twice the energy its bed channels allow it, which keeps the decoding residue in quiet objects from pulling a quiet channel's content up or sideways. And per output and band, an output the split leaves short of the energy a direct render would give it is raised towards it (smoothed over about 200 ms, at most 6 dB), and one the split leaves above it is lowered by only a third of the gap in dB: objects carrying the same signal add up to more in a mix than their energies do, so a lower estimate is mostly the estimate's shortfall. A 7.1 bed maps one to one onto a 7.1 floor, a 5.1 bed's surrounds cover a 7.1 layout's sides and rears, a 7.1 bed's rears fold onto a 5.1 floor, and the LFE is the bed's own. The split runs 2048 frames behind its input: the bridge starts its output that much later after every reset and drains it before the encoder restarts elsewhere and at the end of input, so nothing moves against the picture and nothing is cut off. The exact decode mixed by the renderer, which the split is measured against, is only built with the TrueHDReferenceRender package trait and a LibTrueHD built with its matrix-interpolation feature (measurement builds, never shipped); AETHER_TRUEHD_REFERENCE_RENDER=1 in the environment then selects it. A shipped build contains neither the interpolating decode nor the size, snap, zone, screen and elevation mapping for TrueHD.
The renderer, which gives the split its object gains and routes DTS:X's channels, is allocentric, the model Atmos metadata is authored in: the room is a unit cube, and an object is balanced constant-power between the two speaker planes that bracket its height, the two rows that bracket its depth and the two speakers that bracket its width, so moving an object never changes its loudness. Ceiling speakers sit where ETSI TS 103 448 puts them, a quarter of the room in from each wall rather than in its corners, so an object on the ceiling a third of the way back plays mostly from the front tops, as Apple's own Atmos render places it. A 5.1 bed's surrounds sit halfway down the side walls, also as TS 103 448 places them, so on 5.1.2 and 5.1.4 an object at the back wall stays on them. In the TrueHDReferenceRender measurement build, object size spreads an object over a grid of points summed by power, snap sends it to the nearest speaker, zone masks keep it off the speakers they exclude, and an object without elevation stays off the ceiling. Bed channels the layout has route discretely, LFE goes to the LFE channel, and metadata updates ramp gains sample-accurately as the stream specifies. After a seek, decoding resumes at the next major sync, and the stream's first object metadata follows it up to 32 ms later; the decoder holds the audio in between and plays it at that metadata's positions, rather than leaving every object silent until it arrives (it waits at most 100 ms).
Track choice. With the option on, the automatic audio pick prefers a TrueHD Atmos track over the other tracks of the same language: of the preferredAudioLanguages match, or of the session's default track when no preference decides it; in a file whose audio tracks carry no language tag at all, it replaces the default with the first one. It never crosses a language or a dub (a region or script both tags state and disagree on), never picks a commentary for it, applies only where the bed can be rendered (a VOD video session off the software path, on OS 26 or later), and an explicit audioSourceStreamIndexOverride still wins. Without that, a film carrying DTS-HD and TrueHD Atmos in one language usually opened on whichever the container listed first.
It is opt-in because it is a trade: APAC is lossy, at several Mbit/s, in place of the lossless 7.1 the channel bridge delivers. It applies only to TrueHD that FFmpeg marks as Atmos, at 48 kHz, on OS 26 or later where the APAC encoder exists; anything else, and any failure to start, keeps audioBridgeMode with the audio intact. Silo built the first implementation in its fork and offered it as the reference this one is ported from. The decoder and the renderer are an optional component, compiled only with the TrueHDObjectAudio package trait, off by default; built without it, TrueHD Atmos keeps the channel bridge whatever the option asks, and the session log says so (README › Optional components).
Atmos on the receiver (tvOS). tvOS sends APAC over HDMI as Dolby MAT, and some sinks show Dolby Atmos for it on their own, but not all: a Samsung HW-Q990C soundbar fed by an Apple TV 4K (tvOS 27.2) played APAC without ever showing Atmos, although AVAudioSession reported a MAT format throughout. While an APAC session plays, the engine therefore loops a silent E-AC-3 stream signalled as Dolby Atmos on a second, unmuted player (AtmosOutputActivator). On the same setup that brought the soundbar to Atmos before the first frame and kept it there across a seek. Nothing the app can read (AVAudioSession's output format or channel count) tracked the indicator, so only the sink can confirm it. The stream is 2 s of digital silence written bit by bit from the E-AC-3 and object audio specifications (Scripts/make-silent-atmos.swift), plays only while the engine's player plays or waits to play an APAC item, and is released otherwise. iOS and macOS do not run it.
A linked peak limiter sits between the renderer and the encoder: the APAC encoder carries samples over full scale through unchanged (a tone at amplitude 4.0 decodes back at 4.001), and a rendered bed can get there, since object gains reach +15 dB and up to sixteen elements sum into one speaker. One gain for every channel, so no object moves; instant attack, so nothing leaves above −0.18 dBFS without a lookahead delay that would shift the audio against the picture; a 250 ms release. A bed under the threshold passes bit for bit.
Cost. Decoding the object presentation and encoding APAC each take about one core: on Dolby's Unfold demo (27 s, M1, release build) the TrueHD decode took 1.36 s (0.92 s since LibTrueHD 2.2.0, 0.60 s through its C API since 2.3.0), the APAC encode 1.26 s and rendering plus limiting 0.06 s. Apple's encoder runs on one thread, so the bridge hands each rendered 2048-frame chunk to a serial encode queue and decodes the next one meanwhile (APACEncodePipeline), which took the whole sample from 2.73 s to 1.75 s, and with LibTrueHD 2.2.0's faster decode to 1.40 s, 19x real time against 10x in series. The encoder is now the slower half, so the decoder no longer sets the pace. That time is what a seek past the produced range waits for before its first segment, so it is the one place the trade shows. The queue inherits the producer's QoS, a pool of three chunk buffers holds the producer at most two chunks ahead of the encoder, and the producer waits for the queue (settle()) before a keyframe advances its segment cutter or a write cuts the open segment, so every packet is routed as the encode in series routed it and every segment carries the same audio, byte for byte. Those waits cost about 0.1 s of the sample's overlap.
What the session log says. [SpatialAudioBridge] object configuration names the presentation (LFE + 15 objects for Dolby's Unfold demo). A bed levels line gives each bed channel's RMS and peak with its CoreAudio name, how many objects were active and how many elevated, the LFE element's level and gain, and the limiter's work over the same window when it had any; it comes after the first 5 s of audio (and again after every seek), then every 30 s. The decoder waits for a major sync after a start or seek, so a decoder that returns nothing for 1200 packets in a row, a second of TrueHD, is reported as one that produces nothing, the same failure the channel bridge reports. The object
counts reach a host too, once a second, as LiveTelemetry.objectAudio, so a stats panel can say how
many objects play and how many overhead without reading the log.
DTS:X reaches the bridge the same way TrueHD Atmos does: Apple TV cannot bitstream DTS-HD, so audioBridgeMode decodes the lossless 7.1, and on a DTS:X track that 7.1 is a compatible downmix into which the encoder folded the four height channels. objectAudioRendering = .apac(layout) keeps them in a build with the DTSXObjectAudio package trait (off by default; without it DTS-HD keeps the channel bridge whatever the option asks, README › Optional components): a DTS-HD Master Audio track is decoded by FFmpeg's own dca decoder into the same renderer, limiter and APAC encoder, and the engine side is a speaker mapping (DTSXObjectAudioDecoder). No library comes in for it: FFmpegBuild patches the decoder (#798).
Only DTS:X's standard presentation is rendered: a 7.1 bed and four fixed heights (the top front and top back pairs), each height folded at -3 dB into the bed speaker under it (top front left into L, top back left into the left rear, and so on) so that a 7.1 decoder plays the whole soundtrack. The heights travel in a regular XLL channel set at the end of every lossless frame, behind a syncword upstream FFmpeg recognises (it is what makes it label a track "DTS-HD MA + DTS:X") but then skips. FFmpegBuild's patch decodes that channel set with FFmpeg's own XLL code and, with the decoder option x_heights that DTSXObjectAudioDecoder sets, outputs 7.1.4 and subtracts the fold from the bed at output resolution; without the option the decoder's output is unchanged. It does not read the stream's private metadata: every frame of twelve DTS:X tracks measured (eight Harry Potter films, two Shrek films in two languages, Mary and the Witch's Flower) stated this same fold. The bed and the heights route discretely, one channel per speaker, so DTS:X is rendered into the two layouts that carry all twelve one to one, 7.1.4 and 9.1.6; any other layout keeps the channel bridge, and so does a track in one of the alternate presentations, which add one to five objects (some IMAX-labelled streams, Trinnov's "Experience" trailer): the decoder leaves those at their 7.1, the verdict refuses them, and they play as the 7.1 their compatible bed is.
Which tracks. The probe labels DTS:X from the frames it read, and some DTS:X it does not label at all (Trinnov's "Experience" trailer, whose extension FFmpeg does not recognise). Every 48 kHz DTS-HD MA track is therefore offered to the decoder, and its decoded frames decide: the track renders on the first one that comes out as 7.1.4, and is refused after eight that do not (a stream entered inside a peak-bit-rate smoothing period decodes its first frames from the lossy core alone). The engine shows them to it through the demuxer's peek (peekPackets), which holds what it reads for the producer, so deciding costs no extra read and no seek: on a resume the demuxer is put where the first producer starts before the peek, not after it, and the frames there decide. The peek holds at most 512 packets or 24 MiB, every stream's included, and gives up without a verdict past either. A refused track, or one without a verdict, keeps audioBridgeMode with its lossless 7.1 as before, and a stream that turns out not to be 48 kHz is refused on its first frame. Lossy DTS (core, ES, 96/24, High Resolution) is never offered. The automatic audio pick prefers a track FFmpeg labels DTS:X the way it prefers TrueHD Atmos; a plain-labelled DTS:X track is rendered when it is the track that plays, but cannot be ranked before playback.
Seeks. A DTS-HD Master Audio frame is lossless only against the core decoder's filter and predictor history, which a stream entered mid-way does not have: the first frame after a seek decodes distorted (over 120 seeks into five DTS:X streams, its error a median 2.7 dB below the signal) and the next can still err by -16 dBFS. The decoder decodes the first two frames after a start or seek without output and counts them on its input timeline, so the bridge's first encoded frame sits 21 ms after the segment's start, where those frames end, and nothing fills the gap; the first block then differs from a continuous decode by at most -33 dBFS, and the output is bit-identical again within about ten frames.
On the receiver. Over Jellyfin on an Apple TV 4K (tvOS 27.2), 8 Mile and The Bourne Identity played as APAC with the soundbar showing Atmos, from the middle of the film and across seeks (back to playing in 0.7 to 2.2 s), with the same AtmosOutputActivator TrueHD Atmos uses; Arrival, plain DTS-HD MA, was refused on its first frame and showed PCM. Those runs predate the decoder in FFmpeg.
What the session log says. [HLSVideoEngine] DTS-HD MA track is DTS:X 7.1.4 (first frame, 3 ms) names the layout the verdict found, and a refusal names its reason (no DTS:X extension in the standard presentation, 8 channels, or DTS:X in a presentation with objects, which plays as its 7.1 bed for a stream FFmpeg labels IMAX). A frame whose height channel set does not decode keeps the 7.1.4 layout with silent heights and the bed as coded, and libavcodec logs DTS:X height channels not decoded, output silent heights. A layout other than 7.1.4 or 9.1.6 logs DTS:X is rendered into 7.1.4 or 9.1.6 only. The object configuration and bed levels lines are the same as for TrueHD Atmos. The 1200-packet silence report is too: at 512 samples a DTS frame that is 12.8 s of DTS, against a second of TrueHD.
Subtitle cues come from one read: EVERY embedded subtitle stream (text and bitmap) stays in the session demuxer's keep-set and is harvested by a tap on the host's existing source read (each packet is observed then dropped, never muxed). Harvested packets are retained compressed in a per-session SubtitlePacketStore (300 s trailing window, byte-capped per stream) and a playhead-paced drainer decodes the selected stream near the playhead into the overlay. PGS streams in MPEG-TS (Blu-ray) arrive as one display set split across several PES packets (PCS|WDS|PDS|ODS|END, some without a PTS of their own); the store reassembles those chunks into one self-contained entry at the PCS presentation PTS before retention, so the drainer always decodes complete display sets (Matroska already carries one complete set per packet and is stored as-is), so enabling any embedded track is instant: the selection backfills synchronously from the store, with no positioning seek and no recovery machinery, even on remote disc images; the tap re-attaches with the producer across seeks and restarts. The tap's forward coverage ends at the producer's read position (its forward park, #102), which on direct play sits only a few seconds past the playhead; on VOD sessions a background subtitle-only side reader (the forward prefetcher, #151) therefore extends the store past the drainer's 60 s lead window, by a 15 s margin so the set at the window's forward edge still has its own successor stored (#362; raised to 270 s while a bitmap track's OCR rendition is armed, which clears the OCR worker's own 240 s window by 30 s so the store actually holds what the worker is about to read), so subtitleCues holds cues ahead of the playhead for a host-applied ADVANCE sync offset, text and bitmap alike (live sessions skip it; content past the edge does not exist). It is best-effort: if it cannot open or wedges, the tap-fed behavior above is unchanged. Its positioning seek is anchored on the subtitle stream's own timestamp axis (#234): left to libavformat's default reference the anchor follows whichever stream is not fully discarded, so the pacing stream that gives the park its control point (#230) would move it onto video keyframes, and on Matroska that lands in a later cluster and drops a landing cue that starts further back than the destination. The side reader shares the source link with the video pump, and on Matroska it is a second full copy of the stream (matroska_parse_cluster reads every block off the wire; only matroska_parse_block then honours the discard flag), so a subtitled session asks the link for about twice the media rate. Where the link has little headroom the two starve each other, so the video path has priority (#240): a side reader fetches while the pump is parked, yields while the pump is fetching and completely while a seek is in flight, keeps a bounded grace window after each anchor so a freshly selected track still fills, and takes the link back after a 60 s continuous yield so a pump that never parks cannot disable lookahead for the session. A playhead jump re-anchors the running session in place rather than rebuilding it, since a rebuild pays an open, the Matroska cue-index prewarm and a positioning seek, each a bounded range the origin delivers in full. Measured on a 1.4x-headroom bench, far-seek landings improved about 45% at the median and the side reader's share of the link halved. Sidecars are fetched once. Each packet decodes through avcodec_decode_subtitle2 (except in-band CEA-608, which has an in-house line-21 decoder, see below), and the result lands in a single [SubtitleCue] published list:
- Text codecs (SubRip / ASS / SSA / WebVTT / mov_text) →
SubtitleCue.body = .text(String), or.richText([SubtitleTextRun])when the cue asks for styling (#233). libavcodec converts every text format into an ASS event line before the engine sees it, so one override parser serves all of them: inline\b/\i/\u/\s/\c/\1c/\fn/\fs/\rbecome attributes on the runs (isBold,isItalic,isUnderlined,isStruckThrough,color,fontName,fontSize), and a cue that asks for nothing stays plain.textwith the exact string it produced before.\Nbecomes a real newline either way, andcue.textflattens both forms. Cue-level\anand\poslift out intoSubtitleCue.placement(ASS numpad alignment plus an optional[0..1]-normalised anchor, y from the top) instead of splitting a run. WebVTT cue settings reach the same field from packet side data (AV_PKT_DATA_WEBVTT_SETTINGS): the WebVTT decoder dropsline/position/align, but the demuxer keeps them, soline:10% align:startarrives as an alignment and an anchor.sizeandverticalhave no equivalent in the placement model and are ignored, and apositionwithout alinekeeps only the alignment column, since an anchor point needs both axes. DVB teletext subtitles (live broadcast) decode through libzvbi (libzvbi_teletextdec, configured to emit ASS so the per-character colour broadcasters use to distinguish speakers survives rather than being flattened) with page-state semantics (#107). A page carrying colour publishes asSubtitleCue.body = .richText([SubtitleTextRun])(each run an optional RGBSubtitleColor, nil meaning the host default); a page with no colour stays plain.text, andcue.textflattens either form for consumers that do not render colour. Teletext placement arrives two ways, because libzvbi writes it two ways: on a page flagged as a subtitle page it derives the vertical anchor from the grid row and emits{\anN}, which lands inSubtitleCue.placementlike any other alignment; on a page it does not flag (broadcaster leaves NEWSFLASH / SUBTITLE / SUPPRESS_HEADER unset, or the row-0 header has not been seen yet) it writes the whole page instead, one row per line, and the ordinal of the first non-blank row is the only carrier of the position. The engine applies libzvbi's own third-of-the-page formula to that ordinal, so a caption the broadcaster moved to the top of frame stays there either way, and never overrides an\anthat did arrive. That same split governs whitespace. The unflagged page is the raw grid, every row at full column width with a hard space per cell, so the engine trims the padding off both ends of each row and folds the empty rows it also emits; a page the decoder curates itself arrives untouched, since there the surviving padding is the relative indentation carrying the alignment it chose and the blank lines are its vertical fine-positioning inside the block. The decoded page defaults to libzvbi's auto-detected subtitle page, overridable per channel viaLoadOptions.teletextPage(for example AU page 801, which libzvbi does not always flag as a subtitle page) and changeable while the channel plays viasetTeletextPage(_:), which rebuilds the drain decoder of every channel currently showing a teletext track and leaves the others alone (#364). libzvbi emits page content open-ended ("valid until replaced") and page erases as empty events, so every teletext event trims earlier open cues at its start; roll-up captions build and replace cleanly, an erase clears the line, and a 120 s cap bounds a ghost line if transmission stops without either. Validated end to end against Australian FTA broadcasts (1080i25 H.264, captions on page 801), where the interlaced video deinterlaces through the software path's bwdif filter. - Bitmap codecs (PGS / HDMV PGS / DVB / DVD) →
.image(SubtitleImage). The indexed pixel plane is walked through its palette, premultiplied against alpha, and wrapped as aCGImage. Position is normalised in[0..1]against the composition canvas, whose coded pixel size rides along asSubtitleImage.canvasSize(#112): a cropped-video rip can author a canvas taller than the coded video, so hosts map the canvas width-aligned and center-anchored onto the on-screen video rect to land cues where the disc authored them;.zeromeans treat canvas == video. A display set carrying multiple composition objects (a forced sign plus dialogue is a common real-disc shape) fans into one cue per object, all sharing the set's start PTS, and every object is retained and rendered (#146); each object'sAV_SUBTITLE_FLAG_FORCEDrides along asSubtitleImage.isForced, surfaced per cue viaSubtitleCue.isForced(track-level forcedness stays onTrackInfo.isForced). - External files (a separate
.srt/.ass/.vttURL) → register as first-class tracks (see below) or one-shot viaselectSidecarSubtitle(url:httpHeaders:), which opens its own short-livedAVFormatContext, decodes the whole file once, atomically swaps the result intosubtitleCues. The fetch forwards the session'sLoadOptions.httpHeadersby default (WebDAV auth and friends); pass the call's ownhttpHeadersto override per fetch. - In-band CEA-608 closed captions (
eia_608/ QuickTimec608, a demuxable caption track) →.text(String). FFmpegBuild ships noccaptiondecoder, so these never reachavcodec_decode_subtitle2. Instead a read-only tap on the segment producer's existing source connection reads the caption track'scc_data(its packets are kept in the demuxer's keep-set, observed, then dropped, never muxed, so the loopback-HLS output stays byte-identical), an in-house line-21 decoder (validated against FFmpeg'sccaption_dec.c) turns it into cues, and they publish on the samesubtitleCuesoverlay path as every other codec. First cut: field-1 / channel CC1; CEA-708 (DTVCC) and field 2 are follow-ons. Captions carried only inside the video bitstream (ATSC A/53cc_data, the US broadcast/cable case) are extracted too (#131): on the native remux path the segment producer scans H.264/HEVC video packets foruser_data_registered_itu_t_t35SEI (GA94), reorders the decode-order groups to presentation order by the packet DTS watermark, and feeds the same line-21 decoder; on the software-decode path (MPEG-2 and friends) the triplets come fromAV_FRAME_DATA_A53_CCdecoded-frame side data. Since no caption AVStream exists, a syntheticeia_608track (id 99608) surfaces lazily on the first real (non-padding) caption pair, so uncaptioned channels never show a dead menu entry. Host-overlay only (no PiP / AirPlay), like the bitmap codecs. (#77)
External subtitle files register with the engine and appear in subtitleTracks next to the embedded streams, so a host keeps one track list and one selection call (#88):
- Registration.
LoadOptions.externalSubtitles: [ExternalSubtitleTrack]declares files at load;addExternalSubtitleTrack(_:)registers any time mid-session (returns the createdTrackInfo). The descriptor carriesurl, optionalname/language/ disposition flags, per-trackhttpHeaders(nil forwards the session's), and aformatHintfor URLs whose path hides the extension. - Containers with several subtitle streams. An external URL can be a container rather than a sidecar (an MKV holding English, English SDH and Spanish).
ExternalSubtitleTrack.sourceStreamIndexnames which stream to decode as an ABSOLUTEAVStreamindex inside that container, so a host registers one track per stream against the same URL; nil decodes the container's first subtitle stream. An index that is out of range or names a non-subtitle stream fails the decode rather than falling back, which would be indistinguishable from leaving it nil. Tracks sharing a URL and headers are filled from a SINGLE pass over the container, so N tracks cost one fetch, not N (#266). Note thatTrackInfo.codecis still derived from the URL extension, so a container URL reportssubrip; setformatHint: "ass"when the streams are ASS and the host drives a styled renderer. - VobSub (
.idx+.sub). Only from a local file URL. Thevobsubdemuxer reads the.idxit is handed and opens the.subitself, by the.idx's own name through the file protocol, so the pair has to sit side by side on disk. Over HTTP the engine opens a sidecar through its ownAVIOReaderwithout a path, the demuxer has no name to derive the.subfrom, and the open fails; a host that serves VobSub remotely downloads both files first. Since FFmpegBuild 3.8.0. - Identity. External
TrackInfo.ids are synthetic:AetherEngine.externalSubtitleTrackIDBase(100 000) + registration ordinal, monotonic per load; load-declared tracks getbase + array indexin order.TrackInfo.isExternaldistinguishes them from AVStream-indexed embedded tracks. - Selection.
selectSubtitleTrack(index:)andselectSecondarySubtitleTrack(index:)accept external ids and route onto the whole-file decode internally;activeSubtitleTrackIndexpublishes the external id like any other selection.removeExternalSubtitleTrack(id:)unregisters (an active selection is cleared). - Renditions. Load-declared external tracks join the native WebVTT renditions (next section): their store is filled by one whole-file decode at load (one pass per container, covering every track pointing at it) and marked finished, and a finished store also backfills the fullscreen overlay instantly on select (no re-download; styled-ASS selections re-decode to keep raw markup). A store that could not be filled stays unfinished rather than serving a complete but blank
.vtt. Tracks added after load are host-overlay only until the next load, because the rendition set is fixed in the master playlist at item creation. - Renditions on the
nativeRemoteHLSbypass (#316). That path plays the origin playlist directly and has no loopback master to declare anything in, so a declared sidecar used to be dropped outright. For a VOD source the engine now fetches the origin master, absolutises every variant / audio / key URI against the origin, adds oneEXT-X-MEDIA:TYPE=SUBTITLESper text sidecar (joining the origin's own subtitles group when it has one, so a variant's group choice cannot lose them) and serves that master from the loopback origin. The media never moves: AVPlayer fetches A/V from the origin exactly as before, which is the property the bypass exists for (E-AC-3 / Atmos passthrough). Renditions are declaredDEFAULT=NO,AUTOSELECT=NOand withoutFORCED, the same discipline as the loopback master, so nothing self-engages; the track keeps its external id and selecting it drivesAVMediaSelectionrather than the overlay, so it survives PiP / AirPlay / an external display. Live playlists (noEXT-X-ENDLIST), bitmap sidecars (.sup), an unparseable or unrewritable playlist and an origin slower than the 5 s budget all keep the origin URL and overlay-only subtitles. - Preferences.
preferredSubtitleLanguagesranks external tracks together with embedded ones. A track added mid-session re-runs the preference and auto-activates on a match, but only while the host has made no explicit subtitle call (select / sidecar / clear) in the session, so a deliberate subtitles-off stays off.
LoadOptions can seed the initial audio and subtitle tracks from an ordered language preference, resolved from the engine's single probe so a host honors a saved preference without a separate pre-probe or a post-load reload:
preferredAudioLanguages(ordered BCP-47 tags, ISO 639-1 / 639-2 / 639-3 codes or English names, e.g.["en-GB", "de"]) picks the first-frame audio track: an explicitaudioSourceStreamIndexwins, else the best track for the first preference that matches at all, else the container default. The pick is muxed into the loopback HLS, so it is correct on the first frame with noselectAudioTrackreload.preferredSubtitleLanguagesactivates a subtitle at the end of load. Within the first preference that has a match, it picks the best track by language specificity and then by container disposition: full subtitles rank over SDH (HEARING_IMPAIRED), forced, and commentary (COMMENT), and text over bitmap. No match leaves subtitles off. It drives the host-overlay path, so unlike audio it needs no reload regardless; it only spares a host from language-matchingsubtitleTracksitself. The native menu (below) keeps its own host-driven default selection viasetNativeSubtitleSelected(track:).
Matching (#590). Two labels name the same language when their canonical ISO 639-2/T identity agrees, so case, ISO 639-1 / 639-2/B / 639-2/T / 639-3 and English names all fold together (en == eng == english == en-US, de == deu == ger). A 639-3 member CLDR aliases onto its macrolanguage folds with it (cmn == zh), but one it does not stays its own language: yue and nan never answer a zh preference, because wrong audio is worse than the container default. An empty or missing label, und, and free-form text such as a track title are never a match.
Preference order dominates, so an earlier preference on a worse-ranked track still beats a later preference on a perfect one. Within one preference the more specific tag wins, scored on script first and region second: agreement (including both silent) beats one side stating it, which beats both stating it differently. So an en-GB preference takes en-GB over eng over en-US, and a zh-Hant preference takes zh-TW over a bare zho. For subtitles an explicitly opposite script is not a weak answer but no answer at all, so zh-Hant never resolves to a zh-CN track and subtitles stay off instead; audio has no such reading failure, so there a script mismatch merely ranks last. zh-CN / zh-SG count as Simplified and zh-TW / zh-HK / zh-MO as Traditional, because that is the only field a media server has to make the distinction with, while bare zh / chi / zho states no script and is a fallback for either. Matching reads the tag's own subtags and never Locale.Language's inferred ones, which claim a script for every language they know (zh -> Hans, en -> Latn, nan -> Hans) and would turn an unspecified track into an explicit and sometimes wrong one.
The same resolution picks the DEFAULT=YES native rendition from nativeSubtitlePreferredLanguages, so the inline overlay and the PiP / AirPlay rendition cannot disagree about which track was meant. Nothing is inferred from TrackInfo.name, ExternalSubtitleTrack.name or a sidecar filename; the structured language field is the whole input. The resolved tracks are published on player.activeAudioTrackIndex / player.activeSubtitleTrackIndex (both match TrackInfo.id), and every TrackInfo carries isDefault / isForced / isHearingImpaired / isCommentary (from container dispositions) so a host can rank or filter the track lists the same way.
A second subtitle channel can run alongside the primary for bilingual playback / language learning: selectSecondarySubtitleTrack(index:) for an embedded track and selectSecondarySidecarSubtitle(url:httpHeaders:) for a sidecar file, mirroring the primary API. Its cues land in a separate @Published secondarySubtitleCues list (so the host can render the two channels independently, e.g. top vs bottom), with isSecondarySubtitleActive and isLoadingSecondarySubtitles for UI state; clearSecondarySubtitle() tears it down. The secondary channel decodes through the same demux loop and PTS rules as the primary.
A single packet that carries multiple rects (PGS often emits signs/songs at the top alongside dialogue at the bottom) becomes multiple cues at the same time range, and the host renders all of them. Cues are inserted in sorted order; re-emitted events after a seek dedupe by time range plus content (so two simultaneous speaker lines with identical timing both survive) and the list doesn't grow on rewind.
A PGS composition carries no end of its own: it is published with FFmpeg's open-ended placeholder end (end_display_time = UINT32_MAX) and closed when its successor composition or clear arrives. A seek can outrun that successor, and the retention prune cannot help (it filters on endTime, which a placeholder end never ages out of), so a jump past an open cue used to leave it in the published window covering the new playhead until the next composition displaced it (#357). On a seek the drain now closes every still-open cue that began before its reconstruction window (playhead minus the 15 s backscan) at that window's start: past that line nothing is re-decoded, so nothing there can be confirmed as still open. The cue stays in the retained list for a backward seek, only its unconfirmed end retires, and an authored duration is never touched. The #100 stale-arrival hold is dropped on the same tick for the same reason.
That close is the last resort, not the rule. The end a set is authored with is the PTS of the next packet on its stream, and the packet store holds that packet long before the drain window reaches it, so every tick closes each still-open cue at SubtitlePacketStore.firstPTS(streamIndex:after:) (#362). Without it the drain window's forward edge decides the end whenever it falls between a set and its clear: the set publishes open, the cursor moves past it, and the next thing to touch it is a composition at the next landing, tens or hundreds of seconds later (report: 3.55 s authored, 76.7 s delivered, and 817 s in the same session). For the store to be able to answer, the harvest has to lead the decode: the forward prefetcher parks a 15 s margin BEYOND the drain window (subtitleForwardPrefetchLeadMarginSeconds), because with both lines at 60 s the set at the edge is systematically the one whose clear is stored nowhere. Where the store genuinely has nothing after a set (its harvest frontier, a stream cut short) the cue stays open and #357's boundary close still owns it, since inventing an end there is the laundering this replaced.
The same report has a second face, and it is the store's own bookkeeping rather than the cue's. After a seek the pump restarts behind the landing and fills forward while the store still holds an island the previous run harvested further ahead, so the drain window reads as "packets, hole, packets". Decoding across that hole carries the drain cursor to its far side, and the cursor only moves forward: the hole's packets land a second later and are never read, so a stretch of the film carries no subtitles at all until some later seek resets the window behind it, and the set before the hole is closed at the island rather than at its own clear (report: eleven authored sets delivered as two). The size of the gap cannot tell that apart from a silence the author left, and a threshold on it is actively harmful, since a set is separated from its own clear by its display duration. Harvest ORDER can: a run reads a stream forwards, so within one run PTS and sequence rise together, and a PTS-ascending pair whose sequence DESCENDS is two runs meeting over a span neither of them has read. A tick stops at that boundary and waits (harvestGapCut, #362), which costs nothing visible because the drain runs its lead ahead of the playhead; the wait ends when the filling run re-harvests across the boundary, when the playhead catches up to it, or after a bounded tick budget, so an authored silence can never stall delivery. A tick that waited says so, as outcome=harvestHole gapAt=.
A hole behind the playhead has a third face, and it is the landing itself (#416). The gate that reconstructs the active line at a seek target seeds it from the newest set decoded behind the playhead, and that set is the active line only if nothing on its stream happened in between. The store's silence is read as that proof, and over a stretch nobody read it proves nothing: a run re-aimed just after it harvested a set leaves that set's own clear on the far side of the skipped ground, so it decodes at the landing looking unclosed and publishes over the new scene, ending at the next stored packet, which is the far side of the authored silence rather than its own successor (report: a two-second sound-effect caption standing ten seconds over the wrong scene, on both an Apple TV 4K and a Mac). #362 established that the packets alone cannot show this: a reader re-anchored FORWARD hangs its packets in ascending order behind the stretch it skipped, so the pair looks exactly like an authored silence, and only a reader restarted BEHIND leaves the descending sequence harvestGapCut reads. So the readers now say what they read. SubtitleHarvestCoverage in the packet store keeps one span per run, anchored where the reader positioned and extended as it goes: the forward prefetcher reports its own read position, the pump's run begins where the producer opens or restarts and reaches at least the playhead, since playback is rendering there. A set whose ground up to the playhead is not covered cannot claim the landing (landingWithheld= on the delivery line), and the pass ends on the next authored set as it would have anyway. A store nobody reports to answers every span with yes, so a path without coverage notes behaves exactly as it did. The cost is the landing line in the case where a set really is still up and the proof is missing, which needs an authored dwell long enough to span the whole unread stretch; the alternative was paying it for every normally authored set that ends inside one.
Subtitle cues land in raw source PTS. On the native path, AVPlayer's HLS clock sits at source_pts - producer.videoShiftPts (the producer applies a per-session shift to align the first segment's tfdt with the playlist origin, and the shift can change on every restart). Render the overlay against player.sourceTime so cues match the spoken audio regardless of which producer session is active.
Host-rendered subtitle overlays are invisible in Picture-in-Picture, AirPlay, and external-display sessions because those paths render the AVPlayerLayer content only; the SwiftUI / UIKit view tree is not composited. The engine therefore serves every text subtitle track as a real HLS SUBTITLES rendition over the loopback: the master playlist carries one language-tagged EXT-X-MEDIA:TYPE=SUBTITLES entry per track (DEFAULT=NO,AUTOSELECT=NO) plus SUBTITLES="subs" on the variant, backed by a per-track media playlist (subs_N.m3u8) whose WebVTT segments mirror the video segments 1:1. AVFoundation exposes the renditions as a standard legible AVMediaSelection group that travels with the stream everywhere AVPlayer goes, including PiP. (An earlier design muxed mov_text/tx3g traks into the fMP4 itself; in-band timed text is not HLS-conformant and AVPlayer rejected the stream, so the WebVTT rendition replaced it.)
Opt-in. Off by default (LoadOptions.prepareNativeSubtitles = false): no renditions in the master, no legible menu, output identical to before.
Cue source: the producer pump tap. The segment producer already reads the source's full interleave, so the text subtitle streams stay in its keep-set and every packet is handed to a session-level tap that decodes into per-track cue stores (the same pattern as the CEA-608 tap). Zero side-channel bandwidth, and coverage is by construction the produced region, across seeks and producer restarts. The host overlay is fed separately, by the packet-store drainer (#112 rework, see Subtitles above); a lazy per-selection reader still covers AVKit's ~240 s forward .vtt prefetch beyond the produced region. Load-declared external tracks (#88) have no demuxable stream to tap; their store is filled by one whole-file decode at load and marked finished, so their rendition serves complete .vtt files from the start.
Routing scope. A SUBTITLES rendition can only live in a master playlist, so native subtitles ride the master-routing rules: SDR sources on any panel, HDR / DV sources on HDR-ready panels. HDR-on-SDR-panel and DV Profile 5 on non-DV panels stay media-direct (no master, hence no native subtitles there); the host overlay still covers fullscreen. Bitmap subtitles (PGS / DVB / DVD) join as OCR-fed renditions: while a bitmap track is selected, a worker decodes its harvested packets ahead of the playhead (composition ends resolved at the next composition/clear event, the 5.14.1 sidecar semantics) and recognizes them on-device (Vision, track-language hinted) into plain-text cues for the track's rendition. Recognition is lossy by design; a failed or empty read drops that line from the rendition while fullscreen keeps the pixel-accurate bitmap overlay. External .sup sidecars fill their store from the selection-time sidecar decode's own image cues (no second fetch). Two shapes are out of scope for the loopback renditions entirely: live sources, whose own SUBTITLES renditions reach the host through the overlay instead (below), and the software decode path (AV1 without hardware decode, VP9), which has no AVPlayerItem for a legible group to live on, so prepareNativeSubtitles is inert there and the host overlay owns subtitles on every surface.
Wireless AirPlay (#86, #227). While an iOS session plays to a wireless AirPlay receiver, the engine reloads its loopback over the device's LAN IP: the receiver fetches the stream for itself and cannot reach 127.0.0.1. It keeps the master there, so the SUBTITLES renditions travel and setNativeSubtitleSelected(track:) has a legible group to select against; the EXT-X-MEDIA URIs are relative, so they resolve against the LAN base with no further work. How the session gets onto the LAN address depends on whether it can be reopened where it is: one that can is rebuilt at the playhead, while a sequential origin, which could only be read again from byte 0, and a live session on the engine's own HLS ingest, a forward-only source that cannot be rebuilt at all (#526), swap the player's item onto the new address instead and carry on serving. The live one rejoins its playlist at the player's join point, as a live reload does. Both cross back onto the loopback the same way when the receiver lets go.
Whether an HDR or Dolby Vision master is accepted is the receiver's decision, and it turns on the receiver's own output mode (measured 2026-07-27, iPhone 17 Pro to an Apple TV 4K, DV P8.1 4K source). With the Apple TV's video format fixed to 4K Dolby Vision it takes the DV master and plays it with subtitles. With the format at 4K SDR it refuses every HDR master, and Match Dynamic Range does not help: that setting switches only when tvOS decides the content warrants it, which it evidently never does for AirPlay content. This is the same rule the engine already applies locally, where an HDR source on a panel that is not in HDR mode is served the media playlist, and it is what DrHurt described in #86. Dressing the manifest up does not move it either: against a parked receiver, dropping the DV SUPPLEMENTAL-CODECS, clamping the declared BANDWIDTH, omitting RESOLUTION and declaring HDCP-LEVEL=TYPE-1 each changed nothing, and declaring the range as SDR was already disproven in #98.
The refusal is silent, so it has to be caught by watching progress: no -11868, no failed item, the rate flickers to playing for a single tick so even hasEverPlayed latches, and the picture never starts while AVKit shows its "not playable on this display" sign. Five seconds without a segment fetched on a master handed to a receiver reloads the LAN media playlist, which every receiver takes, and that receiver is remembered by route UID for the rest of the process so it goes straight to media from then on. A second master attempt, to exploit the output switch the first one triggers, was tried on device and only doubled the wait. For subtitles on HDR content over AirPlay the answer is the receiver's setting: fix its video format to HDR or Dolby Vision. $nativeSubtitleRenditionsServed reports which playlist is actually in use, so a host can tell the user their subtitles will not travel to this route rather than dropping them silently. A wired HDMI external display is a different route and keeps the loopback plus its master.
Master-rejection fallback (#98, #130). When AVPlayer rejects the served master (-11868 AVErrorNoCompatibleAlternatesForExternalDisplay, -11848 for an SDR-parked panel, or -1002 when every variant was filtered at master parse time), the engine reloads the bare media playlist in place; a VOD session comes back where the refused item was placed (after a stage-2 recovery reload, that is the recovered position, not the session's first start), and a live session rejoins at the edge instead of replaying its stale start position. HDR / DV on an SDR external display is therefore media-playlist-driven (an AVKit limitation: forcing VIDEO-RANGE=SDR does not fool the external-display compatibility gate, which checks the real colr / codec rather than the manifest string), so the SUBTITLES renditions do not travel there. The separate #35 cold-DV-start readiness gate, whose scenario is an HDR TV, first tries an HDR-preserving reduced master (SUPPLEMENTAL-CODECS dropped so it is plain HDR10, source range and SUBTITLES group kept) before the bare media playlist, so a cold DV start keeps HDR10 plus subtitles instead of dropping straight to subtitle-less media.
Rich ASS styling. With LoadOptions.preserveASSMarkup the tap keeps raw ASS event lines so the host overlay renders full styling (positions, colours); the WebVTT renditions strip the markup at serve time, so PiP shows plain text in the system caption style.
Timing. Served cues are on the AVPlayer clock axis, and producer restarts are timeline-exact (see architecture), so cues stay in sync with the picture across seeks and restarts.
Selection: deliberately not automatic. The renditions ship DEFAULT=NO,AUTOSELECT=NO so AVKit never engages one on its own and a host overlay never double-renders in fullscreen. A host that shows AVKit's stock chrome can still let the user pick from the native legible menu; Sodalite-style hosts select programmatically per surface instead.
Selection: host-driven API. These members on AetherEngine drive the native renditions programmatically:
// true once cues from at least one text track are decoded into the native stores
engine.$nativeSubtitleRenditionAvailable // @Published var Bool
// true while the served playlist carries the SUBTITLES group; goes false on a
// media-playlist fallback, on the wireless-AirPlay hop for an HDR / DV source, and
// when the receiver refuses the master it was handed (#227). Hosts use it to decide
// whether to draw their own subtitle window on a wired external display instead
// (#98), or to tell the user that subtitles will not travel to this route
engine.$nativeSubtitleRenditionsServed // @Published var Bool
// ordered list of all native subtitle renditions (ordinal, language tag, display name)
engine.$nativeSubtitleTracks // @Published var [NativeSubtitleTrack]
// select a rendition by ordinal (nil deselects); language-tag match, positional fallback.
// Re-asserts automatically if AVFoundation drops the selection during a stall recovery.
engine.setNativeSubtitleSelected(track ordinal: Int?)
// convenience for the enter/leave pattern below: true resolves the rendition matching the
// currently-active overlay track and selects it, false deselects
engine.setNativeSubtitleRendering(_ active: Bool)NativeSubtitleTrack carries .ordinal (position in the rendition declaration), .language (ISO 639-2 tag), and .displayName (localized name suitable for a picker label).
The recommended host pattern for PiP / AirPlay:
- Observe
$nativeSubtitleRenditionAvailable(waits for the first cues to be ready before activating). - On entering PiP / AirPlay / external display: call
setNativeSubtitleRendering(true)(or resolve the ordinal yourself viasetNativeSubtitleSelected(track:)), and hide the host overlay. - On leaving: call
setNativeSubtitleRendering(false)and re-enable the host overlay.
This avoids double subtitles during inline playback (where the host overlay is already painting them) and ensures the user sees subtitles the moment the stream is mirrored or sent to PiP.
Device-verification checklist (required before tagging a release):
- Selecting a rendition displays it in the PiP window and survives seeks (including seeks that restart the producer).
- Inline host ASS rendering unchanged: rich styling intact, tap-fed cues appear instantly on selection.
- No double subtitles while inline; no rendition is auto-selected on session start.
- Timing: no constant offset between audio and subtitle cues, before and after seeks.
- SDR / HDR10 picture behavior unchanged with
prepareNativeSubtitles = true(the renditions only add master tags + subtitle endpoints). - HDR-on-SDR-panel and DV Profile 5 on non-DV panels still play (media-direct, no renditions there by design).
- Memory bounded by total cue count across all tracks.
A live channel carries its subtitles the same way a VOD stream does, as an EXT-X-MEDIA:TYPE=SUBTITLES group in the upstream master, and the live ingest demuxes the picked video variant only, so that group never reaches the demuxer: before #359 a channel offering three subtitle languages produced no subtitle track at all. The engine now surfaces the master's renditions as tracks (synthetic ids from liveSubtitleRenditionTrackIDBase, 300000) and, when the host selects one, fetches its WebVTT segments and publishes the cues on the same host-overlay surface the CEA-608 tap uses. This is the overlay path, not a legible AVMediaSelection: the live loopback serves a media playlist, which cannot declare a rendition (see the routing scope above).
It is deliberately lazy. The tracks come from the master's declaration alone and nothing is fetched until a track is selected, so a channel watched without subtitles pays no second HTTP loop. Selection starts a poll of the rendition playlist at its EXT-X-TARGETDURATION, and only unseen segment URIs are fetched, so a poll that arrives before the window moved costs one small request.
Three properties are worth knowing when reading [LiveSubs] log lines:
- Placement comes from the playlist geometry, not from
X-TIMESTAMP-MAP. The spec's own anchor is unusable in the field: measured against a public broadcaster, every segment carries the same constant map whose MPEGTS value sits two hours away from the video PTS. What renditions of one program do share, byte for byte, isEXT-X-MEDIA-SEQUENCEandEXT-X-PROGRAM-DATE-TIME, so a cue is placed at its segment's broadcast time plus its offset inside that segment. An upstream withoutEXT-X-PROGRAM-DATE-TIMEtherefore has no placement at all: the loop says so in the log and stops rather than publishing something plausible and wrong. - The fetch is anchored at the playhead, not at the head of the playlist. A rendition playlist is not a handful of segments (a two hour DVR window is 3600 entries), so everything older than 120 s behind the playhead is marked seen without being fetched. Enabling subtitles and jumping back a little still lands in covered ground.
- A producer seam republishes the source-axis shift, and cues already placed then refer to an axis that no longer exists. The loop watches its own pairing and re-anchors, which is what keeps a long session from drifting further out the longer it runs (6.21.1).
aetherctl play --live-ingest --subs <lang> exercises this against a real channel; it is the only CLI route that does, and its absence is why #359 stayed invisible for as long as it did.
Hosts that render authored ASS styling themselves (positioning, speaker colours, karaoke) opt out of the stripping with LoadOptions(preserveASSMarkup: true): cues then carry the raw event line (override tags, style references, escapes intact), the script header ([Script Info] + [V4+ Styles]) is surfaced, and engine.fontAttachments carries the container's embedded fonts (TTF / OTF) for the renderer's font directory. ASSScriptBuilder reassembles raw event cues + header into a complete script for whole-file renderers such as swift-ass-renderer's loadTrack(content:), hardened against real-world Matroska tracks (synthesized [Events] section, NUL stripping, content-keyed dedupe since real files hardcode ReadOrder: 0).
The flag reaches ASS and SSA tracks only, on the embedded and the sidecar path alike. libavcodec normalises SubRip, WebVTT and mov_text into event lines of the same shape through ff_ass_add_rect, so those rects carry an ASS payload the engine could emit and deliberately never does: no header is surfaced for them (TrackInfo.assHeader is ASS/SSA-only), which would leave a host stripping the nine leading fields by hand. A session that mixes an ASS track with a SubRip one therefore needs no reload to cross between them, and aetherctl play --preserve-ass-markup --subs <track> shows which of the two a given track is (AE#587).
The header arrives differently per source: embedded tracks carry it on TrackInfo.assHeader, and external .ass / .ssa sidecars loaded through selectSidecarSubtitle(url:) under the same preserveASSMarkup flag carry it on engine.sidecarASSHeader (extracted from the file's subtitle-stream extradata; nil for SRT / VTT and when preservation is off). Both pair with the raw event-line cues the same way (AetherEngine#48).
The host stays in charge of the actual paint: text styling, overlay layout, fade transitions, position scaling against the on-screen video rect.
FrameExtractor produces still CGImages from a media URL through an FFmpeg decode context that is fully isolated from playback. It never touches the playback pipeline, the HLS loopback server, or the engine's shared state, so a scrub-preview decode can't perturb the frame on screen. Two modes share one decode core:
thumbnail(at:maxWidth:): seeks to the nearest keyframe, no forward decode, downscaled tomaxWidth(default 320). Cheap and fast; built for scrub previews and Recents lists.snapshot(at:maxSize:): decodes forward to the exact PTS, full ormaxSize-clamped resolution. Built for user-triggered stills.
let frames = engine.makeFrameExtractor() // nil if nothing is loaded
// or, for an arbitrary item (e.g. a Recents row):
let frames = FrameExtractor(url: url, httpHeaders: headers)
await frames.prewarm() // optional: hide cold-start at gesture begin
let preview = await frames.thumbnail(at: 612.0) // CGImage?, nearest keyframe
let still = await frames.snapshot(at: 612.0) // CGImage?, frame-accurate
await frames.shutdown() // prompt teardown of the decode contextHDR sources come out looking right: PQ / HLG BT.2020 frames are tone-mapped to SDR BT.709 through a zscale + tonemap libavfilter graph before the CGImage is built, so HDR10 / HLG / DV P8.x stills match what the user sees instead of washed-out grey. Dolby Vision Profile 5 and AV1 Profile 10.0 (IPT-PQ base layers with no HDR10 fallback) route through DolbyVisionStillConverter, an optional component compiled only with the DolbyVisionStillReshaping package trait (off by default, README › Optional components); built without it, such a source yields no still at all (nil, supportsCacheBackedStills false, no I-frame rendition, no held picture) rather than the planes read as YCbCr. With the trait, the converter applies the RPU colour transform (ycc_to_rgb + PQ EOTF + the IPT-PQ LMS matrices carried in AV_FRAME_DATA_DOVI_METADATA) before tone-mapping, so their stills come out with correct colour instead of the green / magenta cast a plain YCbCr read produces.
FrameExtractor is an actor. Blocking FFmpeg work runs on a dedicated serial queue, never on the cooperative thread pool. The decode context opens lazily on first use; a superseded request (the common case during an active scrub) cancels the in-flight decode so the latest position wins. Results land in a bounded LRU cache (snapshots and thumbnails kept in separate stores, thumbnails bucketed by second). After 10 s idle the context closes and the cache drops automatically; the next request reopens lazily. shutdown() is the explicit, permanent teardown. The engine does not retain the extractor returned by makeFrameExtractor(); the caller owns its lifecycle.
Decrypted disc images play through the normal decode path via a synthetic seekable byte source. DiscReader detects and routes both local .iso URLs and MediaSource.custom ISO readers.
- DVD-Video ISO:
ISO9660Readerreads the ISO9660 bridge filesystem (an image without a bridge volume, UDF only, hasVIDEO_TSlisted byUDFReaderinstead, once noBDMVis found there),DVDIFOParserreads the VMGI (VIDEO_TS.IFO) TT_SRPT to enumerate the disc's titles and each title set's VTS IFO (VTS_NN_0.IFO) program chain for the title duration and chapters,DVDTitleSelectorgroups each title set's content VOBs (whole-VTS, largest first), andConcatIOReaderpresents the selected title's concatenated VOBs as one seekable source demuxed as MPEG-PS. On an unreadable VMGI it falls back to the VOB-size grouping; an unreadable VTS IFO leaves the title's duration and chapters empty but still plays. The main program chain's cells are folded onto one title timeline with the Blu-ray clip fold, each cell's timestamp base read from its navigation packs, since many discs restart the timestamps at a cell; a seek goes to the VOBU's byte through the VTS time map (VTS_TMAPT). Only a chain whose cells follow each other from the first title sector, with no angle block, is folded. - Blu-ray ISO: a read-only
UDFReader(UDF 2.50, including the metadata partition and fragmented-file allocation descriptors) resolves BDMV,MPLSParser+BDTitleSelectorenumerate every.mplsplaylist as a selectable title (longest first so id 0 is the main feature; trivially short menu / FBI-warning playlists filtered), and the selected title's.m2tsclips are concatenated and demuxed as MPEG-TS (H.264 / HEVC / VC-1, AC3 / EAC3 / DTS / TrueHD / LPCM, PGS subtitles). A seek goes by the clips' EP maps (BDMV/CLIPINF/*.clpi): the folded title time names the clip, itsin_timeturns that into the clip's own clock, and the map gives the keyframe's byte, so the seek is one byte reposition and holds on titles whose clips run their clocks over the same range, where a timestamp search lands in the wrong clip. A title with a clip that has no readable EP map seeks by timestamp.
Both: no decryption (CSS / AACS retail discs must be ripped decrypted first), no GPL nav libraries, no menus, BD-J, or multi-angle.
A remote .iso / .img / .udf needs an origin that honours HTTP Range, because a disc image is a filesystem that only random access can walk. The range probe is judged at the response head: an origin that answers it with a 200 fails the load with an error naming the cause (the body is never downloaded), and a range answered wider than asked is cut where the range ends.
Blu-ray folders. A disc that was ripped as a BDMV folder rather than an image plays the same way through MediaSource.custom with a DiscDirectoryReader: the host lists the folder's files (path relative to the disc root, and size) and opens any one of them on request. HTTPDiscDirectoryReader is the ready-made one, with one URL per file, so an origin serves the disc as plain files over Range and never remuxes it. Recognition is the image path with files in place of UDF extents: the same bounded playlist scan and title selection, the selected title's .m2ts files concatenated in playlist order as one MPEG-TS, the same multi-clip timeline, chapters and STN track languages, and the same seek by the clips' EP maps when the folder carries BDMV/CLIPINF. A host that already knows the main playlist names it in preferredPlaylist, and only that playlist is read and published as the title. The concatenating reader opens a clip's file when the read reaches it and keeps at most three open. A controlled probe (limits or cancellation) does not recognize a folder yet; an uncontrolled probe(source:) and load(source:) do.
DVD-Video folders. A listing that holds VIDEO_TS/ files and no playable Blu-ray title is read as DVD-Video instead, by the same title-set code as an image: the selected title set's VOBs are concatenated in part order, which is the same run of bytes as their sectors on a disc, so the cell timeline and the VTS time map apply unchanged. preferredPlaylist does not apply.
Title selection. engine.discTitles (@Published [TitleInfo]) lists the disc's titles (id, name, duration, chapter count) and engine.selectedDiscTitle is the active one; engine.selectTitle(id:) switches title, rebuilding the pipeline from the new title's head. The selection survives audio-track switches and background-resume reloads, and a fresh load defaults to the main title (an out-of-range id clamps to it). Blu-ray enumerates all playlists; a DVD enumerates its title sets (the VMGI TT_SRPT title list, resolved whole-VTS, with the duration read from each VTS's main program chain; per-cell / episodic splitting is deferred).
Chapters. engine.discChapters (@Published [ChapterInfo]) carries the selected title's chapters; engine.selectChapter(id:) seeks to one (a thin seek wrapper, no pipeline rebuild). For Blu-ray they come from the playlist's PlayListMark entries (entry marks only; link points dropped), each mark's timestamp on its clip's STC offset by the clip's in_time and the cumulative duration of preceding play items. For DVD they come from the main program chain's program map plus the cumulative cell playback times. Chapter starts are title-relative (0-based); selectChapter adds the title's content-start base (the native playlist shift, or the software path's container start PTS) so the seek lands on the source-PTS playback axis.
Track languages. Neither disc format carries a track language in the stream, so a title demuxed on its own reports every audio and subtitle track as undetermined and preferredAudioLanguages / preferredSubtitleLanguages have nothing to match on. The languages are read out of the disc's own navigation data instead and backfilled onto the tracks by stream id: on Blu-ray from every PlayItem's STN table (the ISO 639-2 codes beside each stream's PID, first declaration winning), on DVD from the VTS IFO audio and subpicture attribute tables, with the title's main program chain naming the substream each attribute is actually carried as (a stream the chain marks absent is dropped, and without a readable chain the attribute's position is used, which is how the great majority of discs are authored). Only an undetermined track is filled in: a language the container really declares stays authoritative. aetherctl disc-inspect prints what a disc declares per title, so a disc whose tracks stay undetermined can be told apart from a disc that declares nothing (#527).
Subpicture streams and the probe. An MPEG-PS demuxer can meet a new stream at any packet, so libavformat never lets find_stream_info stop early on a DVD. With the playback budget that meant 50 MB read on every open, and a subtitle whose first packet comes after it was not a track at all. For a title whose VTS IFO is readable, the engine creates every subpicture stream the IFO declares before the probe. mpegps routes packets to an existing stream by id, and the engine joins each subpicture's PES fragments itself, because libavformat's dvdsub parser only attaches to streams the demuxer created. Such a title then probes 8 MB / 5 s (#651). Stream indices on such a title start with those subpicture streams. A disc without a readable VTS IFO keeps the full budget.
Container chapters. engine.mediaChapters (@Published [ChapterInfo]) carries the chapters a Matroska or MP4 container declares, read off the probe demuxer at load. It is empty for disc sources, which publish discChapters instead, so exactly one of the two is populated. Unlike disc chapters these need no base: a non-disc source plays on the container's own PTS axis on both backends, so startSeconds is a timestamp a host hands straight to seek(to:). selectChapter(id:) resolves against discChapters only and no-ops for a container chapter id. Ids are assigned sequentially in start order, so they stay usable as list indices, and untitled entries are numbered "Chapter N". A chapter's duration runs to the next chapter's start rather than to its declared end, because muxers routinely write end == start; the last entry falls back to its declared end, then to the container duration.
MediaSource.custom(reader, formatHint:) hands the engine a host-owned byte source: a memory buffer, an encrypted container, a tuner spool, anything that is not a plain URL. read, seek and close are required; cancel(), makeIndependentReader() and discImageProbeEnabled have defaults. Every call arrives on the engine's demux thread, never main, and inside an autorelease pool the engine opens, so a reader built on FileHandle or NSData does not strand one autoreleased object per read on a pump thread that runs for the length of a session (#445).
cancel() unblocks, it does not invalidate. Its whole job is to wake a read that is parked so teardown does not hang; a memory or file reader leaves it at the default no-op. A reader the engine may rebuild in place must be able to serve again immediately afterwards, because the rebuild reuses it. Reading cancel() as terminal (the way a socket's in-flight-request cancel is terminal) is the difference between a rebuilt session and a dead one: the reopen fails on its first read and the session lands in .error carrying libavformat's Operation not permitted. What that death LOOKS like is the reader's own convention, so one host defect has two signatures: a latched reader whose read returns a negative value produces the Operation not permitted above, while one that returns 0 is mapped to AVERROR_EOF and the session ENDS instead of failing, which on a live source is a stream that was never supposed to have an end (cmcpherson274). The engine cancels a reader it intends to reuse exactly once, before the successor opens.
In-place rebuilds reuse the reader. reloadAtCurrentPosition() and reloadAtCurrentPosition(applying:), an audio-track switch, a disc-title switch and a background return all tear the pipeline down and reopen on the retained reader instead of reopening a URL, so a custom source gets the same session-preserving rebuild a URL source does (except that the Profile 7 dual-layer route is decided by load(): reloadAtCurrentPosition(applying:) refuses a preferDualLayerDolbyVision change (or a dolbyVisionHandling change while the preference is on) on a custom Profile 7 source with SessionReloadRefusal.dualLayerRouteNeedsFreshLoad). A reader that reports itself non-seekable is refused instead (SessionReloadRefusal.customSourceNotSeekable), since the rebuild cannot reposition it.
Where the reopen starts depends on whether the source is live. A fresh AVIOContext always starts its byte axis at 0, so the reader's cursor and that axis have to agree, and there are two ways to make them:
-
VOD. The reader is rewound to 0 and the axis starts there. The reopen has to re-read the container header, and the backend seeks to the resume position afterwards.
-
Live. The reader is left exactly where the session left it and the axis is moved to the cursor instead (
[Demuxer] live reopen aligned to the reader's cursor at N bytes). A live source has kept receiving during the rebuild, so rewinding it would replay the host's whole delivered window: measured onaetherctl customio --live, the rewind dropped the playhead from 41.5 s to 1.9 s and asked the host to re-deliver 15 MB, a 61 s window, at I/O speed. Aligning instead makes the rebuild the edge rejoinLiveReloadPolicyalready performs on the URL branch.This costs one thing worth knowing: the reopen re-probes the container from a mid-stream byte offset, so it needs a source that can be joined there. MPEG-TS resyncs on its own sync bytes and recovers within a GOP. A live reader that will not answer
seek(0, SEEK_CUR)cannot be aligned to, so it is rewound as a VOD source would be, and the engine says so rather than doing it silently.
The axis a reader reports on is the axis it is asked on. The alignment above reads the cursor with seek(0, SEEK_CUR) and hands that number straight back as a SEEK_SET; the seekability probe makes the same round trip at every open; libavformat's own probe seeks then travel the same axis. Absolute file offsets and offsets counted from wherever this install's stream joined are both fine, and a reader counting from its join is reported in that axis (aligned to the reader's cursor at N bytes carries bytes-since-join, not a position in a file). What breaks is reporting on one axis and taking SEEK_SET on the other: the round trip then MOVES the source by the join offset, on a probe whose whole purpose is that it moves nothing, and the session reads from a stretch the host has not delivered yet. Both halves are checked. The probe compares its own return against the position it was just told, which costs no extra callback and fires on every open including every in-place rebuild, and the alignment asks once more where the reader is, so a live reopen never reports an alignment that did not hold:
[Demuxer] custom source: the reader reported byte 2097152 and answered a seek back to it with
4194304, so its position report and its SEEK_SET argument are not on the same axis; the source
has been repositioned by the seekability probe
A host whose own position arithmetic is keyed to vending a stream has to re-key it before opting into in-place rebuilds (cmcpherson274's corollary, and the reason his wrapper still refuses one). A rebuild consumes no new stream, so anything a host initialises at the point it hands one over, a live clock baseline, a DVR join anchor, the per-stream ownership that makes an unguarded cursor safe, is not re-initialised by a rebuild and will disagree with the session that comes back. The engine's alignment fixes the bytes; it cannot fix a clock it cannot see.
The optional AetherEngineSMB product plays media off an SMB2/3 share through the normal decode path, no server-side mount. SMBConnection (backed by SMBClient, MIT, a pure-Swift SMB2 client over NWConnection) is a ByteRangeSource; SMBIOReader adapts it to the engine's IOReader, bridging each synchronous demux-thread read to SMBClient's async API across a happens-before semaphore edge. The reader is seekable, so audio-track switching, background reload, embedded subtitles, and scrub previews all work (makeIndependentReader() opens a second cursor over the same connection).
Read-only. NTLMv2 and guest auth (no Kerberos, which tvOS lacks). No writing, locking, or directory browsing. SMBClient negotiates only SMB 2.0.2 and 2.1, so there is no SMB3 transport encryption or AES-CMAC signing; a server configured SMB3-only or with smb encrypt = required won't connect. The connection is persistent (SMBClient plus a FileReader) and reads are serialised per connection, which clears typical media bitrates comfortably. The dependency is linked only by consumers of the AetherEngineSMB product, so the core engine and its tvOS hosts never pull it. SMBClient replaced AMSMB2/libsmb2, which EPERMs on tvOS / iOS. On tvOS the host supplies the local-network entitlement (and NSLocalNetworkUsageDescription) to reach a LAN share.
A live HLS upstream can be ingested directly via HLSLiveIngestReader (a public forward-only IOReader), no media server in the data path. Segments are fetched through a bounded prefetch pipeline (up to 4 in flight, committed to the byte stream strictly in playlist order), so per-segment connection + TTFB latency overlaps and high-bitrate channels buffer ahead of real time (#177). Contract: MPEG-TS segments, including demuxed-audio variants (EXT-X-MEDIA audio groups, fetched by a companion reader and merged by DTS) and packed-audio renditions (raw ADTS framed by ID3 timestamps). AES-128 clear-key segments (EXT-X-KEY:METHOD=AES-128, the standard FAST-channel scheme) are decrypted in-line by HLSSegmentDecryptor: the key is fetched once per clip and memoised, each segment decrypted (AES-128-CBC / PKCS7) before demux. SAMPLE-AES / keyless AES-128 (no URI), fMP4 playlists (EXT-X-MAP), and a key-fetch / decrypt failure terminate with a typed HLSIngestError so the host can fall back to a server-mediated URL. This is standard HLS clear-key, not FairPlay / Widevine.
Server-side ad insertion (SSAI) plays through the direct path instead of bouncing to a server transcode at the ad break. FAST channels (Pluto and similar) splice ad creatives that restart the source clock and often carry a different video PID, resolution, and SPS than the program. The producer detects the program switch, parses the ad's SPS/PPS by hand (H264SPS) to build a fresh codec config, rotates the fMP4 muxer, and emits a versioned #EXT-X-MAP per discontinuity so AVPlayer resyncs cleanly across the init and resolution change; audio is re-anchored to the video timeline at every creative boundary (including amux creatives that mux audio on a separate source clock) and an OutputTimestampSanitizer keeps the stream monotonic across the splice. A no-cut stall watchdog sits underneath as a safety net: it tells a genuinely wedged pod (reading at full rate but unable to cut) from a slow source (a trickle) by read rate, and a wedge-classified stall whose video PTS is still advancing (a source delivering just below real time, #177) is held with the watchdog re-armed (bounded consecutive holds) instead of retuned; only a genuine wedge or an exhausted hold budget escalates to a host retune.
The live path's sliding-window eviction (which bounds resident memory) and DVR rewind are confirmed on Apple TV against a real broadcast feed: behindLiveSeconds holds at real-time pacing and the resident footprint stays bounded within the tvOS jetsam budget. The same behavior is exercised off-device through the aetherctl live / hlsfixture harnesses (sliding-window retention, real-time pacing, mid-stream reconnect, program-boundary discontinuities, DVR timeshift).
Raw live MPEG-TS over HTTP (a tuner, a tuner proxy, or an IPTV panel's continuous form of a channel) is routed like any live source: the native loopback path unless the codec, declared interlace or a host override sends it to the software path. Such a stream opens with whatever the origin holds of the channel, sent at link speed (8 to 18 s on one IPTV panel), and a software session no longer starts on the first packet of that burst, where it would run the whole burst behind the broadcast for as long as it played: once the burst has landed the live feeder places the start SoftwareLiveJoinPolicy.leadSeconds (4 s) behind the newest media, on a keyframe, and the burst stays in the rewind ring (the native path's AVPlayer already sits a holdback behind the newest segment). A live HLS ingest on the software path delivers a whole segment at a time instead, so its holdback follows the segment length (SoftwareLiveHoldback, AE#813): the join starts 1.5 x segment + 1 s behind the newest media (4 s on 2 s segments, 16 s on 10 s segments, at most 45 s), a rebuffer resumes only once one segment plus 2 s stands ahead of the clock, Return to Live lands as far back as a join, and the live-edge verdict counts that distance plus one segment as the edge (the next segment can land right after a placement). Without it a 4 s lead ran dry before every 10 s segment and the session stalled once per segment. Load it with isLive: true plus a dvrWindowSeconds for live semantics and the fastest open (an explicitly live load skips the open-time size probes entirely); SourceProbe.isLive flags no-duration network streams so hosts can detect and reload. Mid-stream-joined sources, which deliver their first samples at arbitrary PTS hours past zero, anchor the clock at the first decoded sample on every session shape (live, live+DVR, and a plain VOD open of the same URL or of a mid-broadcast capture file), publishing session-relative positions while sourceTime stays on the source axis for subtitles (#107).
Things that work today but have a documented edge case, or are deferred behind an upstream dependency:
- The channel bridge drops TrueHD Atmos objects and DTS:X heights. TrueHD / MLP and DTS-HD sources route through the AudioBridge (FFmpeg's EAC3 encoder doesn't produce JOC, and FLAC has no object channels): bed channels and surround layout survive, objects and heights do not.
LoadOptions.objectAudioRendering = .apac(layout)keeps them in a build with theTrueHDObjectAudio/DTSXObjectAudiopackage traits (off by default), README › Optional components), rendered into a speaker bed and delivered as lossy APAC that tvOS sends to an Atmos receiver as Dolby MAT (TrueHD Atmos with heights, DTS:X with heights). EAC3+JOC stream-copy from MKV / MP4 sources is intact either way. .surroundCompataudio bridge caps 7.1 sources to 5.1. FFmpeg's EAC3 encoder currently caps at 6 channels. Once FFmpeg PR 21668 lands the cap and the dynamic bitrate auto-scale to 1024 kbps engage without a code change here. Use.lossless(FLAC) today if 7.1 matters.- Manual
MPNowPlayingInfoCenterwrites race the HLS-loopback path on tvOS 26. The combination produces alibdispatchrace.AVPlayerViewControllerwith its standard transport bar surfaces Now Playing on its own, through AVKit's private MediaRemote registration, and needs nothing from the host. A host with a custom transport instead setsengine.ownsVideoNowPlayingSession = truebeforeload(): the native host then owns anMPNowPlayingSessionbound to its player (published asvideoNowPlayingSession), registers transport commands on that session'sremoteCommandCenter, and stages identity metadata throughsetVideoNowPlayingInfo(_:), which is replayed onto every freshAVPlayerItem. The session auto-publishes elapsed time, rate and duration from the player, so nothing writesMPNowPlayingInfoCenter.default().nowPlayingInfo. The flag is off by default and must stay off under AVKit: two owners produce the half-working state where an empty identity card displaces AVKit's and remote commands route into a session with no handlers. - Audio session is activated per playback, not at process launch. The engine declares the
AVAudioSessioncategory (.playback/.moviePlayback) and multichannel support at init, but does NOT activate the session there. The route-sharing policy is platform-split (#116): tvOS declares.longFormAudio, iOS declares.default, because.longFormAudiomarks the process as a long-form audio client and pinsAVPictureInPictureController.isPictureInPicturePossibleto false for any host-built PiP controller around the engine's player layer. Hosts do not need to re-declare the session for PiP. Activating once at launch used to pin the route to whatever the HDMI link reported at that instant: with tvOS Continuous Audio Connection off the link idles at stereo, so the launch-time activation negotiated the route to 2 channels and pinned it, downmixing non-Atmos multichannel for the whole session (AetherEngine#24). The native video path now lets the host'sAVPlayerViewControlleractivate the session per playback; the renderer paths (software decode, audio-only) activate it themselves, off the main actor and before their host is built (AE#538). Hosts that mount the engine's bareAVPlayerLayerinstead of anAVPlayerViewControllershould ensure the session is active at playback. A genuine sink-side ch=2 (an AVR caching its HDMI EDID incorrectly after standby) can still force a downmix; power-cycling the sink restores it. Atmos passthrough is unaffected either way because EAC3+JOC ships as MAT 2.0 over a 2-channel carrier. - AV1 on Apple TV is software-decoded. No current Apple TV chip ships HW AV1. The
SoftwarePlaybackHost+ dav1d path handles it, but CPU use is meaningfully higher than HW HEVC. On iOS 17+ / macOS 14+ AV1 routes through Apple's HW pipeline transparently. Future Apple TV chips with HW AV1 will be picked up automatically byVTCapabilityProbe. - AV1 Dolby Vision Profile 10.0 has wrong colours when software-decoded. dav1d / libavcodec cannot decode the proprietary DV colour space, so a Profile 10.0 source (DV-only, no fallback base layer) renders with incorrect colours on the SW path. Profiles 10.1 and 10.4 are unaffected because they carry an HDR10 / HLG base layer. Profile 10.0 only renders correctly through the native AVPlayer path on hosts with HW AV1 decode.
- Dolby Vision Profile 5 / AV1 Profile 10.0 thumbnails skip the RPU reshaping curves.
FrameExtractornow applies the DV colour transform (fromAV_FRAME_DATA_DOVI_METADATA) so P5 / P10.0 stills come out with correct colour, validated against a libplacebo render. The per-frame reshaping polynomials are intentionally not applied: they are not what causes the visible green / magenta corruption, and skipping them keeps this a lightweight CPU pass rather than a full Dolby Vision compositor. Brightness / contrast can therefore differ marginally from a fully graded DV render. A frame that carries no DV metadata falls back to the standard path.