fix(audio): rebuild timestamps between apad and atrim so FFmpeg 7.x keeps every mixed clip - #3348
fix(audio): rebuild timestamps between apad and atrim so FFmpeg 7.x keeps every mixed clip#3348miguel-heygen wants to merge 1 commit into
Conversation
…eeps every mixed clip `apad,atrim=0:<total>` is an indefinite pad bounded by a downstream trim. On the FFmpeg 7.x line that pairing misbehaves: audio leaks to `t=0` from three mixed branches onward, and from four branches onward the branch with the largest `adelay` disappears from the mix entirely. Nothing errors, so the render succeeds with wrong audio. `asetpts=N/SR/TB` between the two rebuilds each frame's timestamp from the running sample count, so `atrim` sees a monotonic sample-accurate timeline instead of whatever the release propagates through an indefinite `apad`. Measured on linux/amd64 across 4.2.7, 6.0.1, 7.0.2, 7.1.3, git master from 2026-05 and 8.1.1: only the 7.x builds are affected, and on every unaffected build the new chain is byte-identical to the old one. So this is a no-op everywhere except where it is a fix. The suggested `apad=whole_dur=<total>` is not used. It is rejected by the bundled Windows FFmpeg builds, and it does not bound a branch already longer than the target: with `-t` removed so the graph speaks for itself, a branch delayed 42s with a 5s tail against a 43.3s composition yields 46.999977s under `whole_dur` and 43.300000s under both the old chain and this one. `audioMixer` deliberately lets an FX tail overrun its clip and relies on that trailing `atrim` to cut it. The call sites that rebuilt this chain by hand now share `buildPadToDurationFilter`. Adds a pinned-FFmpeg CI lane, because no existing lane can catch this. `Test` has no FFmpeg at all, `apt-get install ffmpeg` on the runner image is the 6.x line, and the Windows lane pins an 8.x master build. All of those pass the regression test with the bug present. Closes #3344
|
Author of #3346 here. Good call on the pinned 7.x lane — that was the piece I left on the table, and asserting the binary really is 7.x rather than skipping quietly is the right shape for it. Three things from that branch that aren't in this one. All free, take or leave any of them individually; say the word and I'll open a PR against your branch rather than paste diffs here. 1. The FX-tail bound is the argument against
|
| FFmpeg | apad,atrim |
this chain |
|---|---|---|
| 7.0.2 static | leak at t=0, last-delayed branch dropped |
correct |
| 7.1.5 (n7.1.5-16-g9a4bb2c579) | leak at t=0, last-delayed branch dropped |
correct |
| 8.1.2 (n8.1.2-44-g7c533d0f86) | correct | correct |
| 9.0.1 (n9.0.1-6-g9d4ca21220) | correct | correct |
7.1.5 is one release past the 7.1.3 you pinned, so the whole line is affected and pinning the tail of it works too. 9.0 wasn't in either of our tables and it's clean, which makes the window 7.x on both ends rather than 7.x-and-then-we-stopped-looking.
The tail-bound numbers are identical on all four (whole_dur 46.999977, both atrim chains 43.300000), so that argument doesn't depend on the version.
One housekeeping note: you mentioned attributing the fix, and there's no Co-authored-by: trailer on 8dbe0e3 — flagging it once in case it was just missed in the handover. Either way the three offers above stand.
What
apad,atrim=0:<total>, an indefinite pad bounded by a downstream trim, is replaced everywhere withapad,asetpts=N/SR/TB,atrim=0:<total>, and the call sites that rebuilt that chain by hand now share one helper. Adds a CI lane pinned to FFmpeg 7.x, because no existing lane can catch the regression.Closes #3344.
Why
On the FFmpeg 7.x line that pairing misbehaves. Audio leaks to
t=0from three mixed branches onward, and from four branches onward the branch with the largestadelaydisappears from the mix entirely. Nothing errors, so the render succeeds with wrong audio.I measured the graph across six builds on linux/amd64 rather than trusting the version range in the report:
apad,atrimapad,asetpts=N/SR/TB,atrimSo the affected window is 7.x, not "7 and newer", and on every unaffected build the new chain is byte-identical to the old one. This is a no-op everywhere except where it is a fix. It stays in place on the unaffected versions because the next release to regress here cannot be predicted, and the cost is one filter.
Users are exposed because
findFfBinaryfalls through toPATH.ffmpeg-staticis only a dependency ofpackages/aws-lambda, so for everyone else the FFmpeg that runs is whatever their system ships.How
Not
apad=whole_dur=<total>, for two reasons, both checked rather than assumed.The bundled Windows builds reject the option, which this repo already pinned in an assertion and an error-classification fixture (
Error applying option 'whole_dur': Option not found). Using it would fix 7.x by breaking the builds those assertions are about.It also drops a bound the current chain provides.
audioMixerdeliberately lets an FX tail run past its clip and relies on the trailingatrimto stop it at the composition's end. With-tremoved so the graph speaks for itself, one branch delayed 42s with a 5s tail against a 43.3s composition:apad,atrim=0:43.3apad=whole_dur=43.3apad,asetpts=N/SR/TB,atrim=0:43.3Same on every version in the table above. Today
-t totalDurationhides that at the mix call sites, but it is still a bound moving quietly from the filter graph to the muxer.What actually goes wrong is narrower than "
apadplusatrim". Anyatrimreadingapad's output timestamps directly is affected. Rebuilding the timestamps from the running sample count between the two is what fixes it, and the result is then identical towhole_durat every clip count measured, while keeping both theatrimbound and the Windows builds.One place, not several.
buildPadToDurationFilterlives inpackages/engine/src/services/audioPadFilter.tsand is exported from@hyperframes/engine. It takes an already-formatted seconds string on purpose, because the call sites format numbers differently and the helper must not silently change any of them.audioPadTrim.tsis a single-input-afpad with noadelay, and I could not make it misbehave on any version tested. It is changed anyway so the template has one owner, and the pad and trim branches now each say why they use theasetptsexpression they use.Test plan
New
audioPadFilter.test.ts. Pins the chain and its order, since order is the behaviour here, and keeps thewhole_durprohibition asserted.New
audioMixer.padTimestamps.integration.test.ts. Five clips at staggered offsets through the realprocessCompositionAudioand real FFmpeg, assertingt=0is silent and every clip is audible at its own offset. Three or fewer clips passes on a broken build, which is why it mixes five. It resolves FFmpeg throughgetFfmpegBinary()so the skip gate and the code under test cannot disagree underHYPERFRAMES_FFMPEG_PATH, which is how the lane below points it at a specific build.New
audio-pad-ffmpeg7CI lane. Without this the regression test guards nothing.Testhas no FFmpeg at all, and its gated audio tests already report as skipped onmain:Installing FFmpeg there would not have been enough either. The runner image's
apt-get install ffmpegis the 6.x line and the Windows lane pins an 8.x master build, and I confirmed the regression test passes on 6.0.1 with the fix reverted. So the lane pins a 7.1.3 build from the same source and with the same bump rule as the existing Windows action, asserts the binary really is 7.x rather than skipping quietly, and is wired intoTestso it fails closed.Verified on linux/amd64 against that exact pinned asset, driven the way CI drives it with only the two environment variables set:
Suites. Engine 1606 passed, 3 skipped, exit 0. Producer unit lane exit 0.
oxlint,oxfmt --checkandtypecheckclean.Not run: the producer integration lane and any browser or render end-to-end.
audioExtractor.tshas no test of its own, before or after, so that hunk rests on being the same template as the mixer's rather than on a test. It is also currently unreferenced, which the helper's comment now says instead of counting it as a live consumer.