Skip to content

fix(audio): rebuild timestamps between apad and atrim so FFmpeg 7.x keeps every mixed clip - #3348

Open
miguel-heygen wants to merge 1 commit into
mainfrom
fix/3344-audio-pad-timestamps
Open

fix(audio): rebuild timestamps between apad and atrim so FFmpeg 7.x keeps every mixed clip#3348
miguel-heygen wants to merge 1 commit into
mainfrom
fix/3344-audio-pad-timestamps

Conversation

@miguel-heygen

Copy link
Copy Markdown
Collaborator

What

apad,atrim=0:<total>, an indefinite pad bounded by a downstream trim, is replaced everywhere with apad,asetpts=N/SR/TB,atrim=0:<total>, and the call sites that rebuilt that chain by hand now share one helper. Adds a CI lane pinned to FFmpeg 7.x, because no existing lane can catch the regression.

Closes #3344.

Why

On the FFmpeg 7.x line that pairing misbehaves. Audio leaks to t=0 from three mixed branches onward, and from four branches onward the branch with the largest adelay disappears from the mix entirely. Nothing errors, so the render succeeds with wrong audio.

I measured the graph across six builds on linux/amd64 rather than trusting the version range in the report:

FFmpeg apad,atrim apad,asetpts=N/SR/TB,atrim
4.2.7 correct correct, identical output
6.0.1 correct correct, identical output
7.0.2 drops the last-delayed branch correct
7.1.3 drops the last-delayed branch correct
git master, 2026-05 correct correct
8.1.1 correct correct

So the affected window is 7.x, not "7 and newer", and on every unaffected build the new chain is byte-identical to the old one. This is a no-op everywhere except where it is a fix. It stays in place on the unaffected versions because the next release to regress here cannot be predicted, and the cost is one filter.

Users are exposed because findFfBinary falls through to PATH. ffmpeg-static is only a dependency of packages/aws-lambda, so for everyone else the FFmpeg that runs is whatever their system ships.

How

Not apad=whole_dur=<total>, for two reasons, both checked rather than assumed.

The bundled Windows builds reject the option, which this repo already pinned in an assertion and an error-classification fixture (Error applying option 'whole_dur': Option not found). Using it would fix 7.x by breaking the builds those assertions are about.

It also drops a bound the current chain provides. audioMixer deliberately lets an FX tail run past its clip and relies on the trailing atrim to stop it at the composition's end. With -t removed so the graph speaks for itself, one branch delayed 42s with a 5s tail against a 43.3s composition:

branch template output duration
apad,atrim=0:43.3 43.300000
apad=whole_dur=43.3 46.999977
apad,asetpts=N/SR/TB,atrim=0:43.3 43.300000

Same on every version in the table above. Today -t totalDuration hides that at the mix call sites, but it is still a bound moving quietly from the filter graph to the muxer.

What actually goes wrong is narrower than "apad plus atrim". Any atrim reading apad's output timestamps directly is affected. Rebuilding the timestamps from the running sample count between the two is what fixes it, and the result is then identical to whole_dur at every clip count measured, while keeping both the atrim bound and the Windows builds.

One place, not several. buildPadToDurationFilter lives in packages/engine/src/services/audioPadFilter.ts and is exported from @hyperframes/engine. It takes an already-formatted seconds string on purpose, because the call sites format numbers differently and the helper must not silently change any of them.

audioPadTrim.ts is a single-input -af pad with no adelay, and I could not make it misbehave on any version tested. It is changed anyway so the template has one owner, and the pad and trim branches now each say why they use the asetpts expression they use.

Test plan

  • Unit tests added/updated
  • Manual testing performed
  • Documentation updated (if applicable)

New audioPadFilter.test.ts. Pins the chain and its order, since order is the behaviour here, and keeps the whole_dur prohibition asserted.

New audioMixer.padTimestamps.integration.test.ts. Five clips at staggered offsets through the real processCompositionAudio and real FFmpeg, asserting t=0 is silent and every clip is audible at its own offset. Three or fewer clips passes on a broken build, which is why it mixes five. It resolves FFmpeg through getFfmpegBinary() so the skip gate and the code under test cannot disagree under HYPERFRAMES_FFMPEG_PATH, which is how the lane below points it at a specific build.

New audio-pad-ffmpeg7 CI lane. Without this the regression test guards nothing. Test has no FFmpeg at all, and its gated audio tests already report as skipped on main:

↓ src/services/audioMixer.level.test.ts (2 tests | 2 skipped)

Installing FFmpeg there would not have been enough either. The runner image's apt-get install ffmpeg is the 6.x line and the Windows lane pins an 8.x master build, and I confirmed the regression test passes on 6.0.1 with the fix reverted. So the lane pins a 7.1.3 build from the same source and with the same bump rule as the existing Windows action, asserts the binary really is 7.x rather than skipping quietly, and is wired into Test so it fails closed.

Verified on linux/amd64 against that exact pinned asset, driven the way CI drives it with only the two environment variables set:

with fix:    1 passed
without fix: AssertionError: expected -24.898801 to be less than -60

Suites. Engine 1606 passed, 3 skipped, exit 0. Producer unit lane exit 0. oxlint, oxfmt --check and typecheck clean.

Not run: the producer integration lane and any browser or render end-to-end. audioExtractor.ts has no test of its own, before or after, so that hunk rests on being the same template as the mixer's rather than on a test. It is also currently unreferenced, which the helper's comment now says instead of counting it as a live consumer.

…eeps every mixed clip

`apad,atrim=0:<total>` is an indefinite pad bounded by a downstream trim. On
the FFmpeg 7.x line that pairing misbehaves: audio leaks to `t=0` from three
mixed branches onward, and from four branches onward the branch with the
largest `adelay` disappears from the mix entirely. Nothing errors, so the
render succeeds with wrong audio.

`asetpts=N/SR/TB` between the two rebuilds each frame's timestamp from the
running sample count, so `atrim` sees a monotonic sample-accurate timeline
instead of whatever the release propagates through an indefinite `apad`.

Measured on linux/amd64 across 4.2.7, 6.0.1, 7.0.2, 7.1.3, git master from
2026-05 and 8.1.1: only the 7.x builds are affected, and on every unaffected
build the new chain is byte-identical to the old one. So this is a no-op
everywhere except where it is a fix.

The suggested `apad=whole_dur=<total>` is not used. It is rejected by the
bundled Windows FFmpeg builds, and it does not bound a branch already longer
than the target: with `-t` removed so the graph speaks for itself, a branch
delayed 42s with a 5s tail against a 43.3s composition yields 46.999977s under
`whole_dur` and 43.300000s under both the old chain and this one. `audioMixer`
deliberately lets an FX tail overrun its clip and relies on that trailing
`atrim` to cut it.

The call sites that rebuilt this chain by hand now share
`buildPadToDurationFilter`.

Adds a pinned-FFmpeg CI lane, because no existing lane can catch this. `Test`
has no FFmpeg at all, `apt-get install ffmpeg` on the runner image is the 6.x
line, and the Windows lane pins an 8.x master build. All of those pass the
regression test with the bug present.

Closes #3344
@sujeito-operator

Copy link
Copy Markdown

Author of #3346 here. Good call on the pinned 7.x lane — that was the piece I left on the table, and asserting the binary really is 7.x rather than skipping quietly is the right shape for it.

Three things from that branch that aren't in this one. All free, take or leave any of them individually; say the word and I'll open a PR against your branch rather than paste diffs here.

1. The FX-tail bound is the argument against whole_dur, and no test pins it

audioPadFilter.test.ts asserts the prohibition (not.toContain("whole_dur")), but nothing measures the bound behind it. The 46.999977 vs 43.300000 table is in the PR body and dies with the PR body.

The catch is that it can't go through processCompositionAudio: the mixer passes -t totalDuration on the output, so a duration assertion there passes with whole_dur substituted in, with a bare apad, and with no pad filter at all. The container does the work and the test proves nothing about the chain. So it has to drive the graph directly — one 5s source delayed 42s against a 43.3s target, -t removed, ffprobe the duration, assert whole_dur overruns and the shipped chain does not.

Proved non-vacuous rather than asserted — point buildPadToDurationFilter at whole_dur and it fails with:

expected 46.999977 to be close to 43.3, received difference is 3.699977

2. Nothing stops a fourth call site rebuilding the chain by hand

Which is the case that produced this bug: three sites diverged because each owned its own copy of the string. The helper ends that only for as long as nobody writes it out again, and a helper is not a guard.

A source sweep in the shape ffprobeArgvContract.test.ts already uses — discovery for new offenders, a manifest so the sweep can't silently vacuum — asserts that after stripping comments the literal apad appears in audioPadFilter.ts and nowhere else, and that each pad site calls the helper. Four cases, each proved by mutation: hand-rebuild the chain in audioMixer.ts → 2 fail; helper returns whole_dur → 1 fails; narrow the sweep roots → 1 fails.

This also covers mixTracks, which is the gap @miga-heygen flagged on #3346. Worth knowing why I went this way rather than unit-testing mixTracks directly: packages/producer/src/**/*.test.ts is a fallow entry, so a test importing audioExtractor.ts makes it reachable and its fallow-ignore-file unused-file goes stale — the audit then reports parseAudioElements and AudioTrack as unused exports, with no file-level suppression available for either kind. The test would have become the only thing keeping a dead file alive in the import graph.

3. One more row for your matrix, and the 7 line runs further than 7.1.3

Same five-branch graph, builds pulled from the same source as the Windows action:

FFmpeg apad,atrim this chain
7.0.2 static leak at t=0, last-delayed branch dropped correct
7.1.5 (n7.1.5-16-g9a4bb2c579) leak at t=0, last-delayed branch dropped correct
8.1.2 (n8.1.2-44-g7c533d0f86) correct correct
9.0.1 (n9.0.1-6-g9d4ca21220) correct correct

7.1.5 is one release past the 7.1.3 you pinned, so the whole line is affected and pinning the tail of it works too. 9.0 wasn't in either of our tables and it's clean, which makes the window 7.x on both ends rather than 7.x-and-then-we-stopped-looking.

The tail-bound numbers are identical on all four (whole_dur 46.999977, both atrim chains 43.300000), so that argument doesn't depend on the version.


One housekeeping note: you mentioned attributing the fix, and there's no Co-authored-by: trailer on 8dbe0e3 — flagging it once in case it was just missed in the handover. Either way the three offers above stand.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Audio mix drops and misplaces clips on FFmpeg 7+ (apad,atrim branch template)

2 participants