Skip to content

fix(container): fall back to the GitHub mirror for nv-codec-headers (ADR-1200) - #1348

Merged
lusoris merged 1 commit into
masterfrom
fix/container-nv-codec-mirror-fallback
Sep 6, 2026
Merged

lusoris merged 1 commit into
masterfrom
fix/container-nv-codec-mirror-fallback

Conversation

@lusoris

@lusoris lusoris commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Summary

dev/Containerfile fetched nv-codec-headers from a single host, code.ffmpeg.org. That host has been unreachable for over six hours today — curl returning HTTP 000 after a 30-second timeout on every attempt — and the layer simply stalled. Under CLAUDE.md rule 15 and ADR-1102 the dev container is the canonical build and measurement environment, so for that whole window it could not be built at all, with no symptom beyond an apparent hang.

The stated reason for the single source does not apply to the pin in use. The comment read "GitHub mirror lags so use code.ffmpeg.org" — true of an unreleased commit, false of a tag. NV_CODEC_HEADERS_REF is n13.1.15.0, that tag is published on both hosts, and the GitHub tarball carries the cuStreamCreateWithPriority declaration in include/ffnvcodec/dynlink_loader.h that the pin exists for. Checked, not assumed.

What changes

  • Upstream stays first; GitHub runs only when it fails outright.
  • The archive is asserted, not trusted: the build requires include/ffnvcodec/dynlink_cuda.h and greps dynlink_loader.h for cuStreamCreateWithPriority before make install. A fallback that silently installs the wrong headers would be worse than the outage it replaces.
  • The two archives unpack to different top-level directory names (nv-codec-headers vs nv-codec-headers-<tag>), so the build cds into whatever was extracted rather than the hard-coded name that would have broken on the mirror.
  • The primary gets a short leash. Measured: the first draft's --retry 3 --max-time 120 took 487 s to fall through and succeed; --retry 1 --max-time 60 takes 123 s. The healthy path is one 0.5 s fetch either way.

Type

  • build / ci — tooling / infra

Checklist

  • Commits follow Conventional Commits.
  • make format && make lint green locally — pre-commit run --files clean on all eight touched files.
  • Unit tests pass: meson test -C build. — no library code is touched; the change is a container build layer plus documentation.
  • SIMD/GPU code path — none touched.
  • Feature extractor twins — none touched.
  • New .c / .cpp / .cu / .h — none added.
  • Breaking change — no. Same package, same tag; a second source is consulted only when the first fails.
  • ADR row lives in docs/adr/_index_fragments/ with the slug in _order.txt; README.md regenerated with --write and verified with --check.

Bug-status hygiene (ADR-0165)

  • docs/state.md updated — no state delta: this is an upstream host outage, not a tracked defect in this codebase.

Netflix golden-data gate (ADR-0024)

  • I did not modify any assertAlmostEqual(...) score in the Netflix golden Python tests.

Deep-dive deliverables (ADR-0108)

  • Research digest — no digest needed: trivial. One unreachable host, one tag verified present on the mirror.
  • Decision matrix — ADR-1200 ## Alternatives considered weighs five options, including switching to GitHub outright and vendoring the 86 KB of headers.
  • AGENTS.md invariant note — recorded in docs/rebase-notes.md, where this repo keeps container-level invariants: do not collapse the fallback back to one curl, and keep both the content assertion and the find-based cd.
  • Reproducer / smoke-test command — below.
  • CHANGELOG fragment — changelog.d/fixed/nv-codec-headers-mirror-fallback.md.
  • Rebase note — fix/container-nv-codec-mirror-fallback — second source for nv-codec-headers (2026-09-06).

Reproducer

The outage itself, still live at the time of writing:

curl -sS -o /dev/null -w "%{http_code} in %{time_total}s\n" --max-time 30 \
  https://code.ffmpeg.org/FFmpeg/nv-codec-headers/archive/n13.1.15.0.tar.gz
# 000 in 30.004s

The fallback logic exactly as committed:

time { curl -fsSL --retry 1 --connect-timeout 10 --max-time 60 \
        https://code.ffmpeg.org/FFmpeg/nv-codec-headers/archive/n13.1.15.0.tar.gz -o nvh.tgz \
      || curl -fsSL --retry 3 --connect-timeout 20 --max-time 120 \
        https://github.com/FFmpeg/nv-codec-headers/archive/refs/tags/n13.1.15.0.tar.gz -o nvh.tgz; }
tar tzf nvh.tgz | head -1     # nv-codec-headers-n13.1.15.0/

Observed: 123 s total, 86247 bytes, and the tag's tree contains cuStreamCreateWithPriority.

End to end, with the primary still down:

docker build --file dev/Containerfile --target libvmaf-build -t nvmirror-test .

Observed: nv-codec-headers: code.ffmpeg.org unreachable, falling back to the GitHub mirror, then make: Leaving directory '/tmp/nv-codec-headers-n13.1.15.0', #41 DONE, and the next meson step reporting Has header "ffnvcodec/dynlink_cuda.h" : YES. The build then proceeded into the CUDA targets that had been unreachable all day.

Known follow-ups

  • The other fetches in dev/Containerfile remain single-sourced — oneAPI, ROCm, ONNX Runtime, SVT-AV1, vvenc, AMF, FFmpeg. This PR does not claim to have audited them; it fixes the one that actually failed. A build cache or internal proxy would address the class, and is the right shape at a larger scale.

@lusoris lusoris added this to the 1.0.0 — First release milestone Sep 6, 2026
@lusoris
lusoris force-pushed the fix/container-nv-codec-mirror-fallback branch from db37050 to 9062ed3 Compare September 6, 2026 12:54
lusoris added a commit that referenced this pull request Sep 6, 2026
… run (#1349)

The Containerfile's own layout comment has always described compat/ as
"pip-installed via the python/ shim", and the Netflix golden YUV fetch exists,
in its own words, "so that `pytest python/test/` passes inside the container
without any host-side manual provisioning". Nothing ever performed that
install, so the command died at import with ModuleNotFoundError: No module
named 'slugify', and the fixture fetch served no reachable purpose. Found while
running the epic #1246 GPU smoke, which had to fall back to the CLI because the
suite could not be collected.

Installs python/ (package name vmaf, re-exporting compat/python-vmaf per
ADR-0700) into /opt/vmaf-venv, first among the editable packages so the ai/ and
vmaf-tune pins resolve last and win on shared dependencies.

Adds a build-time assertion that the harness imports and the golden suite
collects, because the failure mode is silent: the build stayed green for a long
time while the command those fixtures exist for could not run at all.
--collect-only keeps it to an import plus collection check; the suite itself is
a CI gate, not a container build step. The interpreter is spelled
/opt/vmaf-venv/bin/python explicitly because ENV PATH is set below this layer,
so a bare python3 here would be the system interpreter.

VERIFIED by a full dev-mcp build:
  #72 harness import OK: /build/vmaf/compat/python-vmaf/__init__.py
  #72 62 tests collected in 0.89s
  #73 naming to docker.io/library/vmaf-harness-test:local

That build needed the nv-codec-headers mirror fallback from #1348 applied
locally, because code.ffmpeg.org has been unreachable all day; the fallback was
NOT committed here and is not part of this change. Once #1348 lands this branch
builds on its own.

no docs needed: dev/Containerfile is build infrastructure; the user-visible
effect is that a documented command now works, and docs/development/dev-mcp.md
already tells operators to run the suite in the container

no state delta: closes no tracked bug id -- the gap was between the
Containerfile's stated intent and its contents, not a defect in shipped code

Co-authored-by: Lusoris <lusoris@pm.me>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@lusoris
lusoris marked this pull request as ready for review September 6, 2026 14:55
@lusoris
lusoris force-pushed the fix/container-nv-codec-mirror-fallback branch from 9062ed3 to b9cd0a1 Compare September 6, 2026 14:55
…ADR-1200)

dev/Containerfile fetched nv-codec-headers from a single host,
code.ffmpeg.org. On 2026-09-06 that host was unreachable for over six hours --
curl returning HTTP 000 after a 30 s timeout on every attempt -- and the layer
stalled with no symptom beyond an apparent hang. The dev container is the
canonical build and measurement environment under CLAUDE.md rule 15 and, since
ADR-1102, the environment published artifacts come from. It could not be built
at all for that window.

The stated reason for the single source does not apply to the pin in use. The
comment said "GitHub mirror lags so use code.ffmpeg.org", which is true of an
unreleased commit but not of a TAG: NV_CODEC_HEADERS_REF is n13.1.15.0, that
tag is published on both hosts, and the GitHub tarball carries the
cuStreamCreateWithPriority declaration in include/ffnvcodec/dynlink_loader.h
that the pin exists for -- checked, not assumed.

Upstream stays first; GitHub runs only when it fails outright. Whichever archive
arrives is then asserted rather than trusted: the build requires
include/ffnvcodec/dynlink_cuda.h to exist and greps dynlink_loader.h for
cuStreamCreateWithPriority before make install, because a fallback that silently
installs the wrong headers would be worse than the outage it replaces. The two
archives also unpack to different top-level directory names, so the build cds
into whatever was extracted instead of the hard-coded `nv-codec-headers`.

Verified end to end while code.ffmpeg.org was still down: the layer fell through
to GitHub, installed the headers, and the next meson step reported
`Has header "ffnvcodec/dynlink_cuda.h" : YES`. The build then proceeded into the
CUDA targets that had been unreachable all day.

The primary gets a short leash (--retry 1 --connect-timeout 10 --max-time 60).
With a working fallback behind it, minutes on a dead host are pure latency:
measured, the first draft's --retry 3 --max-time 120 took 487 s to fall through
and succeed, the tightened values take 123 s, and the healthy path is a single
0.5 s fetch either way.

no docs needed: dev/Containerfile is build infrastructure with no
user-discoverable CLI, API or output surface; the operator-visible rationale is
in ADR-1200 and docs/rebase-notes.md

no state delta: an upstream host outage, not a tracked defect in this codebase

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@lusoris
lusoris force-pushed the fix/container-nv-codec-mirror-fallback branch from b9cd0a1 to dc88b2f Compare September 6, 2026 14:58
@lusoris
lusoris merged commit defb879 into master Sep 6, 2026
72 checks passed
@lusoris
lusoris deleted the fix/container-nv-codec-mirror-fallback branch September 6, 2026 15:54
@lusoris lusoris added the type:bug Something isn't working label Sep 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type:bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant