lusoris · Sep 6, 2026 · Sep 6, 2026
@@ -23708,6 +23708,16 @@ results") and updated `BISECT_TRACKER_ISSUE` to `"827"`. Also improved
 making future API errors diagnosable in CI logs.
 
 
+- The dev container no longer becomes unbuildable when `code.ffmpeg.org` is down. The
+  nv-codec-headers layer now falls back to `github.com/FFmpeg/nv-codec-headers` for the same
+  pinned tag, and asserts the archive actually carries `ffnvcodec/dynlink_cuda.h` and the
+  `cuStreamCreateWithPriority` declaration before installing it, so a fallback cannot
+  silently install the wrong headers. That host was unreachable for over six hours on
+  2026-09-06, stalling the layer with no symptom beyond an apparent hang — and under
+  CLAUDE.md rule 15 and ADR-1102 the container is the canonical build environment. See
+  [ADR-1200](docs/adr/1200-nv-codec-headers-mirror-fallback.md).
+
+
 **fix(build): error on `enable_nvtx=true` without `enable_cuda=true` (build-matrix audit §1a)**
 
 `meson setup` previously hard-errored with an opaque "Include dir does not

@@ -0,0 +1,8 @@
+- The dev container no longer becomes unbuildable when `code.ffmpeg.org` is down. The
+  nv-codec-headers layer now falls back to `github.com/FFmpeg/nv-codec-headers` for the same
+  pinned tag, and asserts the archive actually carries `ffnvcodec/dynlink_cuda.h` and the
+  `cuStreamCreateWithPriority` declaration before installing it, so a fallback cannot
+  silently install the wrong headers. That host was unreachable for over six hours on
+  2026-09-06, stalling the layer with no symptom beyond an apparent hang — and under
+  CLAUDE.md rule 15 and ADR-1102 the container is the canonical build environment. See
+  [ADR-1200](docs/adr/1200-nv-codec-headers-mirror-fallback.md).
@@ -529,14 +529,47 @@ USER vmaf
 # -Denable_metal=auto  → auto-disabled on Linux (no Metal SDK present).
 # -Denable_mcp_sse=auto → auto-disabled until POSIX socket impl is complete.
 USER root
-# Install nv-codec-headers (provides ffnvcodec/dynlink_cuda.h for CUDA backend)
-# Pinned to commit that has cuStreamCreateWithPriority; GitHub mirror lags so use code.ffmpeg.org.
+# Install nv-codec-headers (provides ffnvcodec/dynlink_cuda.h for CUDA backend).
+# Pinned to a tag that has cuStreamCreateWithPriority.
+#
+# Two sources, tried in order. code.ffmpeg.org is upstream's own host and stays
+# first; github.com/FFmpeg/nv-codec-headers is the fallback. The original comment
+# here said "GitHub mirror lags so use code.ffmpeg.org", which is true for an
+# unreleased commit but not for a TAG: n13.1.15.0 is published on both, and the
+# GitHub tarball carries the cuStreamCreateWithPriority declaration this pin
+# exists for (verified in include/ffnvcodec/dynlink_loader.h).
+#
+# The fallback is not hypothetical tidiness. On 2026-09-06 code.ffmpeg.org was
+# unreachable for over six hours -- `curl` returning HTTP 000 after a 30 s
+# timeout -- which stalled this layer and made the dev container, the canonical
+# build environment under CLAUDE.md rule 15, unbuildable for that whole window.
+# A single unreachable host should not be able to do that.
+#
+# The primary is given a deliberately short leash (--retry 1, --max-time 60):
+# with a working fallback behind it, spending minutes on an unreachable host is
+# pure latency. Measured with the first draft's generous settings (--retry 3,
+# --max-time 120), the layer took 487 s to fall through to GitHub and succeed;
+# the tightened values cut that to roughly two minutes while leaving the healthy
+# path -- one 0.5 s fetch -- untouched.
+#
+# Note the two archives unpack to DIFFERENT top-level directory names
+# (`nv-codec-headers` from code.ffmpeg.org, `nv-codec-headers-<tag>` from
+# GitHub), so the build cds into whatever was extracted rather than a fixed name.
 ARG NV_CODEC_HEADERS_REF=n13.1.15.0
 RUN cd /tmp \
-    && curl -fsSL "https://code.ffmpeg.org/FFmpeg/nv-codec-headers/archive/${NV_CODEC_HEADERS_REF}.tar.gz" -o nv-codec-headers.tgz \
+    && { curl -fsSL --retry 1 --retry-delay 2 --connect-timeout 10 --max-time 60 \
+           "https://code.ffmpeg.org/FFmpeg/nv-codec-headers/archive/${NV_CODEC_HEADERS_REF}.tar.gz" \
+           -o nv-codec-headers.tgz \
+         || { echo "nv-codec-headers: code.ffmpeg.org unreachable, falling back to the GitHub mirror" >&2; \
+              curl -fsSL --retry 3 --retry-delay 2 --connect-timeout 20 --max-time 120 \
+                "https://github.com/FFmpeg/nv-codec-headers/archive/refs/tags/${NV_CODEC_HEADERS_REF}.tar.gz" \
+                -o nv-codec-headers.tgz; }; } \
     && tar xzf nv-codec-headers.tgz \
-    && cd nv-codec-headers \
-    && make install PREFIX=/usr/local \
+    && src_dir="$(find /tmp -maxdepth 1 -type d -name 'nv-codec-headers*' -print -quit)" \
+    && test -n "$src_dir" \
+    && test -f "$src_dir/include/ffnvcodec/dynlink_cuda.h" \
+    && grep -q cuStreamCreateWithPriority "$src_dir/include/ffnvcodec/dynlink_loader.h" \
+    && make -C "$src_dir" install PREFIX=/usr/local \
     && cd / && find /tmp -maxdepth 1 -name 'nv-codec-headers*' -exec rm -rf {} +
 
 USER vmaf

@@ -0,0 +1,90 @@
+<!-- markdownlint-disable MD013 MD060 -->
+
+# ADR-1200: The dev container falls back to the GitHub mirror for nv-codec-headers
+
+- **Status**: Proposed
+- **Date**: 2026-09-06
+- **Deciders**: Lusoris
+- **Tags**: build, supply-chain, ci
+
+## Context
+
+`dev/Containerfile` installs nv-codec-headers — which supplies
+`ffnvcodec/dynlink_cuda.h` for the CUDA backend — from a single host,
+`code.ffmpeg.org`. Its comment explained the choice: *"Pinned to commit that has
+cuStreamCreateWithPriority; GitHub mirror lags so use code.ffmpeg.org."*
+
+On 2026-09-06 that host was unreachable for over six hours. `curl` returned HTTP 000 after a
+30-second timeout on every attempt, and the layer simply stalled. The dev container is the
+canonical build and measurement environment under CLAUDE.md rule 15 and, since
+[ADR-1102](1102-phase4b9-container-only-publishing.md), the environment published artifacts
+are meant to come from. For that whole window it could not be built at all, and the only
+symptom was a build that appeared to hang.
+
+The stated reason for the single source does not hold for the pin actually in use. "The
+GitHub mirror lags" is true of an unreleased commit, but `NV_CODEC_HEADERS_REF` is a **tag**,
+`n13.1.15.0`, and that tag is published on both hosts. The GitHub tarball was checked and
+carries the `cuStreamCreateWithPriority` declaration in
+`include/ffnvcodec/dynlink_loader.h` — the exact symbol the pin exists for.
+
+## Decision
+
+The layer will try `code.ffmpeg.org` first and fall back to
+`https://github.com/FFmpeg/nv-codec-headers` when it is unreachable. Upstream's own host
+stays authoritative; the mirror only runs when the primary fails outright.
+
+Whichever archive arrives is then **asserted** rather than trusted: the build requires
+`include/ffnvcodec/dynlink_cuda.h` to exist and
+`grep -q cuStreamCreateWithPriority include/ffnvcodec/dynlink_loader.h` to succeed before
+`make install` runs. A fallback that silently installs the wrong headers would be worse than
+the outage it replaces.
+
+The primary is given a short leash — `--retry 1 --connect-timeout 10 --max-time 60`. With a
+working fallback behind it, minutes spent on a dead host are pure latency: measured, the
+first draft's `--retry 3 --max-time 120` took **487 s** to fall through and succeed, and the
+tightened values take **123 s**. The healthy path is a single 0.5 s fetch either way.
+
+The two archives unpack to different top-level directory names — `nv-codec-headers` from
+`code.ffmpeg.org`, `nv-codec-headers-<tag>` from GitHub — so the build `cd`s into whatever
+was extracted instead of a hard-coded name.
+
+## Alternatives considered
+
+| Option | Pros | Cons | Why not chosen |
+|---|---|---|---|
+| Try upstream, fall back to the GitHub mirror, assert the contents (**chosen**) | Survives either host being down; upstream stays authoritative; content assertion means a fallback cannot silently install the wrong headers | Two hosts can now supply a build input, so the trust surface is two names instead of one | — |
+| Switch to GitHub only | One host, and it was the reachable one today | Hands the canonical source to a mirror, and the original comment's concern — that the mirror lags for unreleased commits — becomes real the moment someone moves the pin off a tag | Trades one single point of failure for another, and a worse-provenanced one |
+| Vendor the headers into the repository | No network dependency at all; fully reproducible | ~86 KB of third-party headers to carry and re-sync by hand on every bump, and it hides the upstream pin | Disproportionate for a tagged 86 KB archive |
+| Add a build-time cache or internal proxy | Fixes this for every fetch in the file at once | Real infrastructure to run and keep alive; nothing exists to hang it on today | Right answer at a larger scale, unavailable now |
+| Leave it and wait out the outage | Zero work | It cost more than six hours of a build environment CLAUDE.md rule 15 makes mandatory, with no diagnostic beyond an apparent hang | The status quo is the defect |
+
+## Consequences
+
+- **Positive**: a single unreachable host can no longer make the dev container unbuildable.
+  Verified end to end while `code.ffmpeg.org` was still down: the layer fell through to
+  GitHub, installed the headers, and the next meson step reported
+  `Has header "ffnvcodec/dynlink_cuda.h" : YES`.
+- **Negative**: build inputs may now come from either of two hosts. The content assertion
+  bounds that — an archive that does not carry the expected header and symbol fails the
+  build rather than installing.
+- **Neutral / follow-ups**: other fetches in `dev/Containerfile` (oneAPI, ROCm, ONNX Runtime,
+  SVT-AV1, vvenc, AMF, FFmpeg) remain single-sourced. This ADR does not claim to have
+  audited them; it fixes the one that actually failed. A cache or proxy would address the
+  class.
+
+## Supply-chain impact
+
+- **New dependencies**: none. Same package, same tag, second source.
+- **Build-time fetches**: adds one conditional `curl` to
+  `github.com/FFmpeg/nv-codec-headers`, reached only when the primary fails.
+- **Sigstore-signable**: unchanged. Neither host serves a signed archive; the new
+  content assertion (header present, `cuStreamCreateWithPriority` present) is the
+  check that the bytes are the ones expected, and it did not exist before.
+- **CVE surface delta**: neutral. No new component enters the image.
+
+## References
+
+- req: found while the container rebuild for the epic #1246 retrain gates stalled for over
+  six hours on this exact layer.
+- [ADR-1102](1102-phase4b9-container-only-publishing.md) — container-canonical publishing,
+  which is what makes this outage a release-path problem and not just an inconvenience.
@@ -1012,5 +1012,6 @@ ADRs may exist there for local session continuity, but the tracked
 | [ADR-1185](1185-backend-perf-baseline-methodology.md) | Per-backend performance baselines are median-of-N, one backend per build dir | Accepted | perf, benchmarks, cuda, sycl, hip, docs |
 | [ADR-1198](1198-changelog-unknown-section-is-an-error.md) | An unknown `changelog.d/` subdirectory fails the run instead of warning | Proposed | ci, release, docs, testing |
 | [ADR-1199](1199-cuda-picture-handover-barrier.md) | Order caller-written CUDA pictures once per frame, at the dispatch point | Proposed | cuda, correctness, api, testing |
+| [ADR-1200](1200-nv-codec-headers-mirror-fallback.md) | The dev container falls back to the GitHub mirror for nv-codec-headers | Proposed | build, supply-chain, ci |
 | [ADR-1202](1202-cuda-speed-chroma-4k-launch-bounds.md) | GPU SpEED-chroma twins report singularity separately from failure | Proposed | cuda, sycl, hip, correctness, feature-extractor |
 | [ADR-1203](1203-cuda-psnr-hvs-enable-chroma-default.md) | `psnr_hvs_cuda` defaults `enable_chroma` to true, matching every other backend | Proposed | cuda, correctness, feature-extractor, options |
@@ -0,0 +1 @@
+| [ADR-1200](1200-nv-codec-headers-mirror-fallback.md) | The dev container falls back to the GitHub mirror for nv-codec-headers | Proposed | build, supply-chain, ci |
@@ -921,5 +921,6 @@
 1185-backend-perf-baseline-methodology
 1198-changelog-unknown-section-is-an-error
 1199-cuda-picture-handover-barrier
+1200-nv-codec-headers-mirror-fallback
 1202-cuda-speed-chroma-4k-launch-bounds
 1203-cuda-psnr-hvs-enable-chroma-default
@@ -48885,6 +48885,25 @@ Touches `core/src/libvmaf.c`, which is upstream-mirrored. Three invariants:
    load** — CPU load is not a stressor for this race (1/80 at load 22 versus
    56/60 with three concurrent CUDA processes), and two builds must be compared
    by interleaving runs, never sequentially.
+## fix/container-nv-codec-mirror-fallback — second source for nv-codec-headers (2026-09-06)
+
+Touches `dev/Containerfile` only. Two invariants:
+
+1. **Do not "simplify" the fallback back to a single `curl`.** The original
+   comment justified the single source with "GitHub mirror lags so use
+   code.ffmpeg.org", which is true for an unreleased commit and false for the
+   tag actually pinned — `n13.1.15.0` is published on both, and the GitHub
+   tarball carries the `cuStreamCreateWithPriority` declaration the pin exists
+   for. That host was unreachable for over six hours on 2026-09-06 and made the
+   container unbuildable. See [ADR-1200](adr/1200-nv-codec-headers-mirror-fallback.md).
+
+2. **Keep the content assertion and the `find`-based `cd`.** The build requires
+   `include/ffnvcodec/dynlink_cuda.h` and greps `dynlink_loader.h` for
+   `cuStreamCreateWithPriority` before `make install`; without it a fallback
+   could install the wrong headers silently, which is worse than the outage.
+   And the two archives unpack to DIFFERENT top-level directories
+   (`nv-codec-headers` vs `nv-codec-headers-<tag>`), so the hard-coded
+   `cd nv-codec-headers` that used to be here breaks on the mirror.
 ## docs/retrain-gate-status-1246 — measured retrain gate status (2026-09-06)
 
 Documentation only. One thing worth knowing:
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		\| [ADR-1200](1200-nv-codec-headers-mirror-fallback.md) \| The dev container falls back to the GitHub mirror for nv-codec-headers \| Proposed \| build, supply-chain, ci \|