Add GitHub Packages (GHCR) support, downloading images as OCI layouts

[yarikoptic]

An alternative to #222 for #217. Opened separately rather than pushed onto that branch so the two can be compared; close whichever loses.

Why not #222

Reviewing #222 first, the container-download path added in its last commit has never run successfully — the PR has zero check runs, and its integration test fails immediately. Three defects each independently prevent any image from being downloaded:

• The registry auth is wrong. It sends Authorization: Bearer <raw GitHub PAT> to ghcr.io, reusing the api.github.com client. GHCR does not accept a raw PAT as a bearer token, and an invalid header gets a 403 DENIED rather than an anonymous fallback, even for public images:

  ghcr.io/v2/con/tinuous-inception/manifests/latest   (public)
    no Authorization -> 401
    bogus bearer     -> 403 DENIED
  

  The failure is caught and logged at log.debug, so a run reports "N package versions saved" while having written only a stub JSON file.
• Multi-arch images download zero layers. Sub-manifests are fetched with GET /v2/<img>/blobs/<digest>; manifests are not blobs. Against a real dandi image that returns 404 BLOB_UNKNOWN. Every dandi package is multi-arch.
• The stored manifest digest is fabricated. The manifest JSON is reserialized with sort_keys=True before hashing, so the archived digest is not the upstream digest and the copy can never be verified against, or restored under, its original reference.

Beyond that: a 403 from /orgs/{owner}/packages (a token without read:packages, which includes the Actions GITHUB_TOKEN) is not caught and aborts the whole fetch, logs and artifacts included; the package listing is org-wide with no repo scoping; and the integration test os.chdirs into a deleted temporary directory, which breaks 7 unrelated tests in test_state.py.

Approach

I looked for a Python library rather than shelling out to skopeo.

	verdict
oras (oras-py)	chosen. Official Python SDK of the ORAS project, Apache-2.0, actively released, pure Python (requests + jsonschema). Handles image references, credentials, and the WWW-Authenticate token exchange.
python-dxf	rejected — will not install: its www-authenticate dependency fails to build on current setuptools. No layout support, no manifest-list recursion.
docker / podman-py	rejected — need a running daemon.
containerregistry (Google)	rejected — unmaintained, Bazel-oriented.

oras ships an OCI layout puller (oras.layout.NewLayoutFromRegistry) that does almost exactly what is wanted, and does the things #222 got wrong correctly — verbatim manifest bytes, Docker-Content-Digest, recursive index walk. It cannot be used as-is: it dispatches only on the OCI media types, and GHCR serves the Docker spellings for anything pushed by docker buildx, which is both of our targets:

ValueError: Unsupported manifest mediaType:
  application/vnd.docker.distribution.manifest.list.v2+json

So this PR uses oras for the part that is genuinely subtle and well-tested — the registry client and the auth dance — and keeps the manifest walk in tinuous (src/tinuous/ghcr.py, ~130 lines of logic). That also lets the walk own things a backup tool wants and the library does not do: digest verification, atomic blob writes, and a completeness marker.

Both gaps are being taken upstream so this module can shrink later: the media-type dispatch (unreported; also affects push_to_registry, where a partial fix would silently upload manifests as plain blobs), and blob/manifest digest verification (already filed upstream as oras-py#247).

What it does

paths.packages gets a template; each package version is saved as an OCI image layout:

ci:
  github:
    paths:
      packages: '{year}/{month}/{ci}/packages/{package_name}/{digest}/'
    packages:
      include: ['tinuous-.*']
      regex: true
      owner_wide: false   # default: only this repository's packages
      untagged: false     # default: skip buildx intermediates and attestations

$ podman run oci:2026/08/github/packages/tinuous-inception/sha256%3a530a02.../
Built at: 2026-01-06 20:30:58 UTC

Correctness properties, each covered by a test:

• Credentials are exchanged at the realm the registry names; the resulting bearer token is what is presented.
• Manifests are stored byte-for-byte as served, so archived digests are upstream digests.
• Image indexes are walked recursively via the manifest endpoint; every platform's manifest, config, and layers are stored.
• Blobs stream to a .tmp file, are hashed, and are moved into place only on a match; a failure removes the partial file rather than leaving something a later run mistakes for complete.
• index.json is written last, so an interrupted download is not a valid layout and is redone.
• Images are pulled by digest, not tag, so a tag moving mid-run cannot substitute a different image.
• A version already on disk is skipped; one whose digest differs replaces the layout, so a {tag}-keyed path keeps up with a moved tag instead of freezing on the first image it saw.
• Blobs shared between platforms are fetched once.

Failure modes are loud: a token without read:packages gets RuntimeError: ... a token with the 'read:packages' scope is needed, not a silent empty result.

Repo scoping

GitHub only lists packages per owner. https://github.com/orgs/dandi/packages has ~33 container packages and all of them belong to dandi/example-notebooks, so with #222's org-wide listing a backup of any other dandi repo would try to pull all of them. Here the default is to fetch only packages whose repository.full_name matches the configured repo; owner_wide: true opts into the old behaviour.

Volume

dandi/example-notebooks/001550-paganlab is 908 MB across its two platforms and carries 20 tags. Versions of a package share most of their layers; inside a DataLad dataset git-annex stores each layer once regardless of how many versions reference it. Documented in the README.

Testing

test/fake_registry.py is an in-process registry serving images built from /bin/busybox — single-platform, multi-platform, and a nested name like dandi's, in both Docker and OCI media types, optionally behind a token challenge. Tests need no network and no credentials. Where podman and a static busybox are both present, three tests run podman run oci:<layout> and assert on the container's output; where they are not, only those three skip.

tox -e py:  109 passed        # 61 before this PR
            src/tinuous/ghcr.py  ~96% covered
            103 passed, 3 skipped    # on a machine with no busybox
flake8: clean
mypy:   Success: no issues found in 20 source files

Against the real registry, verified from a sandbox whose egress policy blocks GHCR's blob CDN (pkg-containers.githubusercontent.com — skopeo copy fails there identically), so layer bytes could not be transferred; blob downloads were replaced by an unredirected HEAD, which still proves each blob is reachable and the size the manifest claims. Everything else is the real code path:

target	media type	manifests	blobs
ghcr.io/con/tinuous-inception:latest	manifest.v2	1	4 — 3.9 MB
ghcr.io/dandi/example-notebooks/001550-paganlab:latest	manifest.list.v2	index + linux/amd64 + linux/arm64	30 — 908.1 MB
ghcr.io/dandi/example-notebooks/000409-ibl:hash-064621cba038-arm64	manifest.v2	1	15 — 588.4 MB

All three produced a complete layout whose index.json digest matched the digest the registry reported.

Two things could not be exercised here and want a real run before merge: the /orgs/{owner}/packages listing (my token lacks read:packages, so that path is covered only by unit tests against mocked responses), and a pull of a private package with a real token.

CI

Green on cc49949 — GitHub Actions (lint, typing, codespell, CPython 3.10–3.14, pypy-3.11, test (base), test (datalad)), CircleCI, Codecov, and both AppVeyor builds. Coverage goes up: 60.49% → 66.00% (+5.51%).

For the record, since it was red for a while: on the pre-merge head this PR had four failing checks, all of which #233 resolved rather than anything in this branch. test (base)/test (datalad) were dying on KeyError: 'next_page_token' in src/tinuous/circleci.py, code this PR does not touch; the two pypy jobs could not build pydantic-core on PyPy < 3.11 and were red on master too. AppVeyor was also red and I could not diagnose it — ci.appveyor.com is unreachable from where I was working — and it went green once #233 dropped Python 3.9 from the AppVeyor matrix, which is consistent with the suspicion that it was the 3.9 job failing to resolve the new oras/jsonschema dependency chain (jsonschema >= 4.26 needs 3.10, rpds-py >= 2026 needs 3.11). That remains inferred, not proven — I never saw the log.

[codecov]

Codecov Report

❌ Patch coverage is 91.61074% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.00%. Comparing base (21204ee) to head (cc49949).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
src/tinuous/__main__.py	11.76%	11 Missing and 4 partials ⚠️
src/tinuous/ghcr.py	95.55%	3 Missing and 3 partials ⚠️
src/tinuous/github.py	96.87%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #232      +/-   ##
==========================================
+ Coverage   60.49%   66.00%   +5.51%     
==========================================
  Files          10       11       +1     
  Lines        1420     1715     +295     
  Branches      222      263      +41     
==========================================
+ Hits          859     1132     +273     
- Misses        495      511      +16     
- Partials       66       72       +6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.