Skip to content

feat(client): repo+tag override via constructor + MAT_VIS_DATASET env (#384) - #391

Merged
gerchowl merged 1 commit into
devfrom
feature/384-repo-tag-override
May 10, 2026
Merged

feat(client): repo+tag override via constructor + MAT_VIS_DATASET env (#384)#391
gerchowl merged 1 commit into
devfrom
feature/384-repo-tag-override

Conversation

@gerchowl

@gerchowl gerchowl commented May 10, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes #384.

Adds layered repo + tag override for MatVisClient so callers can point any client at a non-default HF dataset via three mechanisms — constructor kwargs, a single combined env var, or split env vars — without breaking the legacy MAT_VIS_HF_BASE form.

Precedence chain (high → low)

# Mechanism Example
1 Constructor kwargs MatVisClient(repo="gerchowl/mat-vis-tst", tag="v2026.04.99")
2 Combined env var MAT_VIS_DATASET=gerchowl/mat-vis-tst@v2026.04.99
3 Split env vars (PR #388 + new tag) MAT_VIS_HF_DATASET=gerchowl/mat-vis-tst + MAT_VIS_TAG=v2026.04.99
4 Legacy full URL (back-compat) MAT_VIS_HF_BASE=https://huggingface.co/datasets/<repo>/resolve
5 Default gerchowl/mat-vis @ DEFAULT_TAG

Implementation

  • New _resolve_repo_and_tag() helper — pure function returning (repo, tag, base_override). Single call site in __init__. 11 table-driven tests cover every layer + edge cases (combined-env wins over split form, partial kwargs, non-HF private-mirror URLs, @ in branch names, etc.).
  • self._base replaces global HF_BASE reads in _hf_url + manifest URL composition. Every URL the client touches now routes through the resolved coordinate.
  • at() forwards self._repo so per-tag scopes under env-driven overrides don't silently drift back to prod.
  • Cache layout gains a repo-slug segment: <cache_dir>/<client-version>/<repo-slug>/<tag>/. Two clients pointed at the same tag but different repos no longer collide. Pre-client: allow dataset-repo override for MatVisClient (env var + ctor arg) #384 default-repo caches become one-shot orphans on upgrade — one cache miss, no data loss.
  • Standalone client mirrors all of the above so the two clients honor identical precedence. Drift test still green. Also fixes the standalone tree-URL gap called out in client: allow dataset-repo override for MatVisClient (env var + ctor arg) #384's hidden-gap section (line 518 used to hardcode HF_DATASET).
  • Legacy MAT_VIS_HF_BASE URL preserved verbatim for I/O so private-mirror URL shapes that don't match huggingface.co/datasets/<repo> keep working. The repo is best-effort parsed from the URL only for cache namespacing.

Test plan

  • Full client suite: uv run --with pytest --with-editable clients/python pytest clients/python/tests/ → 542 passed, 29 skipped, 2 xfailed.
  • Cross-client drift + version sync: pytest tests/test_standalone_drift.py tests/test_version_sync.py → 11 passed.
  • New unit suite tests/test_client_repo_resolution.py (16 cases): every precedence layer + cache-keying regression test (two clients, same tag, different repos → distinct cache scopes) + URL-composition assertions + at() repo forwarding + legacy URL verbatim preservation.
  • Live smoke against tst dataset: 
    MAT_VIS_DATASET=gerchowl/mat-vis-tst@v2026.04.99-tst-full-369 \
  python -c "from mat_vis_client import MatVisClient; \
             print(len(MatVisClient().index('ambientcg')))"
# → 1949 (matches the acceptance criteria from the issue)

Notes / follow-ups

@gerchowl
feat(client): repo+tag override via constructor + MAT_VIS_DATASET env (…
6724969 
…#384)

Add layered repo + tag override for ``MatVisClient`` so callers can
point any client at a non-default HF dataset via three mechanisms.

Precedence (highest first):

  1. Constructor kwargs ``MatVisClient(repo=..., tag=...)``.
  2. ``MAT_VIS_DATASET=<repo>@<tag>`` — combined env var; last ``@``
     separates repo from tag.
  3. ``MAT_VIS_HF_DATASET=<repo>`` (PR #388) + ``MAT_VIS_TAG=<tag>``
     — split env-var form. Tag falls back to ``DEFAULT_TAG``.
  4. ``MAT_VIS_HF_BASE=<full-url>`` — legacy back-compat preserved.
     Repo is best-effort parsed from the URL for cache namespacing;
     the URL itself is used verbatim for I/O so private mirrors with
     non-HF URL shapes keep working.
  5. Default ``gerchowl/mat-vis`` @ ``DEFAULT_TAG``.

Implementation:

- New ``_resolve_repo_and_tag()`` module helper (pure function,
  table-tested) returns ``(repo, tag, base_override)``. Single call
  site in ``__init__``.
- ``self._base`` (resolve-URL prefix) replaces the global ``HF_BASE``
  read in ``_hf_url`` and manifest URL composition. Every URL the
  client touches now routes through the resolved coord.
- ``at()`` forwards ``self._repo`` so per-tag scopes under env-driven
  overrides don't silently drift back to prod.
- Cache layout gains a repo-slug segment:
  ``<cache_dir>/<client-version>/<repo-slug>/<tag>/...``. Two clients
  pointed at the same tag but different repos no longer collide.
  Pre-#384 default-repo caches become one-shot orphans on upgrade
  (one cache miss, no data loss).
- Standalone client mirrors all of the above so the two clients
  honor identical precedence — drift test still green.

Tests:

- ``test_client_repo_resolution.py`` (new, 16 cases): table-driven
  resolver coverage for every precedence layer plus end-to-end
  client-construction wiring (URL composition, cache namespacing,
  ``at()`` forwarding, legacy URL preserved verbatim).
- Existing tests that referenced the pre-#384 cache layout
  (``v0.6/<tag>/``) updated to the new ``v0.6/<repo-slug>/<tag>/``
  shape.

Validation:

- Full client suite: 542 passed.
- Cross-client drift + version sync: 11 passed.
- Live smoke against ``gerchowl/mat-vis-tst@v2026.04.99-tst-full-369``
  via ``MAT_VIS_DATASET``: 1949 ambientcg entries (matches the
  acceptance criteria in the issue).

Closes #384.
@github-actions github-actions Bot added area:docs Documentation, README, guides area:testing Test infrastructure, BATS, pytest area:client Python/JS/Rust client packages labels May 10, 2026
@gerchowl
gerchowl enabled auto-merge (squash) May 10, 2026 16:53
@gerchowl
gerchowl merged commit 256b4d3 into dev May 10, 2026
5 checks passed
This was referenced May 10, 2026
Closed
Merged
Merged
Closed
Merged
Closed
gerchowl added a commit that referenced this pull request May 12, 2026
@gerchowl
feat(ci): derive matrix — kind: thumb (#402) (#403)
ff03844 
* feat(ci): derive matrix — kind: thumb (#402)

Wire the per-material thumb-bake pipeline from #361 into the existing
derive.yml matrix as a third kind alongside resize and ktx2.

- .github/workflows/derive.yml: adds 'thumb' to the kind enum; plan
  step resolves target-tier=thumb (literal); a new dispatch step
  invokes the new 'thumb' Dagger cell when kind=thumb.
- .dagger/src/mat_vis_ci/main.py: new 'thumb' function. Reuses
  _baker_container, then 'uv pip install playwright>=1.45' and
  'playwright install --with-deps chromium' to add ~300 MB of
  renderer toolchain per-job. Sets MAT_VIS_DATASET=<repo>@<tag>
  (#391) so the embedded mat-vis-client reads the same release the
  publisher writes to. Two phases: render via bake/preview/run.py,
  then publish via mat-vis-baker hf-thumb-publish.
- src/mat_vis_baker/hf_thumb_publish.py: new module + CLI subcommand
  uploading <thumbs_dir>/<source>/<mid>/thumb.png to HF as
  <source>/thumb/<mid>/thumb.png. Reuses the bake/derive helpers
  (_fetch_manifest_with_parent, _merge_manifest_for_source,
  _extend_available_tiers, _create_commit_with_backoff) so cross-kind
  manifest writes CAS-merge cleanly. Extends the catalog's 'maps'
  list with 'thumb' so MatVisClient.fetch_texture(channel='thumb',
  tier='thumb') round-trips for free. Sentinel-last invariant matches
  every other tier (<source>/thumb/.tier_complete).
- tests/test_hf_thumb_publish.py: 9 unit tests covering the
  load-bearing properties (sentinel-last, dry-run silence, HEAD-probe
  skip, bad-PNG rejection, prod guard, idempotent map extension).

Layout choice: kept <source>/thumb/<mid>/thumb.png (standard tier
shape) instead of the <source>/<mid>/thumb/thumb.png mentioned in the
issue text — the former matches _per_file_url so client round-trip
works without touching the client. The 'channel' name is the literal
'thumb'; the 'tier' name is 'thumb'.

Closes the forward-coupling with #363: VisAsset.thumb returns real
bytes once a release lands the new tier.

* feat(ci): expose --limit input on derive.yml for thumb smoke runs

Full ambientcg thumb bake = 1949 materials × ~25s/render ≈ 13 h —
exceeds the 350-min job timeout. Add a --limit input (default 0 = no
cap, preserves resize / ktx2 behavior) plumbed through to the thumb
Dagger cell only.

* fix(ci): pull git-lfs in derive checkout — unblocks kind=thumb (#402)

Live workflow run 25670744187 failed end-to-end: every one of the 50
materials in the ambientcg limit raised the same renderer JS error:

    SyntaxError: Unexpected token 'v', "version ht"... is not valid JSON

Root cause: ``actions/checkout@v4`` defaults to fetching git-lfs
pointer files only, not the underlying objects. The container's
``bake/preview/assets/shader_ball.glb`` (LFS-tracked at oid 859306312d,
5.5MB) was a ~130-byte text file beginning with ``version https://git-
lfs.github.com/spec/v1``. Three.js' GLTFLoader detects the file isn't
a valid binary GLB header, falls through to JSON-parse mode, and
chokes on the leading 'v'.

Fix: add ``with: lfs: true`` to the derive job's checkout step.
Adds <1s overhead for resize/ktx2 kinds (they don't touch the GLB
but the LFS pull is cheap). Mandatory for kind=thumb.

Refs #402, #403.
gerchowl added a commit that referenced this pull request May 15, 2026
@gerchowl
feat(client): repo+tag override via constructor + MAT_VIS_DATASET env (…
7adc5bb 
…#384) (#391)

Add layered repo + tag override for ``MatVisClient`` so callers can
point any client at a non-default HF dataset via three mechanisms.

Precedence (highest first):

  1. Constructor kwargs ``MatVisClient(repo=..., tag=...)``.
  2. ``MAT_VIS_DATASET=<repo>@<tag>`` — combined env var; last ``@``
     separates repo from tag.
  3. ``MAT_VIS_HF_DATASET=<repo>`` (PR #388) + ``MAT_VIS_TAG=<tag>``
     — split env-var form. Tag falls back to ``DEFAULT_TAG``.
  4. ``MAT_VIS_HF_BASE=<full-url>`` — legacy back-compat preserved.
     Repo is best-effort parsed from the URL for cache namespacing;
     the URL itself is used verbatim for I/O so private mirrors with
     non-HF URL shapes keep working.
  5. Default ``gerchowl/mat-vis`` @ ``DEFAULT_TAG``.

Implementation:

- New ``_resolve_repo_and_tag()`` module helper (pure function,
  table-tested) returns ``(repo, tag, base_override)``. Single call
  site in ``__init__``.
- ``self._base`` (resolve-URL prefix) replaces the global ``HF_BASE``
  read in ``_hf_url`` and manifest URL composition. Every URL the
  client touches now routes through the resolved coord.
- ``at()`` forwards ``self._repo`` so per-tag scopes under env-driven
  overrides don't silently drift back to prod.
- Cache layout gains a repo-slug segment:
  ``<cache_dir>/<client-version>/<repo-slug>/<tag>/...``. Two clients
  pointed at the same tag but different repos no longer collide.
  Pre-#384 default-repo caches become one-shot orphans on upgrade
  (one cache miss, no data loss).
- Standalone client mirrors all of the above so the two clients
  honor identical precedence — drift test still green.

Tests:

- ``test_client_repo_resolution.py`` (new, 16 cases): table-driven
  resolver coverage for every precedence layer plus end-to-end
  client-construction wiring (URL composition, cache namespacing,
  ``at()`` forwarding, legacy URL preserved verbatim).
- Existing tests that referenced the pre-#384 cache layout
  (``v0.6/<tag>/``) updated to the new ``v0.6/<repo-slug>/<tag>/``
  shape.

Validation:

- Full client suite: 542 passed.
- Cross-client drift + version sync: 11 passed.
- Live smoke against ``gerchowl/mat-vis-tst@v2026.04.99-tst-full-369``
  via ``MAT_VIS_DATASET``: 1949 ambientcg entries (matches the
  acceptance criteria in the issue).

Closes #384.
gerchowl added a commit that referenced this pull request May 15, 2026
@gerchowl
feat(ci): derive matrix — kind: thumb (#402) (#403)
1caade4 
* feat(ci): derive matrix — kind: thumb (#402)

Wire the per-material thumb-bake pipeline from #361 into the existing
derive.yml matrix as a third kind alongside resize and ktx2.

- .github/workflows/derive.yml: adds 'thumb' to the kind enum; plan
  step resolves target-tier=thumb (literal); a new dispatch step
  invokes the new 'thumb' Dagger cell when kind=thumb.
- .dagger/src/mat_vis_ci/main.py: new 'thumb' function. Reuses
  _baker_container, then 'uv pip install playwright>=1.45' and
  'playwright install --with-deps chromium' to add ~300 MB of
  renderer toolchain per-job. Sets MAT_VIS_DATASET=<repo>@<tag>
  (#391) so the embedded mat-vis-client reads the same release the
  publisher writes to. Two phases: render via bake/preview/run.py,
  then publish via mat-vis-baker hf-thumb-publish.
- src/mat_vis_baker/hf_thumb_publish.py: new module + CLI subcommand
  uploading <thumbs_dir>/<source>/<mid>/thumb.png to HF as
  <source>/thumb/<mid>/thumb.png. Reuses the bake/derive helpers
  (_fetch_manifest_with_parent, _merge_manifest_for_source,
  _extend_available_tiers, _create_commit_with_backoff) so cross-kind
  manifest writes CAS-merge cleanly. Extends the catalog's 'maps'
  list with 'thumb' so MatVisClient.fetch_texture(channel='thumb',
  tier='thumb') round-trips for free. Sentinel-last invariant matches
  every other tier (<source>/thumb/.tier_complete).
- tests/test_hf_thumb_publish.py: 9 unit tests covering the
  load-bearing properties (sentinel-last, dry-run silence, HEAD-probe
  skip, bad-PNG rejection, prod guard, idempotent map extension).

Layout choice: kept <source>/thumb/<mid>/thumb.png (standard tier
shape) instead of the <source>/<mid>/thumb/thumb.png mentioned in the
issue text — the former matches _per_file_url so client round-trip
works without touching the client. The 'channel' name is the literal
'thumb'; the 'tier' name is 'thumb'.

Closes the forward-coupling with #363: VisAsset.thumb returns real
bytes once a release lands the new tier.

* feat(ci): expose --limit input on derive.yml for thumb smoke runs

Full ambientcg thumb bake = 1949 materials × ~25s/render ≈ 13 h —
exceeds the 350-min job timeout. Add a --limit input (default 0 = no
cap, preserves resize / ktx2 behavior) plumbed through to the thumb
Dagger cell only.

* fix(ci): pull git-lfs in derive checkout — unblocks kind=thumb (#402)

Live workflow run 25670744187 failed end-to-end: every one of the 50
materials in the ambientcg limit raised the same renderer JS error:

    SyntaxError: Unexpected token 'v', "version ht"... is not valid JSON

Root cause: ``actions/checkout@v4`` defaults to fetching git-lfs
pointer files only, not the underlying objects. The container's
``bake/preview/assets/shader_ball.glb`` (LFS-tracked at oid 859306312d,
5.5MB) was a ~130-byte text file beginning with ``version https://git-
lfs.github.com/spec/v1``. Three.js' GLTFLoader detects the file isn't
a valid binary GLB header, falls through to JSON-parse mode, and
chokes on the leading 'v'.

Fix: add ``with: lfs: true`` to the derive job's checkout step.
Adds <1s overhead for resize/ktx2 kinds (they don't touch the GLB
but the LFS pull is cheap). Mandatory for kind=thumb.

Refs #402, #403.
@gerchowl
gerchowl deleted the feature/384-repo-tag-override branch July 1, 2026 06:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

area:client Python/JS/Rust client packages area:docs Documentation, README, guides area:testing Test infrastructure, BATS, pytest

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

client: allow dataset-repo override for MatVisClient (env var + ctor arg)

1 participant

@gerchowl