feat(ci): VLM-based visual QA gate — compare our thumbs against upstream previews

[gerchowl]

Problem

We have no automated way to catch visual defects in baked materials before they reach clients. Pixel-level comparison doesn't work because our thumbnails are 3D shader-ball renders while upstream sources provide flat texture swatches — SSIM/perceptual hashing would flag everything.

Proposed solution

Use a vision-language model (Claude) to compare (upstream_preview, our_thumb) pairs and flag discrepancies.

What it catches

• Wrong texture mapped to material
• Inverted/swapped normal maps (lighting looks wrong)
• Metalness/roughness channel swaps (dielectric looks metallic or vice versa)
• Completely black/white/missing renders
• Obvious color shifts (wrong colorspace, gamma issues)

What it doesn't catch

• Subtle banding, compression artifacts
• Minor color calibration differences
• Resolution/sharpness differences (expected between upstream 4K source and our 256px thumb)

Data sources for comparison

Source	Upstream preview	How to get it
ambientcg	256px PNG preview	asset["previewImage"]["256-PNG"] from API
polyhaven	Render preview	https://cdn.polyhaven.com/asset_img/thumbs/{slug}.png
gpuopen	Material thumbnail	material["thumbnail"] from API
physicallybased	N/A (scalar only)	Skip — compare hex color swatch against scalar color_rgb instead

Our thumbnails: <source>/thumb/<material_id>/thumb.png on HF substrate.

Integration point

Step in the #437 delta pipeline:

bake new materials to tst
  → derive thumbs
  → sample N thumbnails (all new + random subset of existing)
  → fetch upstream previews for the same materials
  → batch pairs to VLM: "Does this render match this material?"
  → flag suspicious pairs in the PR comment with side-by-side images
  → maintainer eyeballs flagged pairs before merging

VLM prompt shape

You are reviewing PBR material bake quality. For each pair:
- LEFT: upstream source preview (flat texture swatch or photo)
- RIGHT: our 3D shader-ball render of the same material

Flag ONLY if the render looks fundamentally wrong:
- Wrong material (completely different color/pattern)
- Inverted normals (lighting from wrong direction)
- Channel swap (metal looks plastic or vice versa)
- Missing/black/white render
- Major color shift

Do NOT flag expected differences (3D vs flat, resolution, lighting style).

Return: { "material_id": "...", "verdict": "ok" | "suspicious", "reason": "..." }

Cost estimate

• ~50 pairs per batch (Claude vision supports multiple images)
• ~3000 materials total across all sources
• Full sweep: ~60 batches × ~$0.10/batch ≈ $6 for a complete QA pass
• Delta-only (new materials): typically <50 materials, 1 batch, ~$0.10

Related

• feat(ci): upstream delta detection — auto-PR for new bakeable materials #437 — upstream delta detection pipeline (this is a step in that pipeline)
• feat(docs): GH Pages material-catalog gallery — filterable, searchable, with /tst preview #423 — GH Pages catalog (could host side-by-side comparison view)
• coverage: discover() returns unbakeable materials — coverage checker needs per-material capability classification #435 — discover() classification (VLM should skip unbakeable/scalar-only materials)