ModelCache v0.1 — PVC backend, multi-node

[bassam]

Design lives in design/modelcache/ (branch dennis/modelcache-design). This issue tracks v0.1 implementation.

v0.1 scope

PVC backend, multi-node ready, no dedup. From the design doc:

• ModelCache CRD with artifact discriminator (Weights, Tokenizer, Bytes)
• Sources: huggingFace, s3, http, inline, configMap
• PVC (RWX) storage backend with one-shot prefetch Job (absorbs #61)
• replication: AllMatchingClusters — one RWX PVC per matching cluster, shared across all LWS gang pods
• clusterSelector.matchLabels for cluster filtering
• Mount path intrinsic to the cache; deployments reference by name via caches: [{ name }]
• Scheduling gated on per-cluster cache Ready condition
• Fail-fast when a target cluster has no RWX storage class on InferenceCluster.spec.storage.storageClassName
• Pluggable storage backend pattern shared with #72 KVOffloadTier

Out of scope (tracked separately)

• LoraAdapter / Engine artifact kinds → v0.2
• ContentAddressed backend (Modal-style tiered cache + lazy loading) → v0.2
• Cross-deployment / cross-tenant dedup → v0.2
• gcs / azure / oci / pvc-clone sources → v0.2
• AllMatchingNodes replication mode → v0.2
• Substrate unification #72 → v0.3

Roadmap detail in the design doc § v0.2 and § v0.3.

Examples

Nine (ModelCache + ModelDeployment) examples in design/modelcache/examples/: single-cluster basic, multi-node TensorPipeline gang, multi-cluster replication, separate tokenizer, private S3, opaque Bytes kind, plus three v0.2 previews.

References

• Design doc
• Examples
• #61 (closed) — RWX PVC mechanism
• #72
• PR #75 — engine.env + imagePullSecrets; ModelCache rides on those for credential-bearing sources

[dennis-upbound]

For replication: AllMatchingNodes on RWX-capable clusters, leaning on the mechanism from #61 (closing in favor of this):

Per cluster where the cache resolves:

1. Compose a ReadWriteMany PVC, sized to the cache's source size (could be an explicit spec.storage.size field, or computed from the source).
2. Compose a one-shot Job that pulls the source (HuggingFace, S3, etc.) to the PVC mount. Runs to completion; doesn't need to be long-lived.
3. Any ModelDeployment referencing this cache via cache: [{ name, mountPath }] mounts the PVC at mountPath in every pod. All pods in a LeaderWorker or Disaggregated gang share the same PVC, so the weights are downloaded once per cluster and read N times.

Storage class belongs on InferenceCluster.spec (platform team declares what RWX is available — GCP Filestore, AWS EFS, Azure Files, etc.). If a target cluster has no RWX storage class declared, the matcher rejects placement for any MD referencing a cache that needs AllMatchingNodes. Clean fail-fast.

Practical notes from #61 worth carrying over:

• Single download per cluster, not per pod. For 600GB+ frontier models (Kimi K2, Llama-405B), this is the difference between feasible and "every replica downloads 600GB".
• Job-based, not init-container-based. Avoids the OOM issues KServe's storage-initializer hits at 4Gi/8Gi/16Gi for very large models — Jobs have their own resources, separate from the serving pods.

Other replication strategies (PerNode for ReadOnlyMany or DaemonSet-based pre-pull, content-addressed shared storage like Alluxio/JuiceFS) can come as follow-ups; RWX is the simplest first cut and unblocks every multi-pod use case we have today.

[dennis-upbound]

Trying to nail down what ModelCache absorbs vs what stays in the engine pod spec on #64. Proposing this boundary for v0.1/v0.2:

In scope for ModelCache (mountable as a path the engine reads):

• Model weights from HF / S3 / GCS / Azure / OCI / HTTP / PVC (already shown)
• Compiled engine artifacts — TensorRT-LLM engines, vLLM compiled kernels, prefill checkpoints (already shown)
• Tokenizer files when not bundled with the engine
• Custom chat_template.jinja, vLLM --config-file, SGLang --config-yaml, similar engine config files
• Any auth-gated artifact — pulling via ModelCache means serving pods never see the secret

Out of scope (engine pod spec, not ModelCache):

• Container images — NIM bundles weights into the image, so the auth problem is at the registry, not the artifact source. ModelCache could extend to image pre-pull via DaemonSet later, but it's a different mechanism.
• Runtime env vars — NCCL_, VLLM_, etc.
• shmSize / /dev/shm for vLLM multi-process IPC
• Generic pod-spec knobs (probes, lifecycle hooks)

Worth confirming the "custom chat templates / config files" case is in scope — currently the issue only shows HF/S3 sources. Extending ModelCache.spec.source to accept inline content (source.inline: ...) or a ConfigMap ref (source.configMap: { name: ... }) would cover small text artifacts without a separate primitive.

If we agree on this boundary, the engine block on #64 only needs imagePullSecrets, shmSize, and env — small pod-spec extensions, not new abstractions. Everything else flows through ModelCache.