[BUG] curvine CacheRuntime DataLoad e2e fails: "load job requires UFS-only metadata"

[cheyang]

What is your environment(Kubernetes version, Fluid version, etc.)

• Fluid: master (controller image built from 676f47a, chart v1.1.0-676f47a)
• Kubernetes: Kind, fails on all tested versions — v1.22.17 / v1.24.17 / v1.30.13 / v1.33.2
• Curvine image: curvine/curvine-fluid:latest (community-provided, floating tag)
• CI: E2E Functionality Check → kind-e2e-test → step Fluid basic e2e tests
• Failing run: https://github.com/fluid-cloudnative/fluid/actions/runs/29186069855

Describe the bug

The e2e testcase curvine cache runtime basic e2e (test/gha-e2e/curvine/test.sh) fails consistently on all Kind K8s versions, and on PRs that change no relevant code (observed on the pure chart-version-bump PR #6115, and on unrelated branches). This indicates the failure is not caused by the PR under test.

curvine-dataload never reaches Complete within the 60s wait and stays in Executing; the loader Job repeatedly recreates pods that all exit 1. All four loader pods in the diagnostic bundle (pods-default/curvine-dataload-loader-job-*-dataloader.log) fail identically:

 Loading file to Curvine
Source path: /minio
❌ Error: [curvine-demo-master] ERROR: load job for CV path /minio requires UFS-only metadata
         (curvine-server/src/master/job/job_runner.rs:108):
Error: load /minio failed.

The DataLoad command defined in test/gha-e2e/curvine/cacheruntimeclass.yaml runs cv load /minio --watch immediately after the UFS mount (s3://test → /minio), but the Curvine master rejects the load job because the path has no "UFS-only metadata" loaded first.

What you expect to happen:

curvine-dataload reaches Complete and the curvine e2e passes.

How to reproduce it

Run the E2E Functionality Check workflow on any recent commit, or run test/gha-e2e/curvine/test.sh locally. It fails at wait_dataload_completed "curvine-dataload" (test/gha-e2e/curvine/test.sh:287):

>>> test "curvine cache runtime basic e2e" failed: dataload curvine-dataload status is Executing, not complete for 60s!

Additional Information

Likely root cause on the Fluid integration side:

1. Missing metadata-load step before cv load. Curvine requires the UFS path's metadata to be present before a load job can run. Fluid already has this concept (spec.loadMetadata / FLUID_DATALOAD_METADATA, see docs/*/samples/data_warmup.md and cacheruntime_data_operations.md), but the curvine CacheRuntimeClass DataLoad command only runs cv load $p --watch and neither performs a metadata-load step nor consumes FLUID_DATALOAD_METADATA; the e2e dataload.yaml also does not set spec.loadMetadata: true.
2. Floating image tag. All curvine components and the loader use curvine/curvine-fluid:latest (test/gha-e2e/curvine/cacheruntimeclass.yaml lines 52 / 78 / 102), with no digest pin, so upstream image changes can silently break e2e regardless of Fluid code — consistent with the failure appearing across unrelated PRs.

Suggested fixes:

• Add a metadata-load / listing step before cv load in the DataLoad command (and/or have the command honor FLUID_DATALOAD_METADATA, with the e2e setting spec.loadMetadata: true).
• Pin curvine/curvine-fluid to a fixed tag/digest in the e2e manifests, so upstream drift is decoupled from Fluid CI.

---

Information for the Curvine community

Since curvine/curvine-fluid:latest is community-provided, the following may help maintainers analyze whether this is intended behavior or a regression:

• Curvine image: curvine/curvine-fluid:latest (pulled fresh; behavior may have changed vs. an earlier :latest).
• Error origin: curvine-server/src/master/job/job_runner.rs:108 — load job for CV path /minio requires UFS-only metadata.
• Command that triggers it (from Fluid's CacheRuntimeClass DataLoad spec):

  cv load /minio --watch --conf ${CURVINE_CONF_DIR}/curvine-cluster.toml
  

• Mount setup: UFS s3://test (MinIO, endpoint_url=http://minio:9000, region_name=us-east-1, path_style=true) is mounted to CV path /minio via mountUfs.sh (/app/curvine/mountUfs.sh, timeout 120s) shortly before the load job is submitted.
• Loader topology warning (also present in every loader pod log — unsure if relevant):

  WARNING: No master pods found in topology, using generated journal address list:
  [{'id': 1, 'hostname': 'curvine-demo-master-0.svc-curvine-demo-master.default.svc.cluster.local', 'port': 8996}]
  

Questions for Curvine maintainers:

1. Is "load job requires UFS-only metadata" an intended precondition, i.e. must metadata be explicitly loaded/synced (e.g. via a separate cv subcommand) before cv load <ufs-path>? If so, what is the recommended command/flag?
2. Did the precondition or the cv load semantics change recently in curvine-fluid:latest?
3. Is the "No master pods found in topology" warning related to the load job being rejected?

[cheyang]

Supplementing the report with the relevant logs from the diagnostic bundle (testcase-curvine.tgz) of run 29186069855.

1. e2e driver output (test/gha-e2e/curvine/test.sh)

>>> Found succeeded job write-job
dataload.data.fluid.io/curvine-dataload created
>>> test "curvine cache runtime basic e2e" failed: dataload curvine-dataload status is Executing, not complete for 60s!

2. DataLoad loader pod (the actual failure — identical across all 4 retried pods 74m98 / h5tfm / jbkft / m9wh7)

WARNING: No master pods found in topology, using generated journal address list: [{'id': 1, 'hostname': 'curvine-demo-master-0.svc-curvine-demo-master.default.svc.cluster.local', 'port': 8996}]
export CURVINE_TARGET_PATH="/runtime-mnt/cache/default/curvine-demo/cache-fuse"
export CURVINE_MASTER_HOSTNAME="curvine-demo-master-0.svc-curvine-demo-master.default.svc.cluster.local"
export CURVINE_MASTER_SERVICE="svc-curvine-demo-master"
export CURVINE_MASTER_ENDPOINTS="curvine-demo-master-0.svc-curvine-demo-master.default.svc.cluster.local:8995"

 Loading file to Curvine
Source path: /minio
❌ Error: [curvine-demo-master] ERROR: load job for CV path /minio requires UFS-only metadata(curvine-server/src/master/job/job_runner.rs:108): 
Error: load /minio failed.

3. Curvine master pod (curvine-demo-master-0) — note the curvine build version

Curvine Fluid Entrypoint - Mode: cache-runtime
SERVER_TYPE: master, ACTION_TYPE: start
WARNING: No master pods found in topology, using generated journal address list: [{'id': 1, 'hostname': 'curvine-demo-master-0.svc-curvine-demo-master.default.svc.cluster.local', 'port': 8996}]
Fluid configuration parsed successfully
Starting master service...
datetime: 2026-07-12 08:57:58.103, git version: bd22d56, args: ServerArgs {
    service: "master",
    conf: "/app/curvine/conf/curvine-cluster.toml",
}

Curvine build in curvine/curvine-fluid:latest: git version: bd22d56 (master and worker both report this).

4. Curvine client / FUSE pod (curvine-demo-client-*) — mount + write succeed

2026/07/12 08:58:52.465 INFO fuse_mnt.rs:52 fuse mount success, path "/runtime-mnt/cache/default/curvine-demo/cache-fuse", fd 9
2026/07/12 08:58:54.856 INFO audit.rs:88 cmd=GetMountTable ok=true src= dst= usedUs=1342
2026/07/12 08:58:55.056 INFO audit.rs:88 cmd=GetStatus ok=true  src=/minio dst= usedUs=1396
2026/07/12 08:58:55.057 INFO audit.rs:88 cmd=GetStatus ok=false src=/minio/bar dst= usedUs=289 err=FileNotFound
2026/07/12 08:58:55.058 INFO audit.rs:88 cmd=Open:WCT  ok=true  src=/minio/bar dst= usedUs=437

So the /minio UFS mount is present and writable (the write-job wrote /minio/bar); only the cv load /minio load job is rejected by the master.

Relevant tuning from the client's rendered curvine-cluster.toml:

master_max_concurrent_load_jobs = 16
master_max_background_load_jobs = 1024
master_failed_load_job_retry_interval_str = "30s"
enable_rust_read_ufs = true

5. Fluid cacheruntime-controller — secondary error (not the cause of the e2e failure, but noisy)

engine/dataset.go:131  ReportSummary command is empty or not configured for this cache runtime  {"cacheRuntime":{"name":"curvine-demo"...}}
engine/sync.go:246 ERROR Failed to get cache states, keeping previous cache states in dataset status  {"error":"ReportSummary command is empty or not configured for this cache runtime"}
	.../pkg/ddc/cache/engine/sync.go:246
	.../pkg/ddc/cache/engine/sync.go:96

This means the curvine CacheRuntimeClass has no ReportSummary operation configured, so cache states are never populated — worth fixing separately, but the DataLoad failure above is the blocker for the e2e.

Full diagnostic bundle is attached to the failing run as artifact gha-e2e-logs-kind-e2e-test-v1.22.17.

[cheyang]

testcase-curvine.tgz

[santhosh-7777]

@cheyang is this bug still there i want to work on this>

[xliuqq]

solved by #6124 and #6130