Skip to content

fix(thin): reconcile post-ready ThinRuntime fuse template updates - #6153

Open
cheyang wants to merge 1 commit into
fluid-cloudnative:masterfrom
cheyang:fix/thin-sync-runtime-fuse-template
Open

fix(thin): reconcile post-ready ThinRuntime fuse template updates#6153
cheyang wants to merge 1 commit into
fluid-cloudnative:masterfrom
cheyang:fix/thin-sync-runtime-fuse-template

Conversation

@cheyang

@cheyang cheyang commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Ⅰ. Describe what this PR does

pkg/ddc/base/syncs.go calls Implement.SyncRuntime() on every reconciliation, but
ThinEngine.SyncRuntime() was an empty stub returning (false, nil). The fuse values are therefore
only ever rendered once, during setup, so editing a ready ThinRuntime's spec.fuse never reached
the rendered values ConfigMap or the fuse DaemonSet. The only post-setup mutation path thin had is
ShouldUpdateUFS()updateFuseConfigOnChange(), which rewrites <name>-fuse-conf
(config.json, i.e. mounts/targetPath/runtimeOptions) and never touches the pod template. Operators
had to patch the generated DaemonSet by hand.

This implements SyncRuntime() following the JuiceFS shape, treating the helm values ConfigMap as
the last synced state:

  1. re-render the desired value from the ThinRuntime and its ThinRuntimeProfile,
  2. diff it against the last synced state,
  3. push the differences into the fuse DaemonSet,
  4. only then commit the advanced value back to the ConfigMap.

Doing it in that order — inside retry.RetryOnConflict — means an interrupted sync is retried on the
next reconciliation instead of being silently forgotten.

Fields synced: resources, image, imageTag, imagePullPolicy, envs (which is also how
fuse.options reaches the pod, via MOUNT_OPTIONS), lifecycle, pod labels and annotations,
volumes and volumeMounts.

Fields the chart renders verbatim (resources, image, imagePullPolicy) are compared against the live
DaemonSet, so manual drift is corrected too. Fields the chart merges with entries of its own (envs,
volumes, volumeMounts, labels, annotations) are compared against the last synced value and only the
value-derived entries are replaced, so the chart's own FLUID_RUNTIME_* env variables,
thin-fuse-mount/thin-conf volumes and role: thin-fuse labels survive.

Rollout semantics. The chart already sets updateStrategy: OnDelete
(charts/thin/templates/fuse/daemonset.yaml), so updating the template does not restart running
fuse pods. The strategy is re-checked before every sync and coerced back to OnDelete first if it is
anything else, because pushing a template into a RollingUpdate DaemonSet would restart the fuse
pods and break the applications mounting them. Unlike JuiceFS this deliberately does not bump
LabelRuntimeFuseGeneration, which would make CSI recycle the fuse pod — per the issue, active fuse
pods must not be deleted silently. The change is instead surfaced as a FuseTemplateUpdated event
that spells out that running pods keep the previous template until deleted.

Deliberately not synced, happy to widen the scope if reviewers prefer:

  • nodeSelectortransformFuse injects the fluid.io/f-<ns>-<name> scheduling label that CSI
    relies on to place fuse pods, so changing it after creation breaks mounting. JuiceFS carries the
    same note.
  • hostNetwork, hostPID, targetPath, ports, command, args and the probes, whose post-ready
    change semantics deserve their own discussion.
  • configValue, already reconciled by updateFuseConfigOnChange.
  • The worker StatefulSet — out of scope for this issue, which is about the fuse template.

Two supporting changes:

  • parseFuseOptions now sorts the rendered mount options. Its map iteration order was previously
    unobservable because the value was rendered exactly once, but with SyncRuntime in place it would
    make every reconciliation see a different MOUNT_OPTIONS env variable and keep updating the
    DaemonSet forever. This is the one change here that is a prerequisite rather than a nicety.
  • utils.TransformInternalResourcesToCoreV1Resources, the missing inverse of the existing
    TransformCoreV1ResourcesToInternalResources, needed to compare a value that round tripped through
    the values ConfigMap against the live DaemonSet.

Ⅱ. Does this pull request fix one issue?

fixes #6150

Ⅲ. List the added test cases (unit test/integration test) if any, please explain if no tests are needed.

pkg/ddc/thin/sync_runtime_test.go replaces the spec that asserted the stub's (false, nil) with
fake-client tests. The fixture renders the initial spec exactly the way setupMasterInternal does,
then seeds a values ConfigMap and a fuse DaemonSet that mimics what the chart produces — including
the entries the chart adds on top of .Values.fuse — so a test only has to edit the ThinRuntime.

  • TestSyncRuntimeConvergesFuseTemplate — 8 cases (resources, image + tag, imagePullPolicy,
    lifecycle, env, mount options, volumes + volumeMounts, pod labels + annotations). Each asserts the
    DaemonSet and the values ConfigMap converged, that the chart's own entries survived, and then
    syncs a second time and asserts changed == false.
  • TestSyncRuntimeIsANoOpWithoutSpecChanges — a freshly rendered runtime is not touched
    (DaemonSet resourceVersion unchanged).
  • TestSyncRuntimeCoercesUnsafeUpdateStrategy — a RollingUpdate DaemonSet has its strategy fixed
    first and its template left alone; the next reconciliation then syncs the spec.
  • TestSyncRuntimeWithoutValuesConfigMap — runtimes created with
    AnnotationDisableRuntimeHelmValueConfig have no last synced state, so sync degrades instead of
    failing the reconciliation.
  • TestSyncRuntimeWithoutProfile — a dangling spec.profileName degrades gracefully.
  • pkg/utils/resources_test.goTestTransformInternalResourcesToCoreV1Resources plus a
    TestTransformResourcesRoundTrip that pins the property the sync relies on.

No e2e test is added: this repo has no ThinRuntime e2e harness today and building one is
disproportionate here. The equivalent coverage was run by hand against a real cluster instead, below.

Ⅳ. Describe how to verify it

Verified on a real 3-node ACK cluster (Kubernetes v1.36.1), with an in-cluster NFS server, the
addons/nfs profile, a Dataset/ThinRuntime and a consumer pod that really mounts the NFS —
everything Running before any patch.

Reproduction, with the stock fluidcloudnative/thinruntime-controller:v1.1.0-36f0467. Patch a
ready ThinRuntime's fuse.resources (cpu 14, memory 128Mi256Mi), fuse.env and
fuse.lifecycle, wait 150s. The values ConfigMap and the fuse DaemonSet are byte-identical to
before; DaemonSet generation is still 1. Only spec.fuse differs.

With this PR (controller image built from this branch, same unchanged spec the stock controller
had ignored):

check result
values ConfigMap cpu: "4", memory: 256Mi, new lifecycle, env: after
fuse DaemonSet thin-fuse container same values; generation 1 → 2
running fuse pod uid and restartCount unchanged, still on the old cpu=1 template
consumer pod ready=true, restarts=0 throughout
image/imageTag/imagePullPolicy/podMetadata patch converged; all 12 chart labels and sidecar.istio.io/inject preserved
volumes/volumeMounts patch new entry added; chart's thin-fuse-mount and thin-conf preserved
chart env variables FLUID_RUNTIME_TYPE/_NS/_NAME intact
delete the fuse pod (opt-in rollout) replacement comes up Running with cpu=4 mem=256Mi env=after and the new volume
idempotence, 5 min idle DaemonSet generation and resourceVersion unchanged
event spam FuseTemplateUpdated fired exactly 3 times for 3 patches

Controller logs from the first sync, for reference:

syncFuseSpec: resources changed  {"old": {"requests":{"cpu":"1","memory":"128Mi"}}, "new": {"requests":{"cpu":"4","memory":"256Mi"}}}
syncFuseSpec: env variables changed  {"old": [{"name":"ISSUE_6150","value":"before"},...], "new": [{"name":"ISSUE_6150","value":"after"},...]}
syncFuseSpec: lifecycle changed
syncFuseSpec: some fields are changed in fuse, try to update the fuse daemonset
Committing the changed value to the helm value configmap

go build ./..., go vet, ./pkg/ddc/thin/..., ./pkg/common/... and the touched pkg/utils
tests all pass on linux/amd64.

Ⅴ. Special notes for reviews

  • The parseFuseOptions sort is load-bearing, not a drive-by cleanup. Without it this PR would
    update the fuse DaemonSet on every reconciliation whenever a runtime has two or more fuse options.
    It does mean the rendered MOUNT_OPTIONS string may be reordered once for existing runtimes; mount
    options are order-insensitive and it converges immediately.
  • The scope is the fuse template only. Worker StatefulSet sync, and the fuse fields listed as
    excluded above, are easy follow-ups if you'd rather have them in one change.
  • The OnDelete behaviour is intentional and is what the issue asks for: the template converges
    declaratively, but the operator decides when to roll the pods. The event exists so this is not
    silent. Happy to also reflect it in .status if reviewers prefer that over an event.
  • Unrelated finding while setting up the verification, reported separately: the published
    fluidcloudnative/nfs:v0.1 addon image no longer matches addons/nfs/ or the current chart — the
    image reads the pre-2023 /etc/fluid/config.json layout while the chart mounts the config at
    /etc/fluid/config/config.json, and the image is alpine + mount -t nfs while addons/nfs/docker/
    builds ubuntu + fuse-nfs. Following addons/nfs/readme.md as written yields a CrashLoopBackOff
    fuse pod. I worked around it by inlining the mount script via the profile's command/args.

ThinEngine.SyncRuntime was an empty stub, while pkg/ddc/base/syncs.go calls it on every
reconciliation. The fuse values were therefore only ever rendered once, during setup, so editing a
ready ThinRuntime's spec.fuse never reached the rendered values ConfigMap or the fuse DaemonSet and
operators had to patch the generated DaemonSet by hand.

Implement it following the JuiceFS shape, treating the helm values ConfigMap as the last synced
state: re-render the desired value from the ThinRuntime and its ThinRuntimeProfile, diff it against
that last synced state, push the differences into the fuse DaemonSet, and only then commit the
advanced value back to the ConfigMap, so an interrupted sync is retried instead of forgotten.

Covered fields are resources, image, imageTag, imagePullPolicy, envs (which is also how fuse options
reach the pod), lifecycle, pod labels and annotations, volumes and volumeMounts. nodeSelector is
left alone because transformFuse injects the fuse scheduling label CSI relies on, and configValue is
already reconciled by updateFuseConfigOnChange.

The fuse DaemonSet uses the OnDelete update strategy, so the template is updated without restarting
running fuse pods. The strategy is verified before every sync, and the change is surfaced as a
FuseTemplateUpdated event that spells out the rollout semantics.

parseFuseOptions now sorts the rendered mount options. Its map iteration order was previously
unobservable, but SyncRuntime would otherwise see a different mount options env variable on every
reconciliation and keep updating the DaemonSet forever.

Also add utils.TransformInternalResourcesToCoreV1Resources, the missing inverse of
TransformCoreV1ResourcesToInternalResources, needed to compare a value that round tripped through
the values ConfigMap against the live DaemonSet.

Fixes fluid-cloudnative#6150

Signed-off-by: cheyang <cheyang.cy@alibaba-inc.com>
@sonarqubecloud

sonarqubecloud Bot commented Aug 7, 2026

Copy link
Copy Markdown

@codecov

codecov Bot commented Aug 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 71.00372% with 78 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.17%. Comparing base (05f0665) to head (ad9796c).

Files with missing lines Patch % Lines
pkg/ddc/thin/sync_runtime.go 71.64% 41 Missing and 16 partials ⚠️
pkg/ddc/thin/util.go 53.33% 14 Missing and 7 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6153      +/-   ##
==========================================
+ Coverage   65.13%   65.17%   +0.04%     
==========================================
  Files         485      485              
  Lines       34039    34307     +268     
==========================================
+ Hits        22171    22361     +190     
- Misses      10127    10182      +55     
- Partials     1741     1764      +23     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Implements post-ready reconciliation for ThinRuntime FUSE template changes by using the Helm values ConfigMap as “last synced state”, re-rendering desired values, diffing, and applying supported updates to the live fuse DaemonSet (while enforcing OnDelete update strategy). This unblocks declarative updates to fields like resources/image/env/metadata after a runtime is Ready.

Changes:

  • Implement ThinEngine.SyncRuntime() to diff rendered fuse values vs last-synced values and update the fuse DaemonSet + values ConfigMap.
  • Stabilize mount option rendering by sorting spec.fuse.options so reconciliations are idempotent.
  • Add TransformInternalResourcesToCoreV1Resources (inverse conversion) plus new unit tests and expanded thin sync-runtime fake-client tests.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
pkg/ddc/thin/sync_runtime.go Core implementation of Thin post-ready fuse template sync and DaemonSet patching logic.
pkg/ddc/thin/sync_runtime_test.go New fake-client unit tests covering convergence, idempotence, degraded modes, and updateStrategy coercion.
pkg/ddc/thin/transform_fuse.go Sort mount option rendering to avoid endless diffs/churn when options are map-backed.
pkg/ddc/thin/util.go Add helpers to read/write the Helm values ConfigMap as last-synced state with conflict retry.
pkg/utils/resources.go Add inverse resources conversion for comparing configmap-rendered values with live workload specs.
pkg/utils/resources_test.go Unit tests for inverse conversion and round-trip equality property.
pkg/common/constants.go Add FuseTemplateUpdated event reason constant.
Suppressed comments (2)

pkg/ddc/thin/sync_runtime.go:280

  • If the live container env list already contains the latest value-derived env vars (manual patch / partial drift), removing only oldValue.Fuse.Envs by name and then appending latestValue.Fuse.Envs can create duplicate env var names. To keep the sync idempotent and avoid duplicates, exclude both the old and latest env var names before appending the latest slice.
		container.Env = append(
			utils.GetEnvsDifference(container.Env, oldValue.Fuse.Envs),
			latestValue.Fuse.Envs...)
		oldValue.Fuse.Envs = latestValue.Fuse.Envs
		changed = true

pkg/ddc/thin/sync_runtime.go:290

  • If the live container already has the latest value-derived volumeMounts (manual patch / drift) while oldValue still reflects the prior synced state, excluding only oldValue.Fuse.VolumeMounts and then appending latestValue.Fuse.VolumeMounts can duplicate mount entries by name. Exclude both old and latest names before appending to keep the resulting PodSpec stable.
		container.VolumeMounts = append(
			utils.GetVolumeMountsDifference(container.VolumeMounts, oldValue.Fuse.VolumeMounts),
			latestValue.Fuse.VolumeMounts...)
		oldValue.Fuse.VolumeMounts = latestValue.Fuse.VolumeMounts
		changed = true

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +206 to +210
fusesToUpdate.Spec.Template.Spec.Volumes = append(
utils.GetVolumesDifference(fusesToUpdate.Spec.Template.Spec.Volumes, oldValue.Fuse.Volumes),
latestValue.Fuse.Volumes...)
oldValue.Fuse.Volumes = latestValue.Fuse.Volumes
changed = true
Comment on lines +265 to +271
if latestPullPolicy := corev1.PullPolicy(latestValue.Fuse.ImagePullPolicy); latestPullPolicy != "" &&
container.ImagePullPolicy != latestPullPolicy {
t.Log.Info("syncFuseSpec: image pull policy changed", "old", container.ImagePullPolicy, "new", latestPullPolicy)
container.ImagePullPolicy = latestPullPolicy
oldValue.Fuse.ImagePullPolicy = latestValue.Fuse.ImagePullPolicy
changed = true
}
Comment on lines +85 to +91
if valueToSync == nil {
// The user opted out of the values ConfigMap, so there is no last synced state to diff
// against. Degrade to not syncing rather than failing the whole reconciliation.
t.Log.Info("Helm value configmap not found, skip syncing the runtime spec",
"configmap", t.getHelmValuesConfigMapName())
return nil
}

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Suppressed comments (3)

pkg/ddc/thin/sync_runtime.go:208

  • This replacement is not idempotent after an interrupted sync. If a newly added volume already reached the DaemonSet but the old ConfigMap was not committed, it is absent from oldValue, so GetVolumesDifference retains the live copy and this append adds it again. Kubernetes rejects the duplicate volume name, preventing all subsequent retries from converging. Remove both old and desired value-derived volume names before appending the desired set.
		fusesToUpdate.Spec.Template.Spec.Volumes = append(
			utils.GetVolumesDifference(fusesToUpdate.Spec.Template.Spec.Volumes, oldValue.Fuse.Volumes),
			latestValue.Fuse.Volumes...)

pkg/ddc/thin/sync_runtime.go:266

  • Leaving the live policy unchanged is not equivalent to the chart/Kubernetes default when an explicit policy is removed. For example, changing a profile from Always to empty for an image tagged v1 should default to IfNotPresent, but this condition leaves Always in the DaemonSet and also never advances the values ConfigMap. Resolve the empty value to Kubernetes' image-based default (or otherwise trigger API defaulting) and persist it as a synced change.
	// An empty value means the chart falls back to the Kubernetes default, so leave the daemonset
	// alone instead of clearing a policy that is already in effect.
	if latestPullPolicy := corev1.PullPolicy(latestValue.Fuse.ImagePullPolicy); latestPullPolicy != "" &&
		container.ImagePullPolicy != latestPullPolicy {

pkg/ddc/thin/sync_runtime.go:100

  • A successful DaemonSet update is not recoverable if the process stops before the ConfigMap write. On the next reconciliation, direct fields such as resources/image already match the desired live template, so syncFuseSpec reports false; merged fields also report false through the equality shortcut after only mutating the in-memory value. This branch then skips SaveValueToConfigmap, leaving the supposed last-synced state stale indefinitely. Track template changes and last-synced-value changes separately, and commit whenever the latter needs advancing, even if no new DaemonSet update is required.
		changed = fuseChanged
		if !changed {
			return nil

Comment on lines +226 to +228
fusesToUpdate.Spec.Template.Annotations = utils.UnionMapsWithOverride(
utils.GetMapsDifference(fusesToUpdate.Spec.Template.Annotations, oldValue.Fuse.Annotations),
latestValue.Fuse.Annotations)
Comment on lines +276 to +278
container.Env = append(
utils.GetEnvsDifference(container.Env, oldValue.Fuse.Envs),
latestValue.Fuse.Envs...)
Comment on lines +286 to +288
container.VolumeMounts = append(
utils.GetVolumeMountsDifference(container.VolumeMounts, oldValue.Fuse.VolumeMounts),
latestValue.Fuse.VolumeMounts...)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(thin): reconcile post-ready ThinRuntime FUSE template updates

2 participants