fix(thin): reconcile post-ready ThinRuntime fuse template updates - #6153
fix(thin): reconcile post-ready ThinRuntime fuse template updates#6153cheyang wants to merge 1 commit into
Conversation
ThinEngine.SyncRuntime was an empty stub, while pkg/ddc/base/syncs.go calls it on every reconciliation. The fuse values were therefore only ever rendered once, during setup, so editing a ready ThinRuntime's spec.fuse never reached the rendered values ConfigMap or the fuse DaemonSet and operators had to patch the generated DaemonSet by hand. Implement it following the JuiceFS shape, treating the helm values ConfigMap as the last synced state: re-render the desired value from the ThinRuntime and its ThinRuntimeProfile, diff it against that last synced state, push the differences into the fuse DaemonSet, and only then commit the advanced value back to the ConfigMap, so an interrupted sync is retried instead of forgotten. Covered fields are resources, image, imageTag, imagePullPolicy, envs (which is also how fuse options reach the pod), lifecycle, pod labels and annotations, volumes and volumeMounts. nodeSelector is left alone because transformFuse injects the fuse scheduling label CSI relies on, and configValue is already reconciled by updateFuseConfigOnChange. The fuse DaemonSet uses the OnDelete update strategy, so the template is updated without restarting running fuse pods. The strategy is verified before every sync, and the change is surfaced as a FuseTemplateUpdated event that spells out the rollout semantics. parseFuseOptions now sorts the rendered mount options. Its map iteration order was previously unobservable, but SyncRuntime would otherwise see a different mount options env variable on every reconciliation and keep updating the DaemonSet forever. Also add utils.TransformInternalResourcesToCoreV1Resources, the missing inverse of TransformCoreV1ResourcesToInternalResources, needed to compare a value that round tripped through the values ConfigMap against the live DaemonSet. Fixes fluid-cloudnative#6150 Signed-off-by: cheyang <cheyang.cy@alibaba-inc.com>
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #6153 +/- ##
==========================================
+ Coverage 65.13% 65.17% +0.04%
==========================================
Files 485 485
Lines 34039 34307 +268
==========================================
+ Hits 22171 22361 +190
- Misses 10127 10182 +55
- Partials 1741 1764 +23 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
Implements post-ready reconciliation for ThinRuntime FUSE template changes by using the Helm values ConfigMap as “last synced state”, re-rendering desired values, diffing, and applying supported updates to the live fuse DaemonSet (while enforcing OnDelete update strategy). This unblocks declarative updates to fields like resources/image/env/metadata after a runtime is Ready.
Changes:
- Implement
ThinEngine.SyncRuntime()to diff rendered fuse values vs last-synced values and update the fuse DaemonSet + values ConfigMap. - Stabilize mount option rendering by sorting
spec.fuse.optionsso reconciliations are idempotent. - Add
TransformInternalResourcesToCoreV1Resources(inverse conversion) plus new unit tests and expanded thin sync-runtime fake-client tests.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/ddc/thin/sync_runtime.go | Core implementation of Thin post-ready fuse template sync and DaemonSet patching logic. |
| pkg/ddc/thin/sync_runtime_test.go | New fake-client unit tests covering convergence, idempotence, degraded modes, and updateStrategy coercion. |
| pkg/ddc/thin/transform_fuse.go | Sort mount option rendering to avoid endless diffs/churn when options are map-backed. |
| pkg/ddc/thin/util.go | Add helpers to read/write the Helm values ConfigMap as last-synced state with conflict retry. |
| pkg/utils/resources.go | Add inverse resources conversion for comparing configmap-rendered values with live workload specs. |
| pkg/utils/resources_test.go | Unit tests for inverse conversion and round-trip equality property. |
| pkg/common/constants.go | Add FuseTemplateUpdated event reason constant. |
Suppressed comments (2)
pkg/ddc/thin/sync_runtime.go:280
- If the live container env list already contains the latest value-derived env vars (manual patch / partial drift), removing only oldValue.Fuse.Envs by name and then appending latestValue.Fuse.Envs can create duplicate env var names. To keep the sync idempotent and avoid duplicates, exclude both the old and latest env var names before appending the latest slice.
container.Env = append(
utils.GetEnvsDifference(container.Env, oldValue.Fuse.Envs),
latestValue.Fuse.Envs...)
oldValue.Fuse.Envs = latestValue.Fuse.Envs
changed = true
pkg/ddc/thin/sync_runtime.go:290
- If the live container already has the latest value-derived volumeMounts (manual patch / drift) while oldValue still reflects the prior synced state, excluding only oldValue.Fuse.VolumeMounts and then appending latestValue.Fuse.VolumeMounts can duplicate mount entries by name. Exclude both old and latest names before appending to keep the resulting PodSpec stable.
container.VolumeMounts = append(
utils.GetVolumeMountsDifference(container.VolumeMounts, oldValue.Fuse.VolumeMounts),
latestValue.Fuse.VolumeMounts...)
oldValue.Fuse.VolumeMounts = latestValue.Fuse.VolumeMounts
changed = true
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| fusesToUpdate.Spec.Template.Spec.Volumes = append( | ||
| utils.GetVolumesDifference(fusesToUpdate.Spec.Template.Spec.Volumes, oldValue.Fuse.Volumes), | ||
| latestValue.Fuse.Volumes...) | ||
| oldValue.Fuse.Volumes = latestValue.Fuse.Volumes | ||
| changed = true |
| if latestPullPolicy := corev1.PullPolicy(latestValue.Fuse.ImagePullPolicy); latestPullPolicy != "" && | ||
| container.ImagePullPolicy != latestPullPolicy { | ||
| t.Log.Info("syncFuseSpec: image pull policy changed", "old", container.ImagePullPolicy, "new", latestPullPolicy) | ||
| container.ImagePullPolicy = latestPullPolicy | ||
| oldValue.Fuse.ImagePullPolicy = latestValue.Fuse.ImagePullPolicy | ||
| changed = true | ||
| } |
| if valueToSync == nil { | ||
| // The user opted out of the values ConfigMap, so there is no last synced state to diff | ||
| // against. Degrade to not syncing rather than failing the whole reconciliation. | ||
| t.Log.Info("Helm value configmap not found, skip syncing the runtime spec", | ||
| "configmap", t.getHelmValuesConfigMapName()) | ||
| return nil | ||
| } |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.
Suppressed comments (3)
pkg/ddc/thin/sync_runtime.go:208
- This replacement is not idempotent after an interrupted sync. If a newly added volume already reached the DaemonSet but the old ConfigMap was not committed, it is absent from
oldValue, soGetVolumesDifferenceretains the live copy and this append adds it again. Kubernetes rejects the duplicate volume name, preventing all subsequent retries from converging. Remove both old and desired value-derived volume names before appending the desired set.
fusesToUpdate.Spec.Template.Spec.Volumes = append(
utils.GetVolumesDifference(fusesToUpdate.Spec.Template.Spec.Volumes, oldValue.Fuse.Volumes),
latestValue.Fuse.Volumes...)
pkg/ddc/thin/sync_runtime.go:266
- Leaving the live policy unchanged is not equivalent to the chart/Kubernetes default when an explicit policy is removed. For example, changing a profile from
Alwaysto empty for an image taggedv1should default toIfNotPresent, but this condition leavesAlwaysin the DaemonSet and also never advances the values ConfigMap. Resolve the empty value to Kubernetes' image-based default (or otherwise trigger API defaulting) and persist it as a synced change.
// An empty value means the chart falls back to the Kubernetes default, so leave the daemonset
// alone instead of clearing a policy that is already in effect.
if latestPullPolicy := corev1.PullPolicy(latestValue.Fuse.ImagePullPolicy); latestPullPolicy != "" &&
container.ImagePullPolicy != latestPullPolicy {
pkg/ddc/thin/sync_runtime.go:100
- A successful DaemonSet update is not recoverable if the process stops before the ConfigMap write. On the next reconciliation, direct fields such as resources/image already match the desired live template, so
syncFuseSpecreports false; merged fields also report false through the equality shortcut after only mutating the in-memory value. This branch then skipsSaveValueToConfigmap, leaving the supposed last-synced state stale indefinitely. Track template changes and last-synced-value changes separately, and commit whenever the latter needs advancing, even if no new DaemonSet update is required.
changed = fuseChanged
if !changed {
return nil
| fusesToUpdate.Spec.Template.Annotations = utils.UnionMapsWithOverride( | ||
| utils.GetMapsDifference(fusesToUpdate.Spec.Template.Annotations, oldValue.Fuse.Annotations), | ||
| latestValue.Fuse.Annotations) |
| container.Env = append( | ||
| utils.GetEnvsDifference(container.Env, oldValue.Fuse.Envs), | ||
| latestValue.Fuse.Envs...) |
| container.VolumeMounts = append( | ||
| utils.GetVolumeMountsDifference(container.VolumeMounts, oldValue.Fuse.VolumeMounts), | ||
| latestValue.Fuse.VolumeMounts...) |



Ⅰ. Describe what this PR does
pkg/ddc/base/syncs.gocallsImplement.SyncRuntime()on every reconciliation, butThinEngine.SyncRuntime()was an empty stub returning(false, nil). The fuse values are thereforeonly ever rendered once, during setup, so editing a ready
ThinRuntime'sspec.fusenever reachedthe rendered values ConfigMap or the fuse DaemonSet. The only post-setup mutation path thin had is
ShouldUpdateUFS()→updateFuseConfigOnChange(), which rewrites<name>-fuse-conf(
config.json, i.e. mounts/targetPath/runtimeOptions) and never touches the pod template. Operatorshad to patch the generated DaemonSet by hand.
This implements
SyncRuntime()following the JuiceFS shape, treating the helm values ConfigMap asthe last synced state:
ThinRuntimeand itsThinRuntimeProfile,Doing it in that order — inside
retry.RetryOnConflict— means an interrupted sync is retried on thenext reconciliation instead of being silently forgotten.
Fields synced:
resources,image,imageTag,imagePullPolicy,envs(which is also howfuse.optionsreaches the pod, viaMOUNT_OPTIONS),lifecycle, podlabelsandannotations,volumesandvolumeMounts.Fields the chart renders verbatim (resources, image, imagePullPolicy) are compared against the live
DaemonSet, so manual drift is corrected too. Fields the chart merges with entries of its own (envs,
volumes, volumeMounts, labels, annotations) are compared against the last synced value and only the
value-derived entries are replaced, so the chart's own
FLUID_RUNTIME_*env variables,thin-fuse-mount/thin-confvolumes androle: thin-fuselabels survive.Rollout semantics. The chart already sets
updateStrategy: OnDelete(
charts/thin/templates/fuse/daemonset.yaml), so updating the template does not restart runningfuse pods. The strategy is re-checked before every sync and coerced back to
OnDeletefirst if it isanything else, because pushing a template into a
RollingUpdateDaemonSet would restart the fusepods and break the applications mounting them. Unlike JuiceFS this deliberately does not bump
LabelRuntimeFuseGeneration, which would make CSI recycle the fuse pod — per the issue, active fusepods must not be deleted silently. The change is instead surfaced as a
FuseTemplateUpdatedeventthat spells out that running pods keep the previous template until deleted.
Deliberately not synced, happy to widen the scope if reviewers prefer:
nodeSelector—transformFuseinjects thefluid.io/f-<ns>-<name>scheduling label that CSIrelies on to place fuse pods, so changing it after creation breaks mounting. JuiceFS carries the
same note.
hostNetwork,hostPID,targetPath,ports,command,argsand the probes, whose post-readychange semantics deserve their own discussion.
configValue, already reconciled byupdateFuseConfigOnChange.Two supporting changes:
parseFuseOptionsnow sorts the rendered mount options. Its map iteration order was previouslyunobservable because the value was rendered exactly once, but with
SyncRuntimein place it wouldmake every reconciliation see a different
MOUNT_OPTIONSenv variable and keep updating theDaemonSet forever. This is the one change here that is a prerequisite rather than a nicety.
utils.TransformInternalResourcesToCoreV1Resources, the missing inverse of the existingTransformCoreV1ResourcesToInternalResources, needed to compare a value that round tripped throughthe values ConfigMap against the live DaemonSet.
Ⅱ. Does this pull request fix one issue?
fixes #6150
Ⅲ. List the added test cases (unit test/integration test) if any, please explain if no tests are needed.
pkg/ddc/thin/sync_runtime_test.goreplaces the spec that asserted the stub's(false, nil)withfake-client tests. The fixture renders the initial spec exactly the way
setupMasterInternaldoes,then seeds a values ConfigMap and a fuse DaemonSet that mimics what the chart produces — including
the entries the chart adds on top of
.Values.fuse— so a test only has to edit theThinRuntime.TestSyncRuntimeConvergesFuseTemplate— 8 cases (resources, image + tag, imagePullPolicy,lifecycle, env, mount options, volumes + volumeMounts, pod labels + annotations). Each asserts the
DaemonSet and the values ConfigMap converged, that the chart's own entries survived, and then
syncs a second time and asserts
changed == false.TestSyncRuntimeIsANoOpWithoutSpecChanges— a freshly rendered runtime is not touched(DaemonSet
resourceVersionunchanged).TestSyncRuntimeCoercesUnsafeUpdateStrategy— aRollingUpdateDaemonSet has its strategy fixedfirst and its template left alone; the next reconciliation then syncs the spec.
TestSyncRuntimeWithoutValuesConfigMap— runtimes created withAnnotationDisableRuntimeHelmValueConfighave no last synced state, so sync degrades instead offailing the reconciliation.
TestSyncRuntimeWithoutProfile— a danglingspec.profileNamedegrades gracefully.pkg/utils/resources_test.go—TestTransformInternalResourcesToCoreV1Resourcesplus aTestTransformResourcesRoundTripthat pins the property the sync relies on.No e2e test is added: this repo has no ThinRuntime e2e harness today and building one is
disproportionate here. The equivalent coverage was run by hand against a real cluster instead, below.
Ⅳ. Describe how to verify it
Verified on a real 3-node ACK cluster (Kubernetes v1.36.1), with an in-cluster NFS server, the
addons/nfsprofile, aDataset/ThinRuntimeand a consumer pod that really mounts the NFS —everything
Runningbefore any patch.Reproduction, with the stock
fluidcloudnative/thinruntime-controller:v1.1.0-36f0467. Patch aready
ThinRuntime'sfuse.resources(cpu1→4, memory128Mi→256Mi),fuse.envandfuse.lifecycle, wait 150s. The values ConfigMap and the fuse DaemonSet are byte-identical tobefore; DaemonSet
generationis still1. Onlyspec.fusediffers.With this PR (controller image built from this branch, same unchanged spec the stock controller
had ignored):
cpu: "4",memory: 256Mi, new lifecycle,env: afterthin-fusecontainergeneration1 → 2restartCountunchanged, still on the oldcpu=1templateready=true,restarts=0throughoutimage/imageTag/imagePullPolicy/podMetadatapatchsidecar.istio.io/injectpreservedvolumes/volumeMountspatchthin-fuse-mountandthin-confpreservedFLUID_RUNTIME_TYPE/_NS/_NAMEintactRunningwithcpu=4 mem=256Mi env=afterand the new volumegenerationandresourceVersionunchangedFuseTemplateUpdatedfired exactly 3 times for 3 patchesController logs from the first sync, for reference:
go build ./...,go vet,./pkg/ddc/thin/...,./pkg/common/...and the touchedpkg/utilstests all pass on linux/amd64.
Ⅴ. Special notes for reviews
parseFuseOptionssort is load-bearing, not a drive-by cleanup. Without it this PR wouldupdate the fuse DaemonSet on every reconciliation whenever a runtime has two or more fuse options.
It does mean the rendered
MOUNT_OPTIONSstring may be reordered once for existing runtimes; mountoptions are order-insensitive and it converges immediately.
excluded above, are easy follow-ups if you'd rather have them in one change.
OnDeletebehaviour is intentional and is what the issue asks for: the template convergesdeclaratively, but the operator decides when to roll the pods. The event exists so this is not
silent. Happy to also reflect it in
.statusif reviewers prefer that over an event.fluidcloudnative/nfs:v0.1addon image no longer matchesaddons/nfs/or the current chart — theimage reads the pre-2023
/etc/fluid/config.jsonlayout while the chart mounts the config at/etc/fluid/config/config.json, and the image is alpine +mount -t nfswhileaddons/nfs/docker/builds ubuntu +
fuse-nfs. Followingaddons/nfs/readme.mdas written yields aCrashLoopBackOfffuse pod. I worked around it by inlining the mount script via the profile's
command/args.