KEP-5381: Mutable PersistentVolume Node Affinity #5382

huww98 · 2025-06-06T02:43:09Z

One-line PR description: adding new KEP

Issue link: Mutable PersistentVolume Node Affinity #5381

Other comments:

mowangdk · 2025-06-06T09:43:40Z

keps/sig-storage/5381-mutable-pv-affinity/README.md

+
+1. Change APIServer validation to allow `PersistentVolume.spec.nodeAffinity` to be mutable.
+
+2. Change CSI Specification to allow `ControllerModifyVolume` to return a new accessibility requirement.


Point 2 and 3, though not directly compatible with the title, are connected as upstream and downstream changes. They're currently combined, but if needed, we'll create a new KEP for separation later.

Is this a new field or it can fit into the mutable parameters map? Actually nvm there is the proto change below.

mowangdk · 2025-06-06T09:49:12Z

keps/sig-storage/5381-mutable-pv-affinity/README.md

+These modification can be expressed by `VolumeAttributeClass` in Kubernetes.
+But sometimes, A modification to volume comes with change to its accessibility, such as:
+1. migration of data from one zone to regional storage;
+2. enabling features that is not supported by all the client nodes.


Though it deviates from nodeAffinity's original design（geography related）, this scenario is currently useful for users, hence we include it.

How do we prevent the new update to the nodeAffinity is not compatible with the current node?

Or maybe we have some rules around you can only add wider nodeAffinity support but not shrink it.

I think we just don't enforce this. We ask SPs to not disrupt the running workload. With this promise, we allow SP to return any new nodeAffinity, even if it is not compatible with currently published node. The affinity will be ignored for running pod, just like Pod.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.

sunnylovestiramisu · 2025-06-06T16:23:57Z

keps/sig-storage/5381-mutable-pv-affinity/README.md

+* or being rejected from nodes that actually can access the volume, getting stuck in a `Pending` state.
+
+By making `PersistentVolume.spec.nodeAffinity` field mutable,
+we give storage providers a chance to propagate latest accessibility requirement to the scheduler,


How do we measure the consistency guarantee of the "latest" requirements?

That's a good question, but after some discussion we don't seem to have a suitable solution, especially since there is also an informer cache, and we noticed that the #4876 doesn't mention this problem either, so can we just ignore it?

No, we can't just ignore it. If I understood Sunny's question correctly, its trying to poke at potential race conditions or issues which could lead to unexpected results. For example, (in the current proposal) if two ModifyVolume calls are made in parallel with different accessibility requirements, how do we know what "latest" is? In other words, what are the mechanisms that ensure the actual state of the world reflects the desired state.

These sorts of questions were addressed in KEP 4876 -#4875 (comment).

I think this is fine. The latest requirement is the one corresponding to the mutable_parameters desired by CO passed in ControllerModifyVolumeRequest. In Kubernetes, external-resizer will save the VAC name to PV status after ModifyVolume finishes. It can save the returned requirement to PV just before that.

If anything wrong happens (race, crash, etc.), CO can always issue another ControllerModifyVolume request to fetch the latest topology. This means we should require SP to return accessible_topology if supported. I will update the KEP.

Are we still planning to do ControllerModifyVolume with nodeAffinity changes? I was thinking in CSI meetings there was a discussion about using a new CSI API specific for nodeAffinity.

Yes, but maybe delayed. We want to land this KEP as soon as possible. This should extend the ControllerModifyVolume and not blocking it from GA.

sunnylovestiramisu · 2025-06-06T16:30:29Z

keps/sig-storage/5381-mutable-pv-affinity/README.md

+
+| Condition | gRPC Code | Description | Recovery Behavior |
+|-----------|-----------|-------------|-------------------|
+| Topology conflict | 9 FAILED_PRECONDITION | Indicates that the CO has requested a modification that would make the volume inaccessible to some already attached nodes. | Caller MAY detach the volume from the nodes that are in conflict and retry. |


Is this a infeasible error? What is the retry/rollback story for the users?

Kubernetes does not perform any automatic correction for this, it is exposed to Event as other errors.

Currently, external-resizer should just retry with exponential backoff. In the future, maybe we can retry after any ControllerUnpublishVolume succeeded, maybe after external-resizer and external-attacher combined into the same process.

User should just rollback the VAC to the previous value to cancel the request.

keps/sig-storage/5381-mutable-pv-affinity/README.md

xing-yang · 2025-06-11T15:55:33Z

keps/sig-storage/5381-mutable-pv-affinity/README.md

+  // from a given node when scheduling workloads.
+  // This field is OPTIONAL. If it is not specified, the CO SHOULD assume
+  // the topology is not changed by this modify volume request.
+  repeated Topology accessible_topology = 5;


This should be an alpha_field.

xing-yang · 2025-06-11T15:58:35Z

keps/sig-storage/5381-mutable-pv-affinity/README.md

+We will extend CSI specification to add:
+```protobuf
+message ControllerModifyVolumeResponse {
+  option (alpha_message) = true;


We are moving VolumeAttributesClass to GA in 1.34. That means we need to move ControllerModifyVolume in CSI spec from alpha to GA and cut a CSI spec release before 1.34 release.
I'm wondering if we can move this message to GA while adding a new alpha_field.

I think we have two options to address this issue:

Target the entire KEP in v1.35

Isolate the VAC part, limiting mutable PV node affinity to v1.34, we still got one weeks before enhancement freeze

Maybe we can talk about this on the sig-storage meeting

mowangdk · 2025-06-22T07:06:53Z

keps/sig-storage/5381-mutable-pv-affinity/README.md

+If this happens, the pod will be stuck in a `ContainerCreating` state.
+User will have to manually delete the pod, or using Kubernetes [descheduler](https://github.com/kubernetes-sigs/descheduler) or similar.
+
+We propose to added an `in_progress` field in `ControllerModifyVolumeResponse` to allow more timely affinity update,


Before altering the topology to an unfeasible state, consider adding clearer guidance on handling scheduled pods.

mowangdk · 2025-06-22T07:13:38Z

keps/sig-storage/5381-mutable-pv-affinity/README.md

+
+  // Indicates whether the CO allows the SP to update the topology
+  // as a part of the modification.
+  bool allow_topology_updates = 4;


We also need a new parameters in mutable_parameters right? Let's include an example demonstrating the usage of ControllerModifyVolumeRequest and ControllerModifyVolumeResponse with allow_topology_updates.

huww98 · 2025-07-08T13:27:27Z

According to the last zoom meeting, I've made the following changes:

No major changes to the current design. Added a sentence to CSI spec to instruct SP to only loose the topology.
I propose not to support SPs that don't know the currently attached nodes.
I propose not to support specifying topology in VAC.

Please see the KEP for the reasons and details.

sunnylovestiramisu · 2025-08-07T22:23:46Z

Consider a scenario where the scheduler selects "node-A" for a pod and annotates the corresponding PVC with volume.kubernetes.io/selected-node: node-A. If an administrator concurrently modifies the nodeAffinity of the underlying PV to only allow scheduling on "node-B," a conflict will arise. The pod, when it attempts to start on "node-A," may fail if the volume cannot be mounted there due to the updated affinity rules.

How do we resolve this?

Or do we have a list of scenarios that is considered against the guideline so the admin or user should not be doing that?

huww98 · 2025-08-08T06:38:28Z

@sunnylovestiramisu I think this is discussed in the Caveats section as "Possible race condition".

I've just learnt we reject pod when attachment limit is exceeded. I think we can use similar method to deal with the mis-scheduled pods, i.e., when we detect PV node affinity violation and a specific error code at ControllerPublishVolume, we mark the Pod as Failed and hope some controller to re-create it.

xing-yang · 2025-09-24T19:20:10Z

keps/sig-storage/5381-mutable-pv-affinity/README.md

+`PersistentVolume.spec.nodeAffinity.required` should works like
+`Pod.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution`.
+It is never re-evaluated when the pod is already running.
+It is storage provider's responsibility to ensure that the running workload is not interrupted.


If the PVC is used by a Pod and user changes the PersistentVolume.spec.nodeAffinity, the nodeAffinity rule may be out of sync with the node where the pod is running?

Yeah, we think the modified PersistentVolume.spec.nodeAffinity shouldn't impact running pods; its will effects during the next scheduling or when a new pod claims it.

Whoever modifies nodeAffinity should ensure this will not affect the running pods.

xing-yang · 2025-09-24T19:23:54Z

keps/sig-storage/5381-mutable-pv-affinity/README.md

+### Integrate with the CSI spec and VolumeAttributeClass
+
+We have proposed the plan to integrate in the previous version of this KEP.
+But we did not reach consensus due to lack of SP want to implement this feature.


Can you summarize what the solution is?

Should we propose a separate KEP to discuss this to avoid confusion in this one? (btw we already write about this part in previous version of this KEP)

For that previous version, please refer to the previous commit in this PR

xing-yang · 2025-09-24T19:24:57Z

keps/sig-storage/5381-mutable-pv-affinity/README.md

+
+Type A node only supports cloud_ssd, while Type B node supports both cloud_ssd and cloud_essd.
+We will only allow the modification if the volume is attached to type B nodes.
+And I want to make sure the Pods using new cloud_essd volume not to be scheduled to type A nodes.


Who will be updating PersistentVolume.spec.nodeAffinity? Some downstream controller?

Yeah, We initially planned to implement this in VAC, but due to unresolved questions from last few month's meetings, we've decided to focus on making pv nodeaffinity mutable for now.

Eventually we want external-resizer to update it. Currently, we are intended to do this in our fork of external-resizer, or with a dedicated controller.

huww98 · 2025-09-30T08:52:05Z

I've done a prototype implementation at kubernetes/kubernetes#134339

Also added kubelet behavior when pod with incompatible PV nodeAffinity running on the node, tested with the implementation mentioned above.

filled Production Readiness Review Questionnaire

mowangdk · 2025-10-01T12:13:19Z

keps/sig-storage/5381-mutable-pv-affinity/README.md

+  we fail the Pods that use them, since we are sure the Pods have never been running.
+  (see [Handling race condition](#handling-race-condition) below)
+- For CSI drivers with `requiresRepublish` set to true, we will stop calling NodePublishVolume periodically. and an event is emitted.
+- For CSI drivers with `requiresRepublish` set to false, an event is emitted on kubelet restart. Otherwise the pod should continue to run.


I think we should periodically emit the error event since it doesn't cause pod failures, but it may block volume-related node operations like filesystem expansion. Alerting the user and guiding them to fix PV affinity is necessary.

wojtek-t · 2025-10-14T09:30:27Z

keps/sig-storage/5381-mutable-pv-affinity/README.md

+
+Whoever modifies the `PersistentVolume.spec.nodeAffinity` field should ensure that
+no running Pods on nodes with incompatible labels are using the PV.
+Kubernetes will not verify this. It is expensive and racy.


I'm probably missing something, so let me try to understand the proposal better.

Imagine that my PV is attached to node:

node N1 where pod P1 using it is running

node N2 where pod P2 using it is running
I want to update nodeAffinity in a way that N2 no longer satisfies the new affinity.

What happens to pod P2 if I don't delete it manually?

What happens to the PV itself after its nodeAffinity is updated. Will it be automatically detached from N2?

What happens to pod P2 if I don't delete it manually?

It will continue to run if the volume is actually accessible, with some known limitations listed in the next paragraph.

What happens to the PV itself after its nodeAffinity is updated. Will it be automatically detached from N2?

No. We will still detach it from N2 after P2 deleted. Basically the new affinity only affects new pods.

Can you add that explicitly to the doc to make it clear?

wojtek-t · 2025-10-14T09:32:19Z

keps/sig-storage/5381-mutable-pv-affinity/README.md

+
+This feature involves changes to the kubelet, and APIServer. But they are not strongly coupled.
+
+an n-3 kubelet will not able to fail the mis-scheduled pods. User can still manually delete the pods. Otherwise it should be fine.


So what happens to those pods? In what state do they stuck? Is it ContainerCreating?

Ensuring that kubelet is able to handle that correctly (i.e. all supported kubelets handle that) seems like at least a GA blocker to me.

So what happens to those pods? In what state do they stuck? Is it ContainerCreating?

Yes, ContainerCreating.

seems like at least a GA blocker to me.

If user does not actually modify PV node affinity, there will be no such mis-scheduled pods. User should upgrade and enable feature gate for all components before using this new feature, right?

keps/sig-storage/5381-mutable-pv-affinity/README.md

wojtek-t · 2025-10-14T09:36:55Z

keps/sig-storage/5381-mutable-pv-affinity/README.md

+If a pod being scheduled to a node that is incompatible with the PV's nodeAffinity, the pod will fail.
+Previously, it will be stuck at `ContainerCreating` status.
+
+This should be rare, since we don't allow PV nodeAffinity to be updated,


I don't understand this - the whole goal of this KEP is to allow to update it.

I mean this should be rare before enabling this feature. So that when the feature is enabled, it is rare that some pod existing in the cluster will be affected by the behavior change.

wojtek-t · 2025-10-14T09:39:37Z

keps/sig-storage/5381-mutable-pv-affinity/README.md

+PV `spec.nodeAffinity` becomes mutable.
+
+If a pod being scheduled to a node that is incompatible with the PV's nodeAffinity, the pod will fail.
+Previously, it will be stuck at `ContainerCreating` status.


I'm wondering if we shouldn't actually roll this out with two separate feature-gates:

the kubelet one that handles the mis-scheduled pods
Ensure that this graduated to GA on all supported versions, before graduating the apiserver FG.

the FG that allows for mutating nodeAffinity.

That may be an option. But we can instruct user not to use this feature before kubelet rollout finished to avoid the race condition.

Another KEP also embrace similar mechanism to handle race condition when node volume count exceeded. It also use one feature gate.

If we do use two feature-gate, should we track them in two KEP?

The different that I see between these KEPs is that:

in the mutable allocatable, the whole workflow is generally through system components, there is no user in the picture (the update is triggered by the CSI driver); so as an administrator I have full control over it

in this particular KEP:
(a) we additionally have a user in the picture [a user is the one that can trigger nodeAffinity change]
(b) they can do that in a setup where kubelet will keep the pod stuck in ContainerCreating

(b) is my primary concern here.

No - definitely not two keps. There are different keps with multiple FGs.

For that KEP, there is likely a human operation that triggers the allocatable count change. Someone may re-configured the node, attached another device to the node, etc.

And note this should be rare, it is a race condition that only happens when user update PV and create Pod at the same time. And is easily resolvable by upgrade kubelet afterwards or manually delete Pod.

If I understand this correctly, we need to wait for at least 3 release between enable these two feature gate. That is not a short time. Would it be too careful? Since the enhancement freeze is near. Can we merge this PR and keep the discussion in its dedicated PR? I think this is not blocking for alpha.

For me, changing PV object is an operation that is highly privileged, very similar to Node change. I think we could assume that users who have such permissions know enough about what they can cause by such a change. They may choose to ensure that no Pod users the PV, manually resolve the race, or postpone their PV modification until all kubelets have the feature gate enabled.

OK fair. I would like these thoughts to be better reflected in the KEP.
But let's work on updating the KEP to better reflect all those tradeoffs before Beta - I will let it go for Alpha to not block you for the next 4 months.

keps/sig-storage/5381-mutable-pv-affinity/README.md

wojtek-t · 2025-10-16T10:17:42Z

keps/sig-storage/5381-mutable-pv-affinity/README.md

+
+Whoever modifies the `PersistentVolume.spec.nodeAffinity` field should ensure that
+no running Pods on nodes with incompatible labels are using the PV.
+Kubernetes will not verify this. It is expensive and racy.


Can you add that explicitly to the doc to make it clear?

wojtek-t · 2025-10-16T10:27:55Z

keps/sig-storage/5381-mutable-pv-affinity/README.md

+5. KCM/external-attacher attaches the volume to the node, and find the affinity mismatch.
+
+If this happens, the pod will be stuck in a `ContainerCreating` state.
+Kubelet should detect this contidion and reject the pod.


I would like to see feedback from someone from the SIG node about that.
It sounds fine to me, but I'm not an expert here.

@SergeyKanzhelev

We (sig-storage) are marking pods as Failed when kubelet detects the CSI driver can't attach more volumes to it here

Similarly to this KEP, it solves a race condition and we expect the owner (Deployment/StatefulSet) re-creates the Pod.

wojtek-t · 2025-10-16T10:44:46Z

keps/sig-storage/5381-mutable-pv-affinity/README.md

+PV `spec.nodeAffinity` becomes mutable.
+
+If a pod being scheduled to a node that is incompatible with the PV's nodeAffinity, the pod will fail.
+Previously, it will be stuck at `ContainerCreating` status.


The different that I see between these KEPs is that:

in the mutable allocatable, the whole workflow is generally through system components, there is no user in the picture (the update is triggered by the CSI driver); so as an administrator I have full control over it

in this particular KEP:
(a) we additionally have a user in the picture [a user is the one that can trigger nodeAffinity change]
(b) they can do that in a setup where kubelet will keep the pod stuck in ContainerCreating

(b) is my primary concern here.

No - definitely not two keps. There are different keps with multiple FGs.

wojtek-t · 2025-10-16T11:11:33Z

keps/sig-storage/5381-mutable-pv-affinity/README.md

+For Pods stuck in ContainerCreating due to storage provider unable to attach the volume to the scheduled node,
+The Pod will be rejected by kubelet and re-created at the correct node.
+For Pods stuck in Pending due to no suitable node available,
+scheduler will retry to schedule the Pod according to the updated nodeAffinity.


I think this is answer is misleading, as many of those are indirect consequences of this feature.

I think the answer really is:

nodeAffinity can now be updated for existing volumes

pods that cannot be run due volume that can't be attached are now being failed by kubelet

Yes, you are right that many of those are indirect. I will also add the direct part as you describe.

But those are written to be corresponding to the "Motivation" part of this doc. I'm interpreting the subtitle "working for their instance" as meeting the initial motivation when proposing this change.

jsafrane · 2025-10-16T16:56:53Z

I think it's good enough for alpha. I actually like the extra thoughts about mis-scheduled pods and failing them in kiubelet.
/lgmt
/approve

jsafrane · 2025-10-16T18:17:04Z

/lgtm

wojtek-t · 2025-10-16T18:30:16Z

I think it's acceptable for Alpha from PRR POV.

That said, we should much better reflect all the tradeoffs we're making (e.g. PV update being previliged operation etc.) in the KEP before Beta - the way it is currently described doesn't meet the Beta bar, so I will be more conservative the next release.

@huww98 @jsafrane ^^

/approve PRR

k8s-ci-robot · 2025-10-16T18:30:26Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: huww98, jsafrane, wojtek-t

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~keps/prod-readiness/OWNERS~~ [wojtek-t]
~~keps/sig-storage/OWNERS~~ [jsafrane]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mutable-pv-affinity init

006d6e6

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 6, 2025

k8s-ci-robot requested review from saad-ali and xing-yang June 6, 2025 02:43

k8s-ci-robot added kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory sig/storage Categorizes an issue or PR as relevant to SIG Storage. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 6, 2025

huww98 mentioned this pull request Jun 6, 2025

Mutable PersistentVolume Node Affinity #5381

Open

4 tasks

mowangdk reviewed Jun 6, 2025

View reviewed changes

sunnylovestiramisu reviewed Jun 6, 2025

View reviewed changes

wojtek-t reviewed Jun 9, 2025

View reviewed changes

keps/sig-storage/5381-mutable-pv-affinity/README.md Show resolved Hide resolved

xing-yang reviewed Jun 11, 2025

View reviewed changes

address comments from sig-storage meeting

0602a5f

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 13, 2025

wojtek-t self-assigned this Jun 13, 2025

huww98 mentioned this pull request Jun 20, 2025

KEP-4650: StatefulSet Support for Updating Volume Claim Template #4651

Open

mowangdk reviewed Jun 22, 2025

View reviewed changes

iltyty and others added 3 commits July 1, 2025 22:14

Add new GRPC alternative

b433a3f

not support accessibility_requirements for now

487defd

add typical workflow

e23b658

huww98 force-pushed the mutable-pv-affinity branch 2 times, most recently from e096f8b to 11e029a Compare July 8, 2025 13:21

Address comments from last meeting

51ac059

huww98 force-pushed the mutable-pv-affinity branch from 11e029a to 51ac059 Compare July 9, 2025 10:35

iltyty mentioned this pull request Jul 21, 2025

Modify ControllerModifyVolume to support KEP-5381 container-storage-interface/spec#592

Closed

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jul 28, 2025

Remove CSI things

95584e0

huww98 force-pushed the mutable-pv-affinity branch from 7c76e1a to 95584e0 Compare July 28, 2025 10:07

reject mis-scheduled pod

19d29ec

huww98 force-pushed the mutable-pv-affinity branch from 29c805f to 19d29ec Compare August 8, 2025 07:17

xing-yang reviewed Sep 24, 2025

View reviewed changes

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Sep 30, 2025

add kubelet behavior with incompatible nodeAffinity

d5181dc

filled Production Readiness Review Questionnaire

huww98 force-pushed the mutable-pv-affinity branch from 8984c84 to d5181dc Compare September 30, 2025 08:52

mowangdk reviewed Oct 1, 2025

View reviewed changes

wojtek-t reviewed Oct 14, 2025

View reviewed changes

wojtek-t reviewed Oct 16, 2025

View reviewed changes

Address comments

5c0d167

huww98 force-pushed the mutable-pv-affinity branch from e315d6c to 5c0d167 Compare October 16, 2025 16:50

k8s-ci-robot assigned jsafrane Oct 16, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 16, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 16, 2025

k8s-ci-robot merged commit ca62dd5 into kubernetes:master Oct 16, 2025
4 checks passed

k8s-ci-robot added this to the v1.35 milestone Oct 16, 2025


		1. Change APIServer validation to allow `PersistentVolume.spec.nodeAffinity` to be mutable.

		2. Change CSI Specification to allow `ControllerModifyVolume` to return a new accessibility requirement.


		This feature involves changes to the kubelet, and APIServer. But they are not strongly coupled.

		an n-3 kubelet will not able to fail the mis-scheduled pods. User can still manually delete the pods. Otherwise it should be fine.

KEP-5381: Mutable PersistentVolume Node Affinity #5382

KEP-5381: Mutable PersistentVolume Node Affinity #5382

Conversation

huww98 commented Jun 6, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sunnylovestiramisu Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sunnylovestiramisu Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

huww98 commented Jul 8, 2025

Uh oh!

sunnylovestiramisu commented Aug 7, 2025

Uh oh!

huww98 commented Aug 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

huww98 commented Sep 30, 2025

Uh oh!

mowangdk Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sunnylovestiramisu Jun 6, 2025 •

edited

Loading

sunnylovestiramisu Jun 6, 2025 •

edited

Loading

mowangdk Oct 1, 2025 •

edited

Loading