Empty Statefulset nodeAffinity gets build with string value "null" in complex scenario

[sbuliarca]

What happened?

Starting with kustomize 5.0 we're having this issue where in a statefulset, an empty nodeAffinity gets built with the output as:

nodeAffinity: "null"

which is invalid, as it is a string with the value "null"

We worked around it by patching to drop the empty spec, but maybe this hides some other bug.

What did you expect to happen?

The empty nodeAffinity spec should be removed or built as:

nodeAffinity: null

How can we reproduce it (as minimally and precisely as possible)?

The scenario is a bit complex, so I put it in this repo: https://github.com/sbuliarca/kustomize-bug-replication/

There are several key pieces to the puzzle, and without them all it can not be reproduced:

• the empty nodeAffinity spec
• there must be an inline patch to the stateful set
• there must be at least 2 other file patches:

Expected output

nodeAffinity: null

Actual output

nodeAffinity: "null"

Kustomize version

5.0.3

Operating system

MacOS

[k8s-ci-robot]

This issue is currently awaiting triage.

SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the triage/accepted label.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[natasha41575]

/assign @varshaprasad96

[varshaprasad96]

Dug a bit deep into the issue. From the findings as @sbuliarca mentioned it looks like this happens in special cases where we have 2 patches. The strategic merge patch is applied two times. After the first patch, the output seems to be on the lines of:

{"apiVersion":"apps/v1","kind":"StatefulSet","metadata":{"annotations":{"internal.config.kubernetes.io/previousKinds":"StatefulSet","internal.config.kubernetes.io/previousNames":"test-shared","internal.config.kubernetes.io/previousNamespaces":"default"},"labels":{"app.kubernetes.io/name":"test","dummy":"test"},"name":"test-shared"},"spec":{"selector":{"matchLabels":{"app.kubernetes.io/name":"test-shared"}},"serviceName":"test-shared-headless","template":{"metadata":{"labels":{"app.kubernetes.io/name":"test-shared"}},"spec":{"affinity":{"nodeAffinity":null,"podAffinity":null},"containers":[{"env":[{"name":"DUMMY","value":"dummy-val"}],"image":"alpine","name":"test"}]}}}}

We can see that the nodeAffinity and podAffinity have the values set to null as expected. Looking into the generator, this happens right because while parsing and unmarshalling the yaml, the yaml.RNode for both these fields seems to be considered as an empty node.

Whereas when the second patch is performed, the parser considers both these nodes as Scalars - yaml.ScalarNode to be specific. Hence the values after running the generator are considered as "string", which is why it is transformed into "null".

However, when I apply a single patch with nodeAffinity explicitly set to null instead of leaving it empty, this does not seem to happen. I'm not sure of the fix here, this seems to be a special edge case.

[varshaprasad96]

Based on how deep I could dig, looks like this seems to be an issue with yaml parser rather than Kustomize. This maybe an issue with the yaml library (https://github.com/go-yaml/yaml) which is being used in kustomize.

[natasha41575]

If this is an issue with the yaml library, then it depends on kubernetes-sigs/yaml#76 before we can properly fix this, and then we should file and fix the issue in the new go yaml fork when it's ready.

In the meantime, I think we can't do much about issues with the yaml parser in kustomize, so I will close this for now. You can keep an eye on kubernetes-sigs/yaml#76 for the state of progress, I can try to find time to update that one soon.

[matthewhughes934]

Based on how deep I could dig, looks like this seems to be an issue with yaml parser rather than Kustomize. This maybe an issue with the yaml library (https://github.com/go-yaml/yaml) which is being used in kustomize.

I believe this has the same underlying cause as #5031 I've opened a PR would a proposed solution #5519 which has more details