Default Node Affinity doesn't match on node created by AWS (provider Amazon EC2)

[h3d-haivq]

Basic Info:
Rancher Version: 2.3.2
Kubernetes Version: v1.16.3-rancher1-1
Provider: Amazon EC2

Brief Description:
Pod which used the PVC created by this module cannot be scheduled due to the auto-generated PV's Node Affinity Rule: kubernetes.io/hostname = {{someNodeName}} doesn't match with the actual value of the node.

How to reproduce

• Spin up a cluster with provider is Amazon EC2
• Install this module and try to run the example
• Pod volume-test cannot be scheduled with this message:

0/4 nodes are available: 1 node(s) had taints that the pod didn't tolerate, 3 node(s) had volume node affinity conflict.

More description
The Persistent Volume which is created by this module has the kubernetes.io/hostname value which is equal to the Private DNS value of the instance that run. But actually the kubernetes.io/hostname label has a value of the instance's name and cannot be edited

More image:

AWS Instance board:

Node's label:

Node Affinity from the PV:

[peregrinator]

We're experiencing the same issue. The PV is provisioned correctly but the node affinity label is not correct and there's no way to change it.

Any advice or workarounds here?

[peregrinator]

Follow up here: the problem (at least for our cluster) is that local-path-provisioner isn't working with the Rancher node pool way of setting the kubernetes.io/hostname label.

When using node pools RKE is setting kubernetes.io/hostname to be a variation of the node pool name. Eg. node pool workers-a results in hostname settings of workers-a1, workers-a2, etc.

This obviously conflicts with what local-path-provisioner is expecting as a value. So it appears we need a way to specify a label to use for node affinity or a way to modify what RKE uses to set the value of kubernetes.io/hostname when provisioning a node.

Any thoughts on this @yasker?

[yasker]

@peregrinator Thanks for the analysis. It sounds like this line is the culprit:

local-path-provisioner/provisioner.go

Line 179 in 655eac7

basePath, err := p.getRandomPathOnNode(node.Name)

Currently, we just use node.Name for the node affinity. In the case of kubernetes.io/hostname is different from node.Name, it seems we should use the value of kubernetes.io/hostname instead.

But I am busy with the upcoming Longhorn GA at the moment, so I won't able to address this issue anytime soon. It would be great if someone else can help.

[lauchokyip]

/assign

[lauchokyip]

I can try to work on this @yasker

[lauchokyip]

It's done from here #186