Using Kubernetes v1.20.0, getting "unexpected error getting claim reference: selfLink was empty, can't make reference"

[nealman13]

Using Kubernetes v1.20.0

Attempting to create pvc's, they all remain in "pending" status

Doing a kubectl logs nfs-client-provisioner gives this:

I1210 14:42:01.396466 1 leaderelection.go:194] successfully acquired lease default/fuseim.pri-ifs
I1210 14:42:01.396534 1 controller.go:631] Starting provisioner controller fuseim.pri/ifs_nfs-client-provisioner-64b7476494-p4fcm_dcfca333-3af5-11eb-8248-5aed4ceb7af7!
I1210 14:42:01.496922 1 controller.go:680] Started provisioner controller fuseim.pri/ifs_nfs-client-provisioner-64b7476494-p4fcm_dcfca333-3af5-11eb-8248-5aed4ceb7af7!
I1210 14:42:01.497152 1 controller.go:987] provision "default/pvc1" class "managed-nfs-storage": started
I1210 14:42:01.497157 1 controller.go:987] provision "default/test-claim" class "managed-nfs-storage": started
E1210 14:42:01.500487 1 controller.go:1004] provision "default/pvc1" class "managed-nfs-storage": unexpected error getting claim reference: selfLink was empty, can't make reference
E1210 14:42:01.500502 1 controller.go:1004] provision "default/test-claim" class "managed-nfs-storage": unexpected error getting claim reference: selfLink was empty, can't make reference

selfLink has been disabled in v1.20.0
kubernetes/enhancements#1164

[fangedhex]

I can confirm that I have this problem too on my new cluster using Kubernetes v1.20 with those same logs/errors when creating a test pvc.

[nealman13]

Current workaround is to edit /etc/kubernetes/manifests/kube-apiserver.yaml

Under here:

spec:
  containers:
  - command:
    - kube-apiserver

Add this line:
- --feature-gates=RemoveSelfLink=false

The do this:
kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.yaml
kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.yaml

(I had to do it twice to get it to work)

[petermicuch]

Yep, exactly the same issue. In k8s 1.20 they have removed selfLink as per release notes.

Adding selfLink manually does not help, as that is removed by API server immediatelly.

[fangedhex]

I used @nealman13's workaround until a fix is made and it works now. Thanks !

[petermicuch]

Just to add to what @nealman13 provided. Before first leader election log, there is this error:

E1210 15:41:30.047572       1 event.go:259] Could not construct reference to: '&v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"nfss", SelfLink:"", UID:"75df6842-e437-4604-9034-2d4d932373be", ResourceVersion:"19431", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63743203519, loc:((nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\"eb-af59-dac986fde3f6\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2020-12-10T15:41:30Z\",\"renewTime\":\"2020-12-10T15:41:30Z\",\"leaderTransitions\":1}"}, OwnerReference)(nil), Finalizers:[]string(nil), ClusterName:""}, Subsets:[]v1.EndpointSubset(nil)}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'Le2d51-3afe-11eb-af59-dac986fde3f6 became leader'
I1210 15:41:30.047754       1 leaderelection.go:194] successfully acquired lease default/fuseim.pri-ifs

[petermicuch]

I think the problem is in using the old client-go by this replace, but I am not 100% sure if that is the only legacy we pull here.

Same problem was discussed in #94660 and this was suggested solution.

[james-d-elliott]

Just a note about /etc/kubernetes/manifests/kube-apiserver.yaml. When running the control-plane as an in-cluster service you should not need to apply changes to this file ever (via kubectl apply). As soon as you save the change to disk the apiserver pods will detect it and should conduct a Rolling Update. You can watch the progress with watch kubectl get pods --namespace kube-system. You can confirm the behavior after deleting the provisioner pod and letting it restart.

[iming0319]

The workaround will no longer work after 1.20.
"selfLink is a URL representing this object. Populated by the system. Read-only. DEPRECATED Kubernetes will stop propagating this field in 1.20 release and the field is planned to be removed in 1.21 release."