NullPointerException on finalizer removal when namespace is deleted concurrently

[Donnerbart]

Bug Report

What did you do?

I had a custom operator running in the default namespace.

I've create a test namespace and a custom resource, that is reconciled by the operator.

kubectl create namespace test && kubectl apply -f custom-resource.yml

Then I deleted the custom resource and the namespace directly after each other:

kubectl delete custom-resource.example.com test-example && kubectl delete namespace test

What did you expect to see?

I expected the operator to handle the cleanup and finalizer removal gracefully without errors.

What did you see instead? Under which circumstances?

I saw a NullPointerException in the ReconciliationDispatcher:

15:03:16.744 [ReconcilerExecutor-hivemq-controller-56] ERROR i.j.o.p.e.ReconciliationDispatcher - Error during event processing ExecutionScope{ resource id: ResourceID{name='test-example', namespace='test'}, version: 38510} failed.
java.lang.NullPointerException: Cannot invoke "io.fabric8.kubernetes.api.model.HasMetadata.removeFinalizer(String)" because "r" is null
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.lambda$handleCleanup$1(ReconciliationDispatcher.java:297)
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.conflictRetryingUpdate(ReconciliationDispatcher.java:339)
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleCleanup(ReconciliationDispatcher.java:297)
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:87)
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:62)
	at io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:414)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

Environment

Kubernetes cluster type:

Vanilla K8s running locally in Docker via:

~$ minikube version
minikube version: v1.31.1
commit: fd3f3801765d093a485d255043149f92ec0a695f

$ Mention java-operator-sdk version from pom.xml file

12:36:46.710 [Quarkus Main Thread] INFO  i.q.o.runtime.OperatorProducer - Quarkus Java Operator SDK extension 6.1.0 (commit: f79f1b6 on branch: f79f1b6ffcb49e453b697f37aebc27b34d45482c) built on Thu May 25 00:28:33 CEST 2023
12:36:47.110 [Quarkus Main Thread] INFO  i.javaoperatorsdk.operator.Operator - Operator SDK 4.3.5 (commit: 7341433) built on Tue Apr 04 10:16:12 CEST 2023 starting...
12:36:47.110 [Quarkus Main Thread] INFO  i.javaoperatorsdk.operator.Operator - Client version: 6.6.2

$ java -version

openjdk version "20.0.1" 2023-04-18
OpenJDK Runtime Environment Homebrew (build 20.0.1)
OpenJDK 64-Bit Server VM Homebrew (build 20.0.1, mixed mode, sharing)

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.4", GitCommit:"fa3d7990104d7c1f16943a67f11b154b71f6a132", GitTreeState:"clean", BuildDate:"2023-07-19T12:14:48Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.2", GitCommit:"7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647", GitTreeState:"clean", BuildDate:"2023-05-17T14:13:28Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"linux/amd64"}

Possible Solution

I'll provide a PR with a possible solution.

Additional context

I was only able to reproduce this once, but the stacktrace was clear enough to figure out a possible root cause.