Patch of resource fails due to 'the server rejected our request due to an error in our request'

[tomdw]

Bug Report

What did you do?

Written a reconciler for a Kubernetes resource which patches the resource during cleanup (probably to remove the finalizer).

What did you expect to see?

Succesfull patch of the resource without any ERROR log

What did you see instead? Under which circumstances?

The Patch logs an ERROR due to a 'the server rejected our request due to an error in our request' error from kubernetes, but t the same time the cleanup continues and seems to remove the resource. If it succeeds I do not expect an ERROR log.

ERROR Error during event processing ExecutionScope{ resource id: ResourceID{name='my-resource', namespace='my-namespace'}, version: 2616628} [ X my-resource in my-namespace ]
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://192.168.194.129:443/apis/be.mygroup/v1/namespaces/my-namespace/x/my-resource. Message: the server rejected our request due to an error in our request. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[], group=null, kind=null, name=null, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=the server rejected our request due to an error in our request, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}). 
	at io.fabric8.kubernetes.client.KubernetesClientException.copyAsCause(KubernetesClientException.java:205) 
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:507) 
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524) 
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handlePatch(OperationSupport.java:419) 
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handlePatch(OperationSupport.java:397) 
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handlePatch(BaseOperation.java:764) 
	at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.lambda$patch$2(HasMetadataOperation.java:231) 
	at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.patch(HasMetadataOperation.java:236) 
	at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.edit(HasMetadataOperation.java:65) 
	at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.edit(HasMetadataOperation.java:44) 
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher$CustomResourceFacade.patchResourceWithoutSSA(ReconciliationDispatcher.java:420) 
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.conflictRetryingPatch(ReconciliationDispatcher.java:368) 
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleCleanup(ReconciliationDispatcher.java:266) 
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:95) 
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:68) 
	at io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:467) 
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 
	at java.base/java.lang.Thread.run(Unknown Source) 
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://192.168.194.129:443/apis/be.mygroup/v1/namespaces/my-namespace/x/my-resource. Message: the server rejected our request due to an error in our request. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[], group=null, kind=null, name=null, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=the server rejected our request due to an error in our request, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:642) 
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:622)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:582) 
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:549) 
	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source) 
	at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$10(StandardHttpClient.java:141) 
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) 
	at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source) 
	at java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source)
	at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:51)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) 
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) 
	at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source) 
	at java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source)
	at io.fabric8.kubernetes.client.vertx.VertxHttpRequest.lambda$consumeBytes$1(VertxHttpRequest.java:120) 
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:270) 
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:252) 
	at io.vertx.core.http.impl.HttpEventHandler.handleEnd(HttpEventHandler.java:76) 
	at io.vertx.core.http.impl.HttpClientResponseImpl.handleEnd(HttpClientResponseImpl.java:250)
	at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.lambda$new$0(Http1xClientConnection.java:421) 
	at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:279) 
	at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:157) 
	at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleEnd(Http1xClientConnection.java:731)
	at io.vertx.core.impl.ContextImpl.lambda$execute$7(ContextImpl.java:329) 
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) 
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) 
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) 
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994) 
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) 
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) 
	... 1 common frames omitted | plugin-transformer-cloudfunction-operator

Environment

Kubernetes cluster type:

vanilla

$ Mention java-operator-sdk version from pom.xml file

5.0.4

$ java -version

21

$ kubectl version

1.30.7

Possible Solution

Might be related to which kind of patch is used? See https://stackoverflow.com/questions/57480205/error-while-applying-json-patch-to-kubernetes-custom-resource where a similar problem is described and a merge patch is to be used to avoid such a thing?

Additional context

N/A

[csviri]

Hi @tomdw , could you pls create a simple reproducer, we did not see this before, and the finalizer removal is covered in quite a lot of tests. So would be good to see the details.

[afalhambra-hivemq]

I randomly got the same error and stacktrace when patching the CR status in a regular reconciliation loop (no resource clean-up call), but only when SSA is disabled for patching the primary resource.

.withUseSSAToPatchPrimaryResource(false)
.withSSABasedCreateUpdateMatchForDependentResources(true)

Will try to provide you with a reproducer

[csviri]

Thx, just a note we don't do anything special there, just calling the client that calls the API.

Note that a common issues is when you mix ssa patch with non ssa patch on resources, make sure you settle on one or the other.

[afalhambra-hivemq]

Note that a common issues is when you mix ssa patch with non ssa patch on resources, make sure you settle on one or the other.

mmm, right, then why not just keep one single entry point/method to configure SSA? Otherwise having two withSSAxxx method may lead to wrong configuration, or?

[csviri]

We wanted to be backwards compatible in this case. Also not that patching the primary resource is part of the low level API, and DependentResources is something that you might use or not. Also there is difference between managing patching primary and some other resource, those does not effect each other.

So this level of granularity seems to right for me.

[csviri]

Note that a common issues is when you mix ssa patch with non ssa patch on resources, make sure you settle on one or the other.

mmm, right, then why not just keep one single entry point/method to configure SSA? Otherwise having two withSSAxxx method may lead to wrong configuration, or?

Ahh sorry, misunderstanding, this is how I mean it: if you mix that on same resource. Thus had a resource managed before without SSA and you switch manage to SSA of that same resource.

[afalhambra-hivemq]

Understood. Thanks for clarifying @csviri.
Just to be clear here in this case.
I get random failures with the same 422 error code and same the server rejected our request due to an error in our request error message with the following operator configuration:

.withUseSSAToPatchPrimaryResource(false)
.withSSABasedCreateUpdateMatchForDependentResources(true)

Stacktrace in my case is slight different since for me this is not happening when cleaning-up a resource but in a regular reconciliation loop.

[tomdw]

@csviri @afalhambra-hivemq I converted our operators to use SSA and thus not use .withUseSSAToPatchPrimaryResource(false) anymore. However on cleanup I still get the error mentioned in this issue. Looking into the stacktrace a bit I noticed that in ReconciliationDispatcher.handleCleanup the conflictRetryingPatch method is called with forceNotUseSSA set to true. This implies that for cleanup SSA is never used while the reconciler itself handles the resource with SSA for updates etc. Sure this is the intended behaviour because then you always have a mix of both that might result in the error?

an additional property of the setup I have is that the same custom resource is reconciled by 2 Reconciler processes each creating another dependent resource and thus needing to cleanup both. When I deliberately let 1 of the reconciler processes wait until only its own finalizer is still present on the resource, and reschedule if another is still present. Then the error does not occur. So it might be some kind of concurrent modification problem where both reconcilers try to remove their finalizer at the same time? however I would have expected another error, not a 422, but a 409 I think?

[csviri]

@tomdw as far as I remember it was forced to not use SSA since there were cases even when SSA was used it was not removing finalizers. So yeah, SSA is not that reliable in general.

however I would have expected another error, not a 422, but a 409 I think

Probably because it is JSON Patch used for this, and that is addressing the element by index:
https://stackoverflow.com/questions/27439986/what-is-the-json-patch-format-to-remove-an-element-from-an-array

But you might be right that this can cause an issue.

Do I understand correctly that it does not resolve in all cases, even eventually?

[csviri]

So would be nice to create an Integration test that showcases this multiple finalizer issue. Maybe you could add a PR @tomdw ?
If not will I will create one.

[tomdw]

@csviri if you mean with 'resolve in all cases' that when the code retries that it succeeds eventually, then yes it succeeds but has logged these error stack traces in the mean time. And when I let one reconciler wait for the other's finalizer to be removed then the error does not occur anymore.

As you know the code base a lot more than I do, it might be more efficient if you take on the PR. Have not got the time at the moment.