io.fabric8.kubernetes.client.informers.impl.cache.Reflector - listSyncAndWatch failed

[vicozohili]

Suddenly getting this error:

io.fabric8.kubernetes.client.informers.impl.cache.Reflector - listSyncAndWatch failed for v1/namespaces//pods, will stop
java.util.concurrent.CompletionException: java.util.concurrent.RejectedExecutionException
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1159)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$10(StandardHttpClient.java:142)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:51)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$OkHttpAsyncBody.doConsume(OkHttpClientImpl.java:136)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.util.concurrent.RejectedExecutionException
at io.fabric8.kubernetes.client.utils.internal.SerialExecutor.execute(SerialExecutor.java:47)
at io.fabric8.kubernetes.client.informers.impl.cache.SharedProcessor.execute(SharedProcessor.java:179)
at io.fabric8.kubernetes.client.informers.impl.cache.Reflector.lambda$null$4(Reflector.java:136)
at io.fabric8.kubernetes.client.informers.impl.cache.ProcessorStore.retainAll(ProcessorStore.java:116)
at io.fabric8.kubernetes.client.informers.impl.cache.Reflector.lambda$listSyncAndWatch$6(Reflector.java:130)
at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
... 16 more

Fabric8 Kubernetes Client version

6.13.0

Steps to reproduce

Pod list

Expected behavior

Pod list available

Runtime

OpenShift

Kubernetes API Server version

1.25.3@latest

Environment

Linux

Fabric8 Kubernetes Client Logs

No response

Additional context

No response

[maeryo]

I am also encountering a similar error in version 6.13.0.
When I call watchLog() to view the logs of a container in a pod or deployment, the following error occurs:

2024-06-18 15:13:49.131 ERROR 45818 --- [Http Dispatcher] i.f.k.c.informers.impl.cache.Reflector   : listSyncAndWatch failed for v1/namespaces/ns-1/pods, will stop


java.util.concurrent.CompletionException: java.util.concurrent.RejectedExecutionException
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1159)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
	at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$10(StandardHttpClient.java:142)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
	at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:51)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$OkHttpAsyncBody.doConsume(OkHttpClientImpl.java:136)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.util.concurrent.RejectedExecutionException: null
	at io.fabric8.kubernetes.client.utils.internal.SerialExecutor.execute(SerialExecutor.java:47)
	at io.fabric8.kubernetes.client.informers.impl.cache.SharedProcessor.execute(SharedProcessor.java:179)
	at io.fabric8.kubernetes.client.informers.impl.cache.Reflector.lambda$null$4(Reflector.java:136)
	at io.fabric8.kubernetes.client.informers.impl.cache.ProcessorStore.retainAll(ProcessorStore.java:116)
	at io.fabric8.kubernetes.client.informers.impl.cache.Reflector.lambda$listSyncAndWatch$6(Reflector.java:130)
	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
	... 16 common frames omitted

[manusa]

Could you provide a reproducer project?

/cc @shawkins

[shawkins]

RejectedExecutionException - would indicate that the client has been closed.

[olevitt]

We are facing the same issue since 6.13.0, I managed to reproduce it with the simplest code (created a POC repo here : https://github.com/olevitt/bug-fabric8) :

try (KubernetesClient kubernetesClient = new KubernetesClientBuilder().build()) {
            String result = kubernetesClient.pods()
                    .inNamespace(
                            "namespacename")
                    .withName("podname")
                    .getLog();
        }

This code works and retrieves the logs on all versions but produces the following error on 6.13.0 (and not on earlier versions) :

2024-06-19T09:21:21.748+02:00 ERROR 29629 --- [demo] [Http Dispatcher] i.f.k.c.informers.impl.cache.Reflector   : listSyncAndWatch failed for v1/namespaces/namespacename/podname, will stop

java.util.concurrent.CompletionException: java.util.concurrent.RejectedExecutionException
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1159) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[na:na]
	at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$10(StandardHttpClient.java:142) ~[kubernetes-client-api-6.13.0.jar:na]
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[na:na]
	at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:51) ~[kubernetes-client-api-6.13.0.jar:na]
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[na:na]
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$OkHttpAsyncBody.doConsume(OkHttpClientImpl.java:136) ~[kubernetes-httpclient-okhttp-6.13.0.jar:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]
Caused by: java.util.concurrent.RejectedExecutionException: null
	at io.fabric8.kubernetes.client.utils.internal.SerialExecutor.execute(SerialExecutor.java:47) ~[kubernetes-client-6.13.0.jar:na]
	at io.fabric8.kubernetes.client.informers.impl.cache.SharedProcessor.execute(SharedProcessor.java:179) ~[kubernetes-client-6.13.0.jar:na]
	at io.fabric8.kubernetes.client.informers.impl.cache.Reflector.lambda$null$4(Reflector.java:136) ~[kubernetes-client-6.13.0.jar:na]
	at io.fabric8.kubernetes.client.informers.impl.cache.ProcessorStore.retainAll(ProcessorStore.java:116) ~[kubernetes-client-6.13.0.jar:na]
	at io.fabric8.kubernetes.client.informers.impl.cache.Reflector.lambda$listSyncAndWatch$6(Reflector.java:130) ~[kubernetes-client-6.13.0.jar:na]
	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150) ~[na:na]
	... 16 common frames omitted

[shawkins]

Should be addressed by #6063

@manusa a backport is optional as this creates an erroneous log, but otherwise doen't cause a behavioral issue. The log however is at the error level.

[manusa]

a backport is optional as this creates an erroneous log, but otherwise doen't cause a behavioral issue. The log however is at the error level.

I will probably backport this and release 6.13.1 (will wait a few days or a week in case there is more patching required) since we want to have a last v6 minor as stable as possible.