Error handling for Dependents is incomplete

[teabot]

Bug Report

What did you do?

I have an operator implementation that was working well for some time, and then later started failing in an obscure, unintuitive way; the dependent would report ready but I could see from logs that much of the dependent code was not executed. In fact the state changes required for readiness were not being performed, and conditions that verified such changes (readyPostcondition) were being bypassed. No changes had been made to the operator implementation prior to the occurrence of this issue.

Eventually, using an Exception break-point, we traced the issue to a dependency update, that caused a java.lang.NoSuchFieldError to be thrown in the dependent.

What did you expect to see?

I expected that the NoSuchFieldError would be caught, register as an error in the dependent, prevent its readiness, and would be logged to aid the diagnosis of the root cause.

What did you see instead? Under which circumstances?

The NoSuchFieldError was seemingly swallowed, was not classed as an error in the context of the dependent, and in fact seemed to short-circuit in such a way that the dependent was marked as ready despite not executing most of my dependent logic. Critically, it was it was not logged, which greatly impeded diagnosis of the issue.

Note

The expectation that java.lang.Error should be handled may be contentious, as the JDK specifies that Error "indicates serious problems that a reasonable application should not try to catch. Most such errors are abnormal conditions.". However, I'd argue that by effectively swallowing this class of exceptions, JOSDK ends up with insidious behaviours.

Environment

Kubernetes cluster type: k3s:1.31.0

java-operator-sdk.version: 5.1.1
via:
quarkus-operator-sdk.version: 7.2.0
via:
quarkus.version: 3.26.3

❯ java -version
openjdk version "21.0.7" 2025-04-15 LTS
OpenJDK Runtime Environment Temurin-21.0.7+6 (build 21.0.7+6-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.7+6 (build 21.0.7+6-LTS, mixed mode, sharing)

❯ kubectl version
Client Version: v1.30.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3

Possible Solution

Conceptually, it would seem that the part of the framework that invokes Dependents needs to also catch Throwable. Note that the code that invokes Reconcilers appears to handle this case correctly; when throwing a NoSuchFieldError from within a Reconciler.reconcile implementation, I observed that it is eventually caught and logged as we might expect. There may be other cases where the scope of exception handling of framework code calling user implementations needs to be broadened.

Additional context

Steps to reproduce:

1. Pick a Dependent implementation
2. Override the reconcile method, and simply throw NoSuchFieldError (NOT NoSuchFieldException) from within the reconcile method.
3. Reconcile your Reconciler, observe that the Dependent will report as ready and no error will be logged by the framework.

@KubernetesDependent
@Dependent
public class MyDependent extends CRUDNoGCKubernetesDependentResource<Namespace, MyPrimary>
    implements Creator<Namespace, MyPrimary>, Updater<Namespace, MyPrimary>, Deleter<MyPrimary> {
  ...
  @Override
  public ReconcileResult<Namespace> reconcile(MyPrimary primary, Context<MyPrimary> context) {
    throw new NoSuchFieldError();
  }
}

[csviri]

Hi @teabot , thank you for reporting, do you plan to create a PR for this?

[teabot]

@csviri I did not plan to, as I do not know the internals of JOSDK well enough to do so. With some guidance, I would be happy to.

[csviri]

Hmm, thinking more about this... the thing is, Throwable, or Error, are not really meant to be caught. Could you share some details what leads to this error? Isn't the reason of some live class reloading?

[teabot]

My specific issue aside, the bottom line is that if an java.lang.Error is thrown in a dependent, it gets swallowed, and the dependent reports ready. My case is just one of many possible variants. The consequences of a java.lang.Error thrown in a dependent are less than ideal - I had a system that thought it was healthy, but was actually in an inconsistent state and, short of attaching a debugger, gave no hint of a problem.

In my case, the dependent was managing an external resource: AWS EKS PodIdentityAssociations. It uses the AWS SDK. The error was thrown from within that SDK client due to a dependency version mismatch.

I do not know the specific issue in JOSDK, but somehow, the thread that is invoking my dependent is swallowing the error from the SDK; it vanishes with no log, causes no error in the dependent. It reports ready, despite being quite broken! This can be demonstrated by using the example provided in this issue. Something is preventing such errors from propagating back to the main thread and halting the JVM.

Perhaps this is not an issue of needing to handle java.lang.Errors, but rather the need to remove existing handling that swallows the error? I do not know enough about the internals of JOSDK to be sure.

[csviri]

Thank you @teabot, we will take a look soon