Informer Watch call issue -> javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed:

[ketankr9]

Describe the bug
SSLHandshakeException exception with webSocket/SSE watch call, however list call is working fine.

Client Version
11.0.2

Kubernetes Version
1.20.11-gke.1300

Java Version
(AdoptOpenJDK)(build 1.8.0_242-b08)

To Reproduce
Start a simple informer code.
Main.java

public class Main {
    public static void main(String... args) throws IOException, InterruptedException, ApiException {
        ApiClient apiClient = Config.defaultClient();
        SharedInformerFactory factory = new SharedInformerFactory();

        new NodeWatcher(apiClient, factory);

        Thread.sleep(20 * 60 * 1000L);

        logger.info("Done");
    }
}

NodeWatcher.java

public class NodeWatcher implements ResourceEventHandler<V1Node> {

  public final SharedInformerFactory factory;

  public static final String MOST_RECENT = "MostRecent";
  public static final String EXACT = "Exact";
  public static final String NOT_OLDER_THAN = "NotOlderThan";

  @SneakyThrows
  public NodeWatcher(ApiClient client,  SharedInformerFactory factory) {
    CoreV1Api coreV1Api = new CoreV1Api(client);

    this.factory = factory;

    this.factory.sharedIndexInformerFor(
            (CallGeneratorParams callGeneratorParams)
                    -> {
              try {
                return coreV1Api.listNodeCall(null, null, null, null, null, null, callGeneratorParams.resourceVersion, NOT_OLDER_THAN, callGeneratorParams.timeoutSeconds, callGeneratorParams.watch, null);
              } catch (Exception e) {
                logger.info("Unknown exception occurred"+ e.toString());
                throw e;
              }
            },
            V1Node.class, V1NodeList.class)
            .addEventHandler(this);
    this.factory.startAllRegisteredInformers();
  }

  @Override
  public void onAdd(V1Node obj) {
    logger.info("Added: " + obj.getMetadata().getUid() + " "+obj.getMetadata().getResourceVersion());
  }

  @Override
  public void onUpdate(V1Node oldObj, V1Node newObj) {
    logger.info("update to: " + newObj.getMetadata().getUid()+" resourceVersion: "+newObj.getMetadata().getResourceVersion());
  }

  @Override
  public void onDelete(V1Node obj, boolean deletedFinalStateUnknown) {
    logger.info("Deleted: " + obj.getMetadata().getUid());
  }
}

Expected behavior

1. get list event once with onAdd
2. Then get next updated item with increasing resourceVersion using watch call.
   But the Controller code is not able to execute watch call and exiting every time resulting in list call every 1 seconds.
   As a consequence the heap is getting filled quickly and also we are receiving duplicate events with same resource version.

Error thrown
handling exception: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

KubeConfig
If applicable, add a KubeConfig file with secrets redacted.

- name: gke_abc_us-central1-c_xyz
  user:
    auth-provider:
      config:
        access-token: <Deleted>
        cmd-args: config config-helper --format=json
        cmd-path: /Users/username/Downloads/google-cloud-sdk/bin/gcloud
        expiry: "2022-01-21T13:25:27Z"
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp

Server (please complete the following information):

• OS: [e.g. Linux]
• Environment [e.g. container]
• Cloud: GCP

Additional context
With java ssl logging enabled -Djavax.net.debug=ssl , I see below code, but only for watch call and not list call.

%% Invalidated:  [Session-3, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]
controller-reflector-io.kubernetes.client.openapi.models.V1Node-1, SEND TLSv1.2 ALERT:  fatal, description = certificate_unknown
controller-reflector-io.kubernetes.client.openapi.models.V1Node-1, WRITE: TLSv1.2 Alert, length = 2
controller-reflector-io.kubernetes.client.openapi.models.V1Node-1, called closeSocket()
controller-reflector-io.kubernetes.client.openapi.models.V1Node-1, handling exception: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
controller-reflector-io.kubernetes.client.openapi.models.V1Node-1, called close()
controller-reflector-io.kubernetes.client.openapi.models.V1Node-1, called closeInternal(true)

[brendandburns]

It's really interesting that it works for list, but not for watch. It should be the same auth loading code in both places.

I will audit the code and see if I can see any differences.

[acasian-harness]

hey @brendandburns, did you managed to find any differences? This is blocking us upgrading to 11.0.2 version. Thanks!

[yue9944882]

@ketankr9 @acasian-harness @brendandburns a work-around is to instantiate the informer-factory with apiClient explicitly pinned

SharedInformerFactory factory = new SharedInformerFactory(apiClient);

i think we should explicitly deprecate the no-arg informer-factory to avoid confusion and a warning should be thrown when diverged apiClient instance were found.

java/util/src/main/java/io/kubernetes/client/informer/SharedInformerFactory.java

Lines 224 to 227 in 0642e7b

	public Watch<ApiType> watch(CallGeneratorParams params) throws ApiException {
	Call call = callGenerator.generate(params);
	// bind call with private http client to make sure read timeout is zero.
	call = apiClient.getHttpClient().newCall(call.request());