Regression issue using ExecWatch: Getting forbidden spite of being authenticated

[Sgitario]

In previous versions of fabric8, we were doing:

public String execOnPod(String namespace, String podName, String containerId, String... command)
            throws InterruptedException {

        ByteArrayOutputStream out = new ByteArrayOutputStream();
        CountDownLatch execLatch = new CountDownLatch(1);

        try (ExecWatch execWatch = client.pods().inNamespace(namespace).withName(podName).inContainer(containerId)
                .writingOutput(out)
                .usingListener(new ExecListener() {
                    @Override
                    public void onOpen(Response response) {
                        // do nothing
                    }

                    @Override
                    public void onFailure(Throwable throwable, Response response) {
                        execLatch.countDown();
                    }

                    @Override
                    public void onClose(int i, String s) {
                        execLatch.countDown();
                    }
                })
                .exec(input)) {
            execLatch.await(TIMEOUT_DEFAULT.ZERO.toMinutes(), TimeUnit.MINUTES);
            return out.toString();
        }
    }

This is now returning:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "pods \"prometheus-user-workload-0\" is forbidden: User \"system:anonymous\" cannot get resource \"pods/exec\" in API group \"\" in the namespace \"openshift-user-workload-monitoring\"",
  "reason": "Forbidden",
  "details": {
    "name": "prometheus-user-workload-0",
    "kind": "pods"
  },
  "code": 403
}

Note that we are authenticated using the token: oc login ...

This is not working in 5.7.3 or 5.7.2, but it worked in the past.

[manusa]

I'm not sure if this is related to recent changes in the Pod Exec feature related code or to Config and cluster authentication (#3445).

I assume that you have tested the same code with the different client versions on the same cluster (same environment | different client).

[Sgitario]

I'm not sure if this is related to recent changes in the Pod Exec feature related code or to Config and cluster authentication (#3445).

I assume that you have tested the same code with the different client versions on the same cluster (same environment | different client).

Yep, this was working on the Fabric8 dependencies being used by Quarkus 1.13

[tsaarni]

I doubt it could be #3445 since it was just merged yesterday and this issue is found in previous releases. Or that #3445 could fix it either, since the error says is forbidden: User "system:anonymous" i.e. also the user is not set.

[manusa]

I doubt it could be #3445 since it was just merged yesterday and this issue is found in previous releases. Or that #3445 could fix it either, since the error says is forbidden: User "system:anonymous" i.e. also the user is not set.

Yep, wrong PR linked, but there were recent changes to the TokenRefresjInterceptor and the Config retrieval. The question was more about if with a constant environment, the issue was present only in the recent versions of the client.

[rohanKanojia]

Hello, I tried reproducing this issue but I'm not able to reproduce this. For me, exec seems to be working okay and I don't get 403. I prepared a reproducer based on the code you shared: https://github.com/rohankanojia-forks/kubernetes-client-execwatch-test
and checked these scenarios (I've logged into all clusters via oc login) :

1. CRC 1.35.0 OpenShift 4.9.5 OpenShiftClient 5.7.3: Run locally via java -jar ✔️
2. CRC 1.35.0 OpenShift 4.9.5 OpenShiftClient 5.7.3: Run inside Pod by deploying with OpenShift Gradle Plugin ✔️
3. CRC 1.35.0 OpenShift 4.9.5 OpenShiftClient 5.10.1: Run locally via java -jar ✔️
4. CRC 1.35.0 OpenShift 4.9.5 OpenShiftClient 5.10.1: Run inside Pod by deploying with OpenShift Gradle Plugin ✔️
5. Red Hat Developer Sandbox OpenShift 4.9.9 OpenShiftClient 5.7.3: Run locally via java -jar ✔️
6. Red Hat Developer Sandbox OpenShift 4.9.9 OpenShiftClient 5.7.3: Run inside Pod by deploying with OpenShift Gradle Plugin ✔️
7. Red Hat Developer Sandbox OpenShift 4.9.9 OpenShiftClient 5.10.1: Run locally via java -jar ✔️
8. Red Hat Developer Sandbox OpenShift 4.9.9 OpenShiftClient 5.10.1: Run inside Pod by deploying with OpenShift Gradle Plugin ✔️

We also have an integration test that covers ExecWatch functionality. I also ran PodIT.exec integration test on my CRC cluster and it is passing.

What could I be missing? Would appreciate it if you could provide some more information on how to reproduce this.

[Sgitario]

I've checked the code that was failing and with the latest version, it's working fine. Closing.