Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refresh properties on KubernetesPodOperator when k8s fails due to token expiration #39325

Merged
merged 8 commits into from
May 1, 2024

Conversation

paolo-moriello
Copy link
Contributor

@paolo-moriello paolo-moriello commented Apr 30, 2024

Closes: #32718


The KubernetesPodOperator caches the hook and client used to interact with K8S. This includes also the kube config and related secret auth token. For long running jobs, the token may expire during the execution, causing the job to fail:

Traceback (most recent call last):
  [...]
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py", line 690, in read_pod
    return self._client.read_namespaced_pod(pod.metadata.name, pod.metadata.namespace)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/kubernetes/client/api/core_v1_api.py", line 23483, in read_namespaced_pod
    return self.read_namespaced_pod_with_http_info(name, namespace, **kwargs)  # noqa: E501
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/kubernetes/client/api/core_v1_api.py", line 23570, in read_namespaced_pod_with_http_info
    return self.api_client.call_api(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
                    ^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/kubernetes/client/rest.py", line 240, in GET
    return self.request("GET", url,
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/.local/lib/python3.11/site-packages/kubernetes/client/rest.py", line 234, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (401)
Reason: Unauthorized
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

This PR builds on top of #32719. Before adding more tests, I'd be happy to discuss this or other approaches to solve this issue.

Copy link
Contributor

@amoghrajesh amoghrajesh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly looks good, small nits

airflow/providers/cncf/kubernetes/operators/pod.py Outdated Show resolved Hide resolved
@eladkal eladkal merged commit 43b48a9 into apache:main May 1, 2024
49 checks passed
@paolo-moriello paolo-moriello deleted the k8spodoperator-refresh-token branch May 1, 2024 14:58
RodrigoGanancia pushed a commit to RodrigoGanancia/airflow that referenced this pull request May 10, 2024
…en expiration (apache#39325)

* Refresh KubernetesPodOperator properties when credentials expire

* Add test

* Fix linting errors

* Implement review changes and add more tests

* Improve tests

* Fix linting errors

* Fix linting errors

* Move exception check to utils

---------

Co-authored-by: pmoriello <paolo.moriello@zalando.ch>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers provider:cncf-kubernetes Kubernetes provider related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Kubernetes Pod Operator Auth Failure if Token expires
4 participants