KubernetesPodOperator: Rework of Kubernetes API retry behavior #58397
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AutomationDev85
requested review from
hussein-awala and
jedcunningham
as code owners
boring-cyborg
bot
added
area:providers
provider:cncf-kubernetes
jscheffl
approved these changes
Nov 18, 2025
Contributor
jscheffl
left a comment
jscheffl
left a comment
There was a problem hiding this comment.
Looks very good to me. Thanks for the cleanup and consolidation!
I leave this open for some-second pair of eyes for some days. Else would propose to merge prior next provider wave.
One small nit but rather non-blocking.
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/utils/pod_manager.py
Show resolved
Hide resolved
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/kubernetes_helper_functions.py
Show resolved
Hide resolved
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/utils/test_pod_manager.py
Outdated
Show resolved
Hide resolved
jscheffl
reviewed
Nov 19, 2025
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/kubernetes_helper_functions.py
Show resolved
Hide resolved
AutomationDev85
changed the title
Contributor
Author
|
Sorry found out that I missed to check for the async exception type. I updated the code to catch sync and async exceptions. |
Contributor
Thanks for updating! |
potiuk
reviewed
Nov 22, 2025
Member
potiuk
left a comment
potiuk
left a comment
There was a problem hiding this comment.
It looks good from my slde as well.
potiuk
approved these changes
Nov 22, 2025
Member
|
It looks like it is handling a LOT more common issues in a LOT more generic way. |
This was referenced Nov 27, 2025
Copilot AI
pushed a commit
to jason810496/airflow
that referenced
this pull request
Dec 5, 2025
…e#58397) * Move retry handling to the hook layer and update PodManager accordingly * Removed overlapping code * Clean up code * Detailed logging and use of autouse fixture * move no wait fixture into conftest * Disabled no_retry_wait patch for explicitly marked unit tests. * Fix unit test * Generic retry logic can handle async and sync kubernetes api exceptions --------- Co-authored-by: AutomationDev85 <AutomationDev85>
itayweb
pushed a commit
to itayweb/airflow
that referenced
this pull request
Dec 6, 2025
…e#58397) * Move retry handling to the hook layer and update PodManager accordingly * Removed overlapping code * Clean up code * Detailed logging and use of autouse fixture * move no wait fixture into conftest * Disabled no_retry_wait patch for explicitly marked unit tests. * Fix unit test * Generic retry logic can handle async and sync kubernetes api exceptions --------- Co-authored-by: AutomationDev85 <AutomationDev85>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
We are encountering "Too Many Requests" (HTTP 429) errors from the Kubernetes API when scaling up nodes in our Kubernetes cluster. We introduced already the retry handling in the PodManager with this PR: #58033
but we found also the issue that the PodManager access the KuberentesHook api client direct and so the idea is to add the retry on the Hook level and catch only the spezific funtions on the PodManager with direct access to the API Client with retry handling.
This PR also changes the handling of the retries so that only retry worth statuscodes and errors are retried.
We are encountering frequent HTTP 429 “Too Many Requests” responses from the Kubernetes API during node scale-up operations. A prior change (see PR #58033) introduced retry handling in the PodManager. But some PodManager methods bypassed that logic by using the KubernetesHook API client directly. This change moves the primary retry mechanism into the KubernetesHook and adds targeted retries only for PodManager methods that invoke the API client directly.
Retry behavior is refined to act only on retry-worthy status codes and errors.
We welcome your feedback on this change!
Details of change: