Fix Scheduler restarting due to too many completed pods in cluster #40183
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ephraimbuddy
requested review from
jedcunningham and
hussein-awala
as code owners
boring-cyborg
bot
added
area:providers
provider:cncf-kubernetes
ephraimbuddy
commented
Jun 11, 2024
ephraimbuddy
commented
Jun 11, 2024
airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py
Show resolved
Hide resolved
ephraimbuddy
force-pushed
the
dont-watch-completed-pods
branch
from
860bcb5
to
625170d
Compare
|
This runs at every watcher loop and can get big in the case of many completed pods not deleted |
ephraimbuddy
force-pushed
the
dont-watch-completed-pods
branch
2 times, most recently
from
a1474d8
to
d70cded
Compare
ephraimbuddy
changed the title
airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py
Outdated
Show resolved
Hide resolved
airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py
Show resolved
Hide resolved
2 tasks
ephraimbuddy
force-pushed
the
dont-watch-completed-pods
branch
from
7c32d7f
to
87c4ba1
Compare
eladkal
approved these changes
Jun 13, 2024
romsharon98
approved these changes
Jun 13, 2024
amoghrajesh
approved these changes
Jun 13, 2024
jedcunningham
approved these changes
Jun 13, 2024
airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py
Outdated
Show resolved
Hide resolved
Currently, when a pod completes and is not deleted due to the user's configuration, the watcher keeps listing these pods and checking their status. We should instead stop watching the pod once it succeeds. To do that, pods are created with the executor done label set to False and changed to True when the pod completes. The watcher then watches only those pods that the pod executor done label is False closes: apache#22612
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
…r_utils.py Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
ephraimbuddy
force-pushed
the
dont-watch-completed-pods
branch
from
37ac60e
to
171a655
Compare
jannisko
pushed a commit
to jannisko/airflow
that referenced
this pull request
Jun 15, 2024
…pache#40183) * Fix Scheduler restarting due to too many completed pods in cluster Currently, when a pod completes and is not deleted due to the user's configuration, the watcher keeps listing these pods and checking their status. We should instead stop watching the pod once it succeeds. To do that, pods are created with the executor done label set to False and changed to True when the pod completes. The watcher then watches only those pods that the pod executor done label is False closes: apache#22612 * Update airflow/providers/cncf/kubernetes/pod_generator.py Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> * Add back removed section * Don't add pod key label from get go * Update airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
96 tasks
romsharon98
pushed a commit
to romsharon98/airflow
that referenced
this pull request
Jul 26, 2024
…pache#40183) * Fix Scheduler restarting due to too many completed pods in cluster Currently, when a pod completes and is not deleted due to the user's configuration, the watcher keeps listing these pods and checking their status. We should instead stop watching the pod once it succeeds. To do that, pods are created with the executor done label set to False and changed to True when the pod completes. The watcher then watches only those pods that the pod executor done label is False closes: apache#22612 * Update airflow/providers/cncf/kubernetes/pod_generator.py Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> * Add back removed section * Don't add pod key label from get go * Update airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, when a pod completes and is not deleted due to the user's configuration, the watcher keeps listing these pods and checking their status. We should instead stop watching the pod once it succeeds. To do that, pods are created with the executor done label set to False and changed to True when the pod completes. The watcher then watches only those pods that the pod executor done label is False
closes: #22612