improve deferrable KPO handling of deleted pods in between polls #56976
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This makes the error's stacktrace easier to review in the logs and does not depend on `trigger_reentry` to do so.
This makes it easier to know which value for poll_interval is actually being used.
It can happen that the pod doesn't exist anymore when running `trigger_reentry`. In that case `self.hook.get_pod` will cause a 404 API exception, which then will end up calling `self._clean` which assumed `self.pod` not to be None.
m1racoli
requested review from
hussein-awala and
jedcunningham
as code owners
boring-cyborg
bot
added
area:providers
provider:cncf-kubernetes
ashb
approved these changes
Oct 21, 2025
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/triggers/pod.py
Show resolved
Hide resolved
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/triggers/pod.py
Outdated
Show resolved
Hide resolved
This doesn't work in Airflow 2.
Lzzz666
pushed a commit
to Lzzz666/airflow
that referenced
this pull request
Oct 30, 2025
…che#56976) * refactor: log exception in KubernetesPodTrigger This makes the error's stacktrace easier to review in the logs and does not depend on `trigger_reentry` to do so. * refactor: log poll_interval on trigger start This makes it easier to know which value for poll_interval is actually being used. * fix: handle missing pod when trying to cleanup after trigger reentry It can happen that the pod doesn't exist anymore when running `trigger_reentry`. In that case `self.hook.get_pod` will cause a 404 API exception, which then will end up calling `self._clean` which assumed `self.pod` not to be None. * fix: don't use structured logging when logging poll interval This doesn't work in Airflow 2.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
While
KubernetesPodTriggeris polling the KPO pod, in case of large values forpoll_interval(i.e 900 seconds) it can happen that the completed podhas been already been cleaned up in-between polls. This causes the following chain of events:KubernetesPodTriggerKubernetesPodOperator.trigger_reentryending up in another 404 pod not found inself.hook.get_podKubernetesPodOperator._cleanbeing called as part of thefinallyblockKubernetesPodOperator.pod_manager.await_pod_completionfailing to handleself.pod == Noneending up inAttributeError: 'NoneType' object has no attribute 'metadata'We improve this situation with the following adjustments:
KubernetesPodOperator._callearly ifself.podisNone