Webserver cannot read logs of pods launched by a scheduler of mismatching Airflow version

[wolfier]

Apache Airflow version

2.10.2

If "Other Airflow 2 version" selected, which one?

No response

What happened?

The webserver was unable to fetch the logs of task instance.

*** No logs found in GCS; ti=%s <TaskInstance: long_running_tasks_poc_copy.simple_kpo scheduled__2024-10-03T00:00:00+00:00 [running]>
*** Attempting to fetch logs from pod long-running-tasks-poc-copy-simple-kpo-k5v9qx6i through kube API
*** Reading from k8s pod logs failed: ('Cannot find pod for ti %s', <TaskInstance: long_running_tasks_poc_copy.simple_kpo scheduled__2024-10-03T00:00:00+00:00 [running]>)

The corresponding Kubernetes worker pod was in the running phase but the weberver could not find the pod in the namespace. It turns out it was because the label selector was not matching for the label airflow-version. The pod had the label airflow-version=2.9.3+astro.4 and the webserver was using the Airflow version 2.9.3+astro.5 which created the label selector airflow-version=2.9.3+astro.5.

The reason why the version were different is because the deployment's Airflow version was upgraded from 2.9.3+astro.4 to 2.9.3+astro.5 after the task instance was launched.

---

Closed by #42751

What you think should happen instead?

The Airflow version should not matter in terms of the webserver's ability to read the logs of a pod.

How to reproduce

1. Launch a long running task using the KubernetesExecutor
2. Check task logs in webserver (should be visible)
3. Upgrade the deployment to a different version
4. Check task logs in webserver (should now be missing due to failure to read)

Operating System

n/a

Versions of Apache Airflow Providers

No response

Deployment

Astronomer

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct