Under which category would you file this issue?
Task SDK
Apache Airflow version
3.2.0 (it might have happened in an earlier version after 3.1.6 release)
What happened and how to reproduce it?
We recently upgraded from Airflow 3.1.6 to 3.2.0. They are installed by downloading the docker images from https://hub.docker.com/r/apache/airflow/tags?name=3.2.0 (tag 3.2.0-python3.12, for 3.1.6, it was downloaded using 3.1.6-python3.12)
In our infra, we have to use worker pods IP addresses instead of hostnames, otherwise the API servers cannot reach the log servers running on worker pods. We have hostname_callable set to airflow.utils.net.get_host_ip_address to solve this issue in 3.1.6.
However, after installing 3.2.0, we noticed the api server cannot reach our worker pods and can't display real time log any more, and while checking the hostname of the task instance, it is using hostname instead of ip address.
Here are some commands I have run on the worker pods:
airflow config get-value core hostname_callable
airflow.utils.net.get_host_ip_address
python - <<EOF
from airflow.configuration import conf
print(conf.getimport("core", "hostname_callable", fallback="airflow.utils.net.getfqdn")())
from airflow.utils.net import get_hostname
print("get_hostname from utils.net:", get_hostname())
import airflow.sdk.api.client as client
print("get_hostname from api client", client.get_hostname())
EOF
10.245.149.226
get_hostname from utils.net: 10.245.149.226
get_hostname from api client 12563645-d9656854-ffcvx
I think we can mitigate the issue by reverting to an older version of task SDK but this seems a regression.
What you think should happen instead?
When runtime task instance is created, it should respect the hostname callback set in core configs.
Operating System
"Debian GNU/Linux 12 (bookworm)"
Deployment
Other Docker-based deployment
Apache Airflow Provider(s)
No response
Versions of Apache Airflow Providers
apache-airflow-task-sdk==1.2.0 <- broken oneapache-airflow-task-sdk==1.1.6 <- working one from 3.1.6 image
Official Helm Chart version
Not Applicable
Kubernetes Version
No response
Helm Chart configuration
No response
Docker Image customizations
Not Applicable
Anything else?
No response
Are you willing to submit PR?
Code of Conduct
Under which category would you file this issue?
Task SDK
Apache Airflow version
3.2.0 (it might have happened in an earlier version after 3.1.6 release)
What happened and how to reproduce it?
We recently upgraded from Airflow 3.1.6 to 3.2.0. They are installed by downloading the docker images from https://hub.docker.com/r/apache/airflow/tags?name=3.2.0 (tag 3.2.0-python3.12, for 3.1.6, it was downloaded using 3.1.6-python3.12)
In our infra, we have to use worker pods IP addresses instead of hostnames, otherwise the API servers cannot reach the log servers running on worker pods. We have
hostname_callableset toairflow.utils.net.get_host_ip_addressto solve this issue in 3.1.6.However, after installing 3.2.0, we noticed the api server cannot reach our worker pods and can't display real time log any more, and while checking the hostname of the task instance, it is using hostname instead of ip address.
Here are some commands I have run on the worker pods:
I think we can mitigate the issue by reverting to an older version of task SDK but this seems a regression.
What you think should happen instead?
When runtime task instance is created, it should respect the hostname callback set in core configs.
Operating System
"Debian GNU/Linux 12 (bookworm)"
Deployment
Other Docker-based deployment
Apache Airflow Provider(s)
No response
Versions of Apache Airflow Providers
apache-airflow-task-sdk==1.2.0<- broken oneapache-airflow-task-sdk==1.1.6<- working one from 3.1.6 imageOfficial Helm Chart version
Not Applicable
Kubernetes Version
No response
Helm Chart configuration
No response
Docker Image customizations
Not Applicable
Anything else?
No response
Are you willing to submit PR?
Code of Conduct