[3.0.4][Remote Logging] "Unable to find AWS Connection ID" under KubernetesExecuotor configuration

[potiuk]

Discussed in #54602

^{Originally posted by hcnhcn012 August 18, 2025}
airflow version: 3.0.4
deployment method: helm
executor: KubernetesExecuotor

---

I noticed this issue was happened in former versions and it should be fixed in 3.0.4 (see: #50583), but it still happend for me when i upgrade airflow version by changing helm value:

defaultAirflowTag: "3.0.4"
airflowVersion: "3.0.4"

I can confirm that the aws connection was created correctly (checked airflow connections list), but the worker pod can not find this connection. Here is my configuration.

---

Airflow:

[kubernetes_executor]
delete_worker_pods = False
multi_namespace_mode = False
namespace = airflow
pod_template_file = /opt/airflow/pod_templates/dynamic_deps_template.yaml
worker_container_repository = apache/airflow
worker_container_tag = 3.0.4

[logging]
colored_console_log = False
remote_logging = True
remote_base_log_folder = s3://airflow-worker-logs
remote_log_conn_id = worker_remote_log_s3_conn

Pod template (only show base container):

  containers:
    - name: base
      image: "apache/airflow:3.0.4-python3.12"
      imagePullPolicy: IfNotPresent
      env:
        - name: AIRFLOW__CORE__EXECUTOR
          value: LocalExecutor
        - name: AIRFLOW__CORE__FERNET_KEY
          valueFrom:
            secretKeyRef:
              name: airflow-fernet-key
              key: fernet-key
        - name: AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
          valueFrom:
            secretKeyRef:
              name: pai-docai-airflow-metadata
              key: connection
        - name: AIRFLOW_CONN_AIRFLOW_DB
          valueFrom:
            secretKeyRef:
              name: pai-docai-airflow-metadata
              key: connection
        - name: AIRFLOW__CORE__DAGS_FOLDER
          value: "/opt/airflow/dags/pai-docai-dataflow/paidocaidataflow/dags"
        - name: PYTHONPATH
          value: "/opt/airflow/python-packages:/opt/airflow/dags/pai-docai-dataflow:/opt/airflow/dags"
        - name: PROJECT_ROOT
          value: "/opt/airflow/dags/pai-docai-dataflow"

        - name: AIRFLOW__API__BASE_URL
          value: "http://pai-docai-airflow-api-server.airflow.svc.cluster.local:8080"
        - name: AIRFLOW__API__AUTH_BACKENDS
          value: "airflow.api.auth.backend.default"
        - name: AIRFLOW__LOGGING__REMOTE_LOGGING
          value: "True"
        - name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
          value: "s3://airflow-worker-logs"
        - name: AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID
          value: "worker_remote_log_s3_conn"

      volumeMounts:
        - name: airflow-dags
          mountPath: /opt/airflow/dags
        - name: python-packages
          mountPath: /opt/airflow/python-packages
        - name: airflow-logs
          mountPath: /opt/airflow/logs

---

original failure log:

{"event":"Unable to find AWS Connection ID 'worker_remote_log_s3_conn', switching to empty.","level":"warning","logger":"airflow.task.hooks.airflow.providers.amazon.aws.hooks.s3.S3Hook","timestamp":"2025-08-18T08:15:49.647380Z"}

{"event":"No connection ID provided. Fallback on boto3 credential strategy (region_name=None). See: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html","level":"info","logger":"airflow.providers.amazon.aws.hooks.base_aws.BaseSessionFactory","timestamp":"2025-08-18T08:15:49.647765Z"}
``` </div>

[potiuk]

I believe the problem is that when you use kubernetes executor, the Pods executed with it do not have access to supervisor that has the key to communicate with API server (wildly guessing but it looks like this @ashb @amoghrajesh @kaxil ).

[vinitpayal]

@potiuk I feel it's not only specific to kubernetes executor but something common for reading AWS secrets from metadata database.

Similar issue is being faced in the EKSPodOperator as well where connection id is defined but while using EKSPodOperator it reports the same.

Was debuggin the same by adding some logs and seems like in AwsConnectionWrapper it's able to read the connection but later gets switched to the kubernetes_default connection.

[2025-08-18, 16:40:23] WARNING - Using Connection.get_connection_from_secrets from `airflow.models` is deprecated.Please use `from airflow.sdk import Connection` instead: category="DeprecationWarning": filename="/usr/local/lib/python3.12/site-packages/airflow/models/connection.py": lineno=471: source="py.warnings"
[2025-08-18, 16:40:23] INFO - Connection Retrieved 'aws_eks': source="airflow.hooks.base"
[2025-08-18, 16:40:23] INFO - AWS Connection (conn_id='aws_eks', conn_type='aws') credentials retrieved from login and password1.: source="airflow.providers.amazon.aws.utils.connection_wrapper.AwsConnectionWrapper"
[2025-08-18, 16:40:23] INFO - aws conn reterived: AKIAZXXXXXXXXXXXXXXXX | secret: *** | session token: None: source="airflow.providers.amazon.aws.utils.connection_wrapper.AwsConnectionWrapper"
[2025-08-18, 16:40:23] INFO - secret: ***: chan="stdout": source="task"
[2025-08-18, 16:40:24] INFO - Building pod run-data-operations-script-rnpqbnb2 with labels: {'dag_id': 'manual_data_operations_command_run', 'task_id': 'run_data_operations_script', 'run_id': 'manual__2025-08-18T111021.6180000000-14be046c6', 'kubernetes_pod_operator': 'True', 'try_number': '1'}: source="airflow.task.operators.utils.operators.eks_custom_operator.CustomEksRunTaskOperator"
[2025-08-18, 16:40:24] INFO - Connection Retrieved 'kubernetes_default': source="airflow.hooks.base"
[2025-08-18, 16:40:25] ERROR - exec: process returned 1. 1 -> aws_eks
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/airflow/providers/amazon/aws/hooks/base_aws.py", line 625, in conn_config
    connection = self.get_connection(self.aws_conn_id)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/airflow/hooks/base.py", line 64, in get_connection
    conn = Connection.get_connection_from_secrets(conn_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/airflow/models/connection.py", line 512, in get_connection_from_secrets
    raise AirflowNotFoundException(f"The conn_id `{conn_id}` isn't defined")
airflow.exceptions.AirflowNotFoundException: The conn_id `aws_eks` isn't defined

[2025-08-18T11:10:25.251+0000] {base_aws.py:636} WARNING - Unable to find AWS Connection ID 'aws_eks', switching to empty.
[2025-08-18T11:10:25.252+0000] {base_aws.py:201} INFO - No connection ID provided. Fallback on boto3 credential strategy (region_name='eu-central-1'). See: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
[2025-08-18T11:10:25.254+0000] {base_aws.py:201} INFO - No connection ID provided. Fallback on boto3 credential strategy (region_name='eu-central-1'). See: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
1 -> aws_default
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/airflow/providers/amazon/aws/hooks/base_aws.py", line 625, in conn_config
    connection = self.get_connection(self.aws_conn_id)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/airflow/hooks/base.py", line 64, in get_connection
    conn = Connection.get_connection_from_secrets(conn_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/airflow/models/connection.py", line 512, in get_connection_from_secrets
    raise AirflowNotFoundException(f"The conn_id `{conn_id}` isn't defined")
airflow.exceptions.AirflowNotFoundException: The conn_id `aws_default` isn't defined

It's happening with

Airflow version: 3.0.4
Python version: 3.12
Running it using docker container

[itrozmyslov]

I faced the same issue.
Adding a variable to extraEnv helped me:

extraEnv: |
  - name: AIRFLOW_CONN_WORKER_REMOTE_LOG_S3_CONN
    value: '{"conn_type":"aws","login":"minioa","password":"minio","extra":{"endpoint_url":"http://minio:9000","region_name":"us-east-1","verify":false}}'

This seems like a hack, but it allows airflow to work until a fix is found.