Task fails and cannot read logs. Invalid URL 'http://:8793/log/...': No host supplied

[pedro-cf]

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.10.1

What happened?

I'm having an issue with an airflow instance where a task fails and I cannot read the logs.

Logs:

*** Could not read served logs: Invalid URL 'http://:8793/log/dag_id=my_dag/run_id=dynamic__apple_3_my_dag_cb353081__2024-09-09T14:41:22.596199__f73c5571719e4f35bf195ded40e5e25b/task_id=cleanup_temporary_directory/attempt=1.log': No host supplied

Event logs:

Executor CeleryExecutor(parallelism=128) reported that the task instance <TaskInstance: my_dag.cleanup_temporary_directory dynamic__apple_3_my_dag_cb353081__2024-09-09T14:41:22.596199__f73c5571719e4f35bf195ded40e5e25b [queued]> finished with state failed, but the task instance's state attribute is queued. Learn more: https://airflow.apache.org/docs/apache-airflow/stable/troubleshooting.html#task-state-changed-externally

Additionally I checked the logs directory for the dag_id/run_id and it's missing the respective task_id folder.

What you think should happen instead?

I should be able to access the logs.

How to reproduce

Not sure how to.

Operating System

Ubuntu 24.04 LTS

Versions of Apache Airflow Providers

No response

Deployment

Other Docker-based deployment

Deployment details

Deployed with docker-compose on Docker Swarm setup on 2 VMs.

Anything else?

Additionally I checked the logs directory for the dag_id/run_id and it's missing the respective task_id folder.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[andrew-stein-sp]

having the same issue with 2.10.1 in k8s, using the CeleryKubernetesExecutor.

Could this be related to the inheritance issue that was discussed in #41891?

[pedro-cf]

Additionally I checked the logs directory for the dag_id/run_id and it's missing the respective task_id folder.

[adriens]

Having the same issue on 2.10.0 through a podman-compose

[adriens]

We have upgraded on 2.10.1 like @andrew-stein-sp and we could reproduce the same behavior

[sosystems-dev]

got the same behavior since upgrading from version 2.9.3 to 2.10.1.
We are using LocalExecutor

[mn7k]

I have the same issue with 2.10.0, using the CeleryExecutor.
It worked before I upgrading from version 2.9.0 to 2.10.0.

*** Could not read served logs: Invalid URL 'http://:8793/log/dag_id=service_stop/run_id=manual__2024-09-18T09:42:54+09:00/task_id=make_accountlist_task/attempt=1.log': No host supplied

eventlog

Executor CeleryExecutor(parallelism=6) reported that the task instance <TaskInstance: service_stop.make_accountlist_task manual__2024-09-18T09:42:54+09:00 [queued]> finished with state failed, but the task instance's state attribute is queued. Learn more: https://airflow.apache.org/docs/apache-airflow/stable/troubleshooting.html#task-state-changed-externally

Scheduler has a error log at the same hour as eventlog.

[2024-09-18T00:43:18.036+0000] {celery_executor.py:291} ERROR - Error sending Celery task: module 'redis' has no attribute 'client'
Celery Task ID: TaskInstanceKey(dag_id='service_stop', task_id='make_accountlist_task', run_id='manual__2024-09-18T09:42:54+09:00', try_number=1, map_index=-1)
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/celery/executors/celery_executor_utils.py", line 220, in send_task_to_executor
    result = task_to_run.apply_async(args=[command], queue=queue)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/task.py", line 594, in apply_async
    return app.send_task(
           ^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/base.py", line 797, in send_task
    with self.producer_or_acquire(producer) as P:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/base.py", line 932, in producer_or_acquire
    producer, self.producer_pool.acquire, block=True,
              ^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/base.py", line 1354, in producer_pool
    return self.amqp.producer_pool
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/amqp.py", line 591, in producer_pool
    self.app.connection_for_write()]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/base.py", line 829, in connection_for_write
    return self._connection(url or self.conf.broker_write_url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/base.py", line 880, in _connection
    return self.amqp.Connection(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/kombu/connection.py", line 201, in __init__
    if not get_transport_cls(transport).can_parse_url:
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/kombu/transport/__init__.py", line 91, in get_transport_cls
    _transport_cache[transport] = resolve_transport(transport)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/kombu/transport/__init__.py", line 76, in resolve_transport
    return symbol_by_name(transport)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/kombu/utils/imports.py", line 59, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/home/airflow/.local/lib/python3.12/site-packages/kombu/transport/redis.py", line 282, in <module>
    class PrefixedRedisPipeline(GlobalKeyPrefixMixin, redis.client.Pipeline):
                                                      ^^^^^^^^^^^^
AttributeError: module 'redis' has no attribute 'client'

[damiah]

same issue for us when upgrading to 2.10.2

[nikithapk]

We’re encountering the same issue as well.

[adriens]

We have switched from Bitnami docker-compose to official Apache docker-compose and we could make it run successfuly 🤩

[Dzhalolov]

try to check that dags exist in worker, schedule and webserver. I deploy Airflow in K8S and get this error when putting my dags into scheduler(expecting that in will replicate into another pods), but when I check dags folder in worker it was empty

[mn7k]

At this time (#42136 (comment)), I used the airflow db upgrade command , but I realized it has been deprecated.
I retried the upgrade using the airflow db migrate -n "2.10.2" command, and it works for me now.

https://airflow.apache.org/docs/apache-airflow/2.10.0/installation/upgrading.html#offline-sql-migration-scripts