Celery Executor is not working with redis-py 5.0.0

[alexbegg]

Apache Airflow version

2.7.0

What happened

After upgrading to Airflow 2.7.0 in my local environment my Airflow DAGs won't run with Celery Executor using Redis even after changing celery_app_name configuration in celery section from airflow.executors.celery_executor to airflow.providers.celery.executors.celery_executor.

I see the error actually is unrelated to the recent Airflow Celery provider changes, but is related to Celery's Redis support. What is happening is Airflow fails to send jobs to the worker as the Kombu module is not compatible with Redis 5.0.0 (released last week). It gives this error (I will update this to the full traceback once I can reproduce this error one more time):

AttributeError: module 'redis' has no attribute 'client'

Celery actually is limiting redis-py to 4.x in an upcoming version of Celery 5.3.x (it is merged to main on August 17, 2023 but it is not yet released: celery/celery#8442 . The latest version is v5.3.1 released on June 18, 2023).

Kombu is also going to match Celery and limit redis-py to 4.x in an upcoming version as well (the PR is draft, I am assuming they are waiting for the Celery change to be released: celery/kombu#1776)

For now there is not really a way to fix this unless there is a way we can do a redis constraint to avoid 5.x. Or maybe once the next Celery 5.3.x release includes limiting redis-py to 4.x we can possibly limit Celery provider to that version of Celery?

What you think should happen instead

Airflow should be able to send jobs to workers when using Celery Executor with Redis

How to reproduce

1. Start Airflow 2.7.0 with Celery Executor with Redis 5.0.0 installed by default (at the time of this writing)
2. Run a DAG task
3. The scheduler fails to send the job to the worker

Workaround:

1. Limit redis-py to 4.x the same way the upcoming release of Celery 5.3.x does, by using this in requirements.txt: redis>=4.5.2,<5.0.0,!=4.5.5
2. Start Airflow 2.7.0 with Celery Executor
3. Run a DAG task
4. The task runs successfully

Operating System

Debian GNU/Linux 11 (bullseye)

Versions of Apache Airflow Providers

apache-airflow-providers-celery==3.3.2

Deployment

Docker-Compose

Deployment details

I am using bitnami/airflow:2.7.0 image in Docker Compose when I first encountered this issue, but I will test with Breeze as well shortly and then update this issue.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[alexbegg]

I am not able to reproduce a 2nd time. Once I implemented the workaround I mentioned to downgrade to Python redis package 4.x (redis>=4.5.2,<5.0.0,!=4.5.5) in my environment, and the tasks start running again, when I take off that pip requirement and redis-py goes back to 5.0.0, the tasks now still continue to run. I am going to investigate further using Breeze to see what is happening.

[alexbegg]

I tried with Breeze with

breeze start-airflow --use-airflow-version 2.7.0 --executor CeleryExecutor --airflow-extras celery,postgres --backend postgres

and I also can't reproduce the issue I was having.

It was a recurring issue after upgrading to 2.7.0 even after multiple restarts of my Docker Compose environment, and only got resolved after downgrading redis-py to 4.x.

The issue was strange because the error says module 'redis' has no attribute 'client', but I confirmed the installed redis python package at version 5.0.0 did still have a client module inside of it...

If nobody else can figure out what might have happened, maybe close this issue (since Celery package will soon restrict Redis-py to <5.0.0 anyways)

[potiuk]

Thanks @alexbegg -> yes we should limit redis deps in this case . PR #33773 created. We will also cherry-pick it to 2.7.1

[alexbegg]

@potiuk just an FYI that Celery has now released v5.3.3 which includes the change to limit Redis client to 4.x, and Kombu's PR is now open and approved, so likely will be merged soon.