AIRFLOW__SCHEDULER__TASK_QUEUED_TIMEOUT configuration ignored

[CreeperBeatz]

Apache Airflow version

3.1.5

If "Other Airflow 3 version" selected, which one?

No response

What happened?

For my workload, I ingest video files and then create tasks to process each one. It is normal to create 30+ tasks that will get processed in the span of 40+min. On backfill, it's absolutely normal for my GPU workers to not be available and for a task to wait 1h+ to be executed.

This is why I changed AIRFLOW__SCHEDULER__TASK_QUEUED_TIMEOUT to 10 000 (see image below).

I also changed my task definition to allow a task to be in processing for 24h, so this is not the issue either.

However, even when setting that, my tasks fail and the retry mechanism kicks in. I've bandaged fixed it by increasing the number of retries, but it's not optimal.

What you think should happen instead?

AIRFLOW__SCHEDULER__TASK_QUEUED_TIMEOUT should be taken into account, and tasks should not timeout on around 15min.

All tasks eventually complete. With 20 runs × 2.5 min each and 1 worker, the last task should wait ~50 min in the queue - well within the 100,000s timeout.

How to reproduce

1. Install Airflow

uv pip install "apache-airflow[celery]==3.1.5" \
    --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.1.5/constraints-3.12.txt"

2. Initialize and configure

airflow db migrate
sed -i 's/^executor = .*/executor = CeleryExecutor/' airflow.cfg
sed -i 's/^task_queued_timeout = .*/task_queued_timeout = 100000.0/' airflow.cfg

3. Create the DAG

Save as dags/long_running_task_dag.py:

from datetime import datetime, timedelta
from time import sleep

from airflow.sdk import dag, task


@dag(
    dag_id="long_running_task_bug_repro",
    schedule=None,
    start_date=datetime(2025, 1, 1),
    catchup=False,
    default_args={
        "owner": "airflow",
        "retries": 0,
    },
)
def long_running_task_bug_repro():

    @task(
        queue="gpu_queue",
        execution_timeout=timedelta(hours=24),
    )
    def gpu_simulation():
        """Simulate GPU processing - sleeps for 2.5 minutes."""
        sleep(150)

    gpu_simulation()


long_running_task_bug_repro()

4. Start services

Terminal 1 - Scheduler:

airflow api-server --port 8080 &
airflow scheduler &
airflow dag-processor &
airflow triggerer &
airflow celery worker --queues default

Terminal 2 - Single worker (simulates limited GPU resources):

airflow celery worker --queues gpu_queue --concurrency 1

5. Trigger multiple DAG runs

for i in $(seq 1 20); do
    airflow dags trigger long_running_task_bug_repro
done

6. Wait ~15 minutes

Operating System

Ubuntu LTS 22.04

Versions of Apache Airflow Providers

apache-airflow-providers-celery==3.14.0
apache-airflow-providers-common-compat==1.10.0
apache-airflow-providers-common-io==1.7.0
apache-airflow-providers-common-sql==1.30.0
apache-airflow-providers-smtp==2.4.0
apache-airflow-providers-standard==1.10.0

Deployment

Virtualenv installation

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[sania-2912]

Hi @potiuk,

I’ve gone through the reproduction steps and can consistently reproduce the issue on Airflow 3.1.5 with CeleryExecutor.

From my view, it looks like task_queued_timeout is not being honored correctly for Celery tasks waiting in a non-default queue with limited worker concurrency, leading to premature retries (~15 min) despite a much higher configured timeout.

I’d like to work on a fix for this and submit a PR. My plan is to:

1. Trace where queued task timeout is enforced in the scheduler/executor flow
2. Verify whether the timeout is being overridden or hard-limited elsewhere (e.g. scheduler loop, heartbeat, or executor state handling)
3. Add a regression test covering long queue wait times with CeleryExecutor

Please let me know if there are any architectural considerations or preferred areas to start with. Happy to proceed.

Thanks!

[potiuk]

Proceed. The discussion is better when there is PR and I am not necessarily the best person to comment but when you open PR it will likely drag attention of those who can take a look (just please don't tag me as it might be someone else to look at it )