Skip to content

Tasks state stuck #1428

Open
Open
@ttftw

Description

@ttftw

I'm using Flower in a Docker container, running it with celery -A worker flower.

When I leave Flower running, it will accumulate tasks that are stale and not update status. When I look in Redis, I see the task completed and I can very the work was actually completed, but the task stays stuck on "received" or "started".

Image

The difference between "Processed" and "Succeeded" are all tasks stuck on "received" or "started" but all tasks are actually complete.

Image

Some of the tasks, even though are complete, do not show a time they were started

Image

Over time, these will accumulate where 20-50% or so of my tasks are showing a stuck state even though every task actually completed.

Sometimes other fields are missing, like "name" and "args", but args was passed or else it wouldn't have completed.

Image

I did notice when I pasted in the sys info below, Flower seems to see "settings -> transport:amqp results:disabled", even though the transport is Redis and results are enabled. I'm not sure if this is an artifact of me exec this command into the container.

Here is the config the worker/beat/Flower is using:

app = Celery(
    "worker",
    broker=f"{REDIS_URL}/{REDIS_DB_BROKER}?ssl_cert_reqs=CERT_REQUIRED",
    backend=f"{REDIS_URL}/{REDIS_DB_RESULTS}?ssl_cert_reqs=CERT_REQUIRED",
)

app.conf.update(
    worker_pool="threads",
    worker_pool_restarts=True,
    broker_connection_retry_on_startup=True,
    beat_schedule={
        "update-cache": {
            "task": "worker.update_cache",
            "schedule": crontab(minute="*"),
        },
        "build-cache": {
            "task": "worker.build_cache",
            "schedule": crontab(minute="0"),
        },
    },
)

System information
flower -> flower:2.0.1 tornado:6.4.2 humanize:4.11.0
software -> celery:5.4.0 (opalescent) kombu:5.4.2 py:3.13.2
billiard:4.2.1 py-amqp:5.3.1
platform -> system:Linux arch:64bit
kernel version:6.1.0-26-amd64 imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:amqp results:disabled

deprecated_settings: None

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions