Issue with Concurrent Execution using Dask in Flask-Celery Setup

[sri-dhurkesh]

I've been working on a Flask project utilizing Celery to handle asynchronous tasks, particularly processing CSV files using the Pandas dataframe. Recently, I attempted to replace Pandas with Dask inside Celery, but encountered a runtime error - "raise RuntimeError('cannot schedule new futures after shutdown')" when executing concurrent tasks simultaneously.

I'm reaching out to the community for insights and possible solutions to overcome this issue. If anyone has experience with integrating Dask into Celery or has ideas on how to address this specific error, your input would be greatly appreciated.

Looking forward to your suggestions and discussions. Thanks in advance!