Caching Issue causing exit 139 (segmentation fault)

[GabHoo]

Hi!
I am experiencing problem with old requests since they are stored somehow. This happened once when I stopped the docker container but my script was still running and kept sending requests. Later, after I restarted the server etc, my celery workers restarted from old pending requests, even if at that point, my script was not running anymore. I can see how this behavior is desirable in some situations, but in my case I would like to free the cache, and have the workers available and free when I start the container fresh.
How to do that?

Thank you for your work!

[GabHoo]

EDIT:
I also realized that when restarting, it will happens that 2 workers start working on the same document, maybe because the queue of the document from this storage is not properly handled. But because they start working on the same document (it only happens after a couple of jobs are done) they then exit with code 139, which as suggested here could be cause by trying to access memory same memory address by different processes

[drmingler]

@GabHoo To avoid task persistence on restart you can add this to the celery_config.py file

celery_app.conf.broker_transport_options = {
    'visibility_timeout': 3600,  # 1 hour
    'fanout_prefix': True,
    'fanout_patterns': True
}