Is it possible to persist data accross container restarts?

[johaag]

Hello all,

First of all, I want to thank @danyi1212 for this great project! It looks very promising.

We are currently evaluating Celery Insights as a successor for Flower. With Flower, it is possible to persist the home folder of the user, which is running Flower in the container. This way, past data is present in Flower, even if the container is restarted.

Is this also possible for Celery Insights?

Edit: Flower will only persist its data, if a DB is configured. In our case this DB is stored in the home folder, hence persisting the home folder worked for us.

[johaag]

Seems like it is working in the very same way as for Flower by creating a volume and mapping it to the home folder of user myuser (which is the user running Celery Insights in the container).

This is how I add it to the Docker Compose file:

services:
  celery-insights:
    image: ghcr.io/danyi1212/celery-insights:latest
    ports:
      - "8555:8555"
    environment:
      BROKER_URL: redis://redis:6379
      RESULT_BACKEND: redis://redis:6379
    volumes:
      - insights_data:/home/myuser/
    networks:
      - my-net

volumes:
  insights_data:

[danyi1212]

Very interesting, I didn't know that. That may be something more low-level inside Celery itself, I'll be looking into how that works 🤔

[johaag]

Sorry to say, that my testing of my "solution" was not good enough. Seems like I was mislead by cached data in the browser.

Testing it again it turns out, that persisting the home folder will not make the data persistent. For Flower we actually have configured a DB, which is stored in the home folder. I was not aware of that in the first place.

Sorry for the confusion!

[danyi1212]

Hey, currently there is no way of persisting the data between restarts.

This is possible, though I wouldn't follow Flower's approach of using a Shelve DB due to limited performance.
I've had plans of integrating a pluggable storage layer that could use a Pebble DB / Redis / etscd / TiKV or else.

If this will get enough interest, I could definitely push on this idea 🌟

[johaag]

Thank you for your input. Looks like you responded at the same time as I added my response, that it seems to work by persisting the running user's home drive. Would you consider this to be a valid workaround until you integrated a pluggable storage layer (which would boost performance)?
Or are there other aspects other than performance to not do it that way?

[danyi1212]

Considering it does work for you, that means that it is already being written to disk.
As a current workaround, that may work fine as it most probably the same as Flower. If they recommend it, I will follow it as well.

For a long-term solution, I cannot know the integrity or overhead of this existing storage, I'll have to check that. Using a pluggable storage layer will probably be a better eventual solution.

Thank you for sharing that! 🙂

[johaag]

Persisting the running user's home drive does actually not work (see my other comment).