Async_delete does not delete files from local storage when using UI

[StijnPruijssers]

Problems deleting automatically deleting artifacts with async_delete service

I have been happily using self-hosted ClearML for a while. I was cleaning up datasets/tasks and noticed that everything on the file server remains even after deleting with the UI. The odd thing is this only happens when using the UI to delete tasks. When using the CLI to delete a dataset/task the files are removed from the file server.

My setup/config: windows 11 + docker compose. The only real change is that I use a NAS drive as the file server which I mounted via NFS. The server has full access to the NAS and all permissions. The docker compose file is pasted at the bottom.

I also have a .env file I use which specifies the CLEARML_FILES_HOST environment variable as such: CLEARML_FILES_HOST="<file-serverhost-url:8081>". I spin up the containers by calling docker compose -f clearml_compose.yml --env-file config.env up.

Looking in the logs of the async_delete service in docker I see no incoming logs either when using the delete via CLI or UI. Logs:

However, in the clearml-fileserver logs I see that there are deletions if I use the CLI but not the UI:

Is there a way this can be fixed so the task management/clean up is also possible from UI?

version: "3.6"
services:

  apiserver:
    command:
    - apiserver
    container_name: clearml-apiserver
    image: clearml/server:latest # When started it pulled version clearml/server:2.2.0
    restart: unless-stopped
    volumes:
    - c:/opt/clearml/logs:/var/log/clearml
    - c:/opt/clearml/config:/opt/clearml/config
    - NAS:/mnt/fileserver
    depends_on:
      - redis
      - mongo
      - elasticsearch
      - fileserver
    environment:
      CLEARML_ELASTIC_SERVICE_HOST: elasticsearch
      CLEARML_ELASTIC_SERVICE_PORT: 9200
      CLEARML_MONGODB_SERVICE_HOST: mongo
      CLEARML_MONGODB_SERVICE_PORT: 27017
      CLEARML_REDIS_SERVICE_HOST: redis
      CLEARML_REDIS_SERVICE_PORT: 6379
      CLEARML_SERVER_DEPLOYMENT_TYPE: win10
      CLEARML__apiserver__pre_populate__enabled: "true"
      CLEARML__apiserver__pre_populate__zip_files: "/opt/clearml/db-pre-populate"
      CLEARML__apiserver__pre_populate__artifacts_path: "/mnt/fileserver"
      CLEARML__services__async_urls_delete__enabled: "true"
      CLEARML__services__async_urls_delete__fileserver__url_prefixes: "[${CLEARML_FILES_HOST:-}]"
    ports:
    - "8008:8008"
    networks:
      - backend
      - frontend

  elasticsearch:
    networks:
      - backend
    container_name: clearml-elastic
    environment:
      bootstrap.memory_lock: "true"
      cluster.name: clearml
      cluster.routing.allocation.node_initial_primaries_recoveries: "500"
      cluster.routing.allocation.disk.watermark.low: 500mb
      cluster.routing.allocation.disk.watermark.high: 500mb
      cluster.routing.allocation.disk.watermark.flood_stage: 500mb
      discovery.type: "single-node"
      http.compression_level: "7"
      node.name: clearml
      reindex.remote.whitelist: "'*.*'"
      xpack.security.enabled: "false"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    image: elasticsearch:8.17.0
    restart: unless-stopped
    volumes:
      - c:/opt/clearml/data/elastic_7:/usr/share/elasticsearch/data
      - /usr/share/elasticsearch/logs

  fileserver:
    networks:
      - backend
      - frontend
    command:
    - fileserver
    container_name: clearml-fileserver
    image: clearml/server:latest # When started it pulled version clearml/server:2.2.0
    environment:
      CLEARML__fileserver__delete__allow_batch: "true"
      CLEARML__fileserver__auth__enabled: "false"
    restart: unless-stopped
    volumes:
    - c:/opt/clearml/logs:/var/log/clearml
    - NAS:/mnt/fileserver
    - c:/opt/clearml/config:/opt/clearml/config

    ports:
    - "8081:8081"

  mongo:
    networks:
      - backend
    container_name: clearml-mongo
    image: mongo:6.0.19
    restart: unless-stopped
    command: --setParameter internalQueryMaxBlockingSortMemoryUsageBytes=196100200
    volumes:
    - c:/opt/clearml/data/mongo_4/db:/data/db
    - c:/opt/clearml/data/mongo_4/configdb:/data/configdb

  redis:
    networks:
      - backend
    container_name: clearml-redis
    image: redis:7.4.1
    restart: unless-stopped
    volumes:
    - c:/opt/clearml/data/redis:/data

  webserver:
    command:
    - webserver
    container_name: clearml-webserver
    image: clearml/server:latest # When started it pulled version clearml/server:2.2.0
    restart: unless-stopped
    volumes:
    - c:/clearml/logs:/var/log/clearml
    depends_on:
      - apiserver
    ports:
    - "8080:80"
    networks:
      - backend
      - frontend

  async_delete:
    depends_on:
      - apiserver
      - redis
      - mongo
      - elasticsearch
      - fileserver
    container_name: async_delete
    image: clearml/server:latest # When started it pulled version clearml/server:2.2.0
    networks:
      - backend
    restart: unless-stopped
    environment:
      CLEARML_ELASTIC_SERVICE_HOST: elasticsearch
      CLEARML_ELASTIC_SERVICE_PORT: 9200
      CLEARML_MONGODB_SERVICE_HOST: mongo
      CLEARML_MONGODB_SERVICE_PORT: 27017
      CLEARML_REDIS_SERVICE_HOST: redis
      CLEARML_REDIS_SERVICE_PORT: 6379
      PYTHONPATH: /opt/clearml/apiserver
      CLEARML__services__async_urls_delete__fileserver__url_prefixes: "[${CLEARML_FILES_HOST:-}]"
    entrypoint:
      - python3
      - -m
      - jobs.async_urls_delete
      - --fileserver-host
      - http://fileserver:8081
    volumes:
    - c:/opt/clearml/logs:/var/log/clearml
    - c:/opt/clearml/config:/opt/clearml/config

volumes:
  NAS:
   name: NAS
   driver_opts:
      type: "nfs"
      o: "addr=192.168.0.2,nolock,rw,soft,nfsvers=4"
      device: ":/volume1/AI/Apps/clearml"

networks:
  backend:
    driver: bridge
  frontend:
    name: frontend
    driver: bridge

[evgeny-tech]

Hi @StijnPruijssers, can you please give an example of the command that you issue in SDK and it successfully deletes the data and the example of the parallel operation in UI that does not work?
Please inspect logs from async_delete service. Do you see any difference when deleting with SDK and with UI?