Container restarts due to segmentation violation

[instantdreams]

Describe the bug

My simple opencloud container instance restarts multiple times, and the logs indicate this is due to failures in the service.go module.

Steps to reproduce

1. Start instance
2. Wait a while; optionally try to add a file or folder
3. Check the logs and identify a segmentation violation

Expected behavior

For the instance not to restart multiple times.

Actual behavior

The instance always restarts and remains stable for an indeterminate length of time.

Setup

Environment details:

• Debian 6.1.135-1
• Docker Engine 28.4.0
• Docker Compose 2.39.2
• config and apps directories mounted on ext4
• data directory mounted on zfs

Here is my compose.yaml file:

Details

name: opencloud
services:
  opencloud:
    command:
      - -c
      - opencloud init || true; opencloud server
    container_name: opencloud
    domainname: example.com
    entrypoint:
      - /bin/sh
    environment:
      DIRECTORY_APPS: /srv/opencloud/apps
      DIRECTORY_CONFIG: /srv/opencloud/config
      DIRECTORY_DATA: /mnt/storage/opencloud
      IDM_ADMIN_PASSWORD: [redacted]
      IDM_CREATE_DEMO_USERS: "false"
      MICRO_REGISTRY_ADDRESS: localhost:9233
      NATS_NATS_HOST: 0.0.0.0
      NATS_NATS_PORT: "9233"
      NOTIFICATIONS_SMTP_AUTHENTICATION: auto
      NOTIFICATIONS_SMTP_ENCRYPTION: starttls
      NOTIFICATIONS_SMTP_HOST: smtp.sendgrid.net
      NOTIFICATIONS_SMTP_INSECURE: "false"
      NOTIFICATIONS_SMTP_PASSWORD: [redacted]
      NOTIFICATIONS_SMTP_PORT: "465"
      NOTIFICATIONS_SMTP_SENDER: admin@example.com
      NOTIFICATIONS_SMTP_USERNAME: [redacted]
      OC_DISABLE_VERSIONING: "true"
      OC_INSECURE: "true"
      OC_LOG_COLOR: "false"
      OC_LOG_LEVEL: debug
      OC_LOG_PRETTY: "false"
      OC_PERSISTENT_STORE: nats-js-kv
      OC_PERSISTENT_STORE_NODES: localhost:9233
      OC_URL: https://opencloud.example.com
      PROXY_ENABLE_BASIC_AUTH: "true"
      PROXY_HTTP_ADDR: 0.0.0.0:9200
      PROXY_TLS: "false"
      STORAGE_USERS_DRIVER: posix
      STORAGE_USERS_POSIX_GENERAL_SPACE_ALIAS_TEMPLATE: '{{.SpaceType}}/{{.SpaceName | replace " " "-" | lower}}'
      STORAGE_USERS_POSIX_GENERAL_SPACE_PATH_TEMPLATE: projects/{{.SpaceName | replace " " "-" | lower}}
      STORAGE_USERS_POSIX_PERSONAL_SPACE_ALIAS_TEMPLATE: '{{.SpaceType}}/{{.User.Username | lower}}'
      STORAGE_USERS_POSIX_PERSONAL_SPACE_PATH_TEMPLATE: users/{{.User.Username | lower}}
      STORAGE_USERS_POSIX_PROPAGATOR: sync
      STORAGE_USERS_POSIX_SCAN_DEBOUNCE_DELAY: 4s
      STORAGE_USERS_POSIX_USE_SPACE_GROUPS: "false"
      STORAGE_USERS_POSIX_WATCH_FS: "false"
      STORAGE_USERS_POSIX_WATCH_TYPE: inotifywait
      TZ: America/Edmonton
    extra_hosts:
      - opencloud.example.com=192.168.1.91
    hostname: opencloud
    image: opencloudeu/opencloud-rolling:latest
    ports:
      - mode: ingress
        target: 9200
        published: "9202"
        protocol: tcp
    restart: unless-stopped
    volumes:
      - type: bind
        source: /srv/opencloud/config
        target: /etc/opencloud
        bind:
          create_host_path: true
      - type: bind
        source: /mnt/storage/opencloud
        target: /var/lib/opencloud
        bind:
          create_host_path: true
      - type: bind
        source: /srv/opencloud/apps
        target: /var/lib/opencloud/web/assets/apps
        bind:
          create_host_path: true
      - type: bind
        source: /etc/localtime
        target: /etc/localtime
        read_only: true
        bind:
          create_host_path: true

Here is a link to my opencloud-20250929.log log files with debug enabled.

Additional context

I am trying to stand up a file server only, no collabra, ODIC, or other apps. I wish to stabilise it before adding additional utilities.

[rhafer]

I think the crash itself should be fixed by: #1565 (that fix will be part of the next release)

However as @micbar noted in https://gist.github.com/instantdreams/3f62595ab58691d1d547ea3d98552d85?permalink_comment_id=5780626#gistcomment-5780626 there seems to be some configuration error with the search service. You might want to retry with the opencloud-rolling:daily image, which includes the above fix already. That might give a hint in the logs about what is actually wrong with the search service.

[instantdreams]

I will spin up the container with the daily build.

I've made some comments on my opencloud-20250929.log gist but I'll repeat them here.

Directory /srv is on an ext4 partition on the primary SSD for the server.
Directory /mnt/storage/ is on a ZFS array.

When I started the container I used the following steps:

mkdir /srv/opencloud/config/
mkdir /srv/opencloud/apps/
mkdir /mnt/storage/opencloud/
sudo chown --recursive 1000:1000 /srv/opencloud/
sudo chown --recursive 1000:1000 /mnt/storage/opencloud/
cd /srv/opencloud/
cp .env.example .env
cp .env.secrets.example .env.secrets
nano .env.secrets  # then add the passwords and tokens
docker compose pull
docker compose up --detach

I created a /srv/opencloud/data/ directory and changed the configuration to point there. The instance is now running without going into panic and I've been able to upload files successfully.

I have performed a comparison of the /srv/opencloud/data/ and /mnt/storage/opencloud/ directories and found the following difference:

search folder:

$ ls -lha /mnt/storage/opencloud/search/bleve/store/ /srv/opencloud/data/search/bleve/store/
/mnt/storage/opencloud/search/bleve/store/:
total 66K
drwx------ 2 1000 1000   3 Sep 29 10:57 .
drwx------ 3 1000 1000   4 Sep 29 10:57 ..
-rw------- 1 1000 1000 64K Sep 29 10:57 root.bolt

/srv/opencloud/data/search/bleve/store/:
total 352K
drwx------ 2 1000 1000 4.0K Sep 30 09:26 .
drwx------ 3 1000 1000 4.0K Sep 30 08:35 ..
-rw------- 1 1000 1000 288K Sep 30 09:26 000000000093.zap
-rw------- 1 1000 1000  64K Sep 30 09:26 root.bolt

thumbnails folder:

$ ls -lha /mnt/storage/opencloud/thumbnails/ /srv/opencloud/data/thumbnails/
ls: cannot access '/mnt/storage/opencloud/thumbnails/': No such file or directory

/srv/opencloud/data/thumbnails/:
total 12K
drwx------ 3 1000 1000 4.0K Sep 30 09:08 .
drwxr-xr-x 9 1000 1000 4.0K Sep 30 09:08 ..
drwx------ 3 1000 1000 4.0K Sep 30 09:08 files

So with the ZFS mount opencloud was not able to create the zap file or the thumbnails directory.

[micbar]

So with the ZFS mount opencloud was not able to create the zap file or the thumbnails directory.

Is that reproducible or just a hiccup?

[instantdreams]

Using the daily version I fully reset my instance using the following:

sudo rm --recursive /mnt/storage/opencloud/
sudo rm --recursive /srv/opencloud/apps/
sudo rm --recursive /srv/opencloud/data/
sudo rm /srv/opencloud/config/opencloud.yaml
mkdir /srv/opencloud/apps
mkdir /mnt/storage/opencloud
sudo chown 1000:1000 --recursive /srv/opencloud/
sudo chown 1000:1000 --recursive /mnt/storage/opencloud/
docker compose pull;docker compose down;sleep 4;docker compose up --detach;

I then performed the following steps:

1. Access opencloud.example.com
2. Log in using admin account
3. Create personal account
4. Log in using personal account
5. Upload folder to personal account

There are no panics and everything seems to be working.

[instantdreams]

Maybe a hiccup, maybe shifting to the daily build fixed it.

[rhafer]

Ok. I'll close this for now. Feel free to re-open if the issue reappears.