Server crashed while api tests

[ScharfViktor]

Describe the bug

https://ci.opencloud.eu/repos/3/pipeline/756/213

server is unavailable and this continues until the wrapper restarts the server

server log:

{"level":"error","service":"storage-users","pkg":"rgrpc","traceid":"8f0fcc558e3102216b0cc93a5b158ffc","error":"node.Xattr /root/.opencloud/storage/users/users/61c21065-20f4-42b4-9326-125e18ddd87a user.oc.name: no data available","id":"0792389f-0e43-4064-978c-3f071f66a265","time":"2025-03-25T12:57:04Z","message":"could not read node"}
{"level":"error","service":"storage-users","pkg":"rgrpc","traceid":"8f0fcc558e3102216b0cc93a5b158ffc","error":"node.Xattr /root/.opencloud/storage/users/users/61c21065-20f4-42b4-9326-125e18ddd87a user.oc.name: no data available","status":{"code":15,"message":"error listing spaces","trace":"8f0fcc558e3102216b0cc93a5b158ffc"},"filters":[{"type":2,"Term":{"Id":{"opaque_id":"0031b3bc-f80e-4283-85bd-251520fe070c$0792389f-0e43-4064-978c-3f071f66a265!0792389f-0e43-4064-978c-3f071f66a265"}}},{"type":4,"Term":{"SpaceType":"+grant"}}],"time":"2025-03-25T12:57:04Z","message":"failed to list storage spaces"}
{"level":"debug","time":"2025-03-25T12:57:04Z","message":"space deleted event: {idp:\"https://opencloud-server:9200\"  opaque_id:\"f0151c7a-fa9f-438e-94d4-f9dbd79c8fb5\"  type:USER_TYPE_PRIMARY opaque_id:\"0031b3bc-f80e-4283-85bd-251520fe070c$0792389f-0e43-4064-978c-3f071f66a265!0792389f-0e43-4064-978c-3f071f66a265\" Alice Hansen map[] 2025-03-25 12:57:04.633143505 +0000 UTC}"}
panic: send on closed channel

goroutine 6326 [running]:
github.com/nats-io/nats%2ego.(*kvs).WatchFiltered.func2()
	/go/src/github.com/opencloud-eu/opencloud/vendor/github.com/nats-io/nats.go/kv.go:1102 +0x85
created by time.goFunc
	/usr/local/go/src/time/sleep.go:215 +0x2d
2025/03/25 12:57:04 [ocwrapper] opencloud server exited with code 2
2025/03/25 12:57:04 [ocwrapper] Retry starting opencloud server... (retry 1)
2025/03/25 12:57:05 [ocwrapper] Stopping OpenCloud server...
2025/03/25 12:57:05 [ocwrapper] Restarting OpenCloud server...
{"level":"fatal","service":"opencloud","error":"listen tcp 127.0.0.1:9250: bind: address already in use","time":"2025-03-25T12:57:05Z","message":"could not start listener"}
2025/03/25 12:57:05 [ocwrapper] Process found. Waiting...
2025/03/25 12:57:05 [ocwrapper] opencloud server exited with code 1
2025/03/25 12:57:05 [ocwrapper] Retry starting opencloud server... (retry 2)
{"level":"fatal","service":"opencloud","error":"listen tcp 127.0.0.1:9250: bind: address already in use","time":"2025-03-25T12:57:06Z","message":"could not start listener"}
2025/03/25 12:57:06 [ocwrapper] Process found. Waiting...
2025/03/25 12:57:06 [ocwrapper] opencloud server exited with code 1
2025/03/25 12:57:06 [ocwrapper] Retry starting opencloud server... (retry 3)
{"level":"fatal","service":"opencloud","error":"listen tcp 127.0.0.1:9250: bind: address already in use","time":"2025-03-25T12:57:06Z","message":"could not start listener"}
2025/03/25 12:57:06 [ocwrapper] Process found. Waiting...
2025/03/25 12:57:06 [ocwrapper] opencloud server exited with code 1
2025/03/25 12:57:06 [ocwrapper] Retry starting opencloud server... (retry 4)
{"level":"error","service":"storage-users","pkg":"rgrpc","error":"could not find space for path /root/.opencloud/storage/users/projects","path":"/root/.opencloud/storage/users/projects","time":"2025-03-25T12:57:06Z","message":"could not assimilate item"}
{"level":"error","service":"storage-users","pkg":"rgrpc","error":"error: not found: path not found in cache:/root/.opencloud/storage/users/projects","path":"/root/.opencloud/storage/users/projects","time":"2025-03-25T12:57:06Z","message":"could not get ids for path"}
{"level":"error","service":"storage-users","pkg":"rgrpc","error":"error: not found: path not found in cache:/root/.opencloud/storage/users","path":"/root/.opencloud/storage/users","time":"2025-03-25T12:57:06Z","message":"could not get ids for path"}

Steps:

• build dev container: make -C opencloud dev-docker
• run docker LOCAL_TEST=true STORAGE_DRIVER=posix make -C tests/acceptance/docker start-server
• run test suite make test-acceptance-api TEST_SERVER_URL="https://opencloud-server:9200" BEHAT_FEATURE=tests/acceptance/features/apiSharingNgLinkSharePermission/createLinkShare.feature

Expected behavior

tests are green

Actual behavior

server fail

Setup

Please describe how you started the server and provide a list of relevant environment variables or configuration files.

Details

OC_XXX=somevalue
OC_YYY=somevalue
PROXY_XXX=somevalue

Additional context

Add any other context about the problem here.

[rhafer]

I suspect that this as caused by #464
The panic happens inside the nats module in a location touched by this recent commit: nats-io/nats.go@a6608dc

Unfortunately I am not able to reproduce the issue locally.

[micbar]

Fixed with the downgrade of nats.