posixfs revisions are flaky

[butonic]

When uploading the same file over again and again there seems to be a race condition:

2025-10-24T14:02:41+02:00 ERR failed to update existing node error="Failed to set all xattrs: xattr.Set /home/jfd/.opencloud/storage/users/users/some-admin-user-id-0000-000000000000/.oc-nodes/09/85/63/ff/-e134-45aa-b82e-c2d77d70191f.REV.2025-10-24T12:02:41.153848281Z user.oc.blobsize: no such file or directory" driver=posix host.name=wrecker pid=204432 pkg=rhttp request-id=wrecker/5WfkpmfQgS-003028 service=storage-users
2025-10-24T14:02:41+02:00 ERR error uploading file error="Failed to set all xattrs: xattr.Set /home/jfd/.opencloud/storage/users/users/some-admin-user-id-0000-000000000000/.oc-nodes/09/85/63/ff/-e134-45aa-b82e-c2d77d70191f.REV.2025-10-24T12:02:41.153848281Z user.oc.blobsize: no such file or directory" datatx=simple host.name=wrecker path=/16aad981-8c1b-4435-8941-954bd0b95076 pkg=rhttp service=storage-users
2025-10-24T14:02:41+02:00 ERR http end="24/Oct/2025:14:02:41 +0200" host=127.0.0.1 host.name=wrecker method=PUT pkg=rhttp proto=HTTP/1.1 service=storage-users size=0 start="24/Oct/2025:14:02:41 +0200" status=500 time_ns=104049581 traceid=82f09122932b60bb2db28c1727095d1e uri=/data/simple/16aad981-8c1b-4435-8941-954bd0b95076 url=/16aad981-8c1b-4435-8941-954bd0b95076
2025-10-24T14:02:41+02:00 ERR http end="24/Oct/2025:14:02:41 +0200" host=127.0.0.1 host.name=wrecker method=PUT pkg=rhttp proto=HTTP/1.1 service=frontend size=0 start="24/Oct/2025:14:02:41 +0200" status=500 time_ns=106043922 traceid=633b8c07dd83f50f86100c9723966e2d uri=/data url=/
2025-10-24T14:02:41+02:00 ERR PUT request to data server failed name=eu.opencloud.web.ocdav path=/test.txt request-id=wrecker/5WfkpmfQgS-003011 service=ocdav spaceid=storage-users-1$d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 traceid=aae0674bc10f8708a01476a5383daf16

How to reproduce

I updated the benchmark cli to add more options:

go run ./opencloud/cmd/opencloud benchmark client  -X PUT -k --data-binary @/tmp/10mb -H 'Content-Type: application/octet-stream' -j 10 -u admin:admin 'https://localhost:9200/dav/spaces/storage-users-1$d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8/test.txt'

This uses 10 goroutines to upload a 10mb file to the same file in opencloud and create a lot of revisions.

After 1k files I kill the process and see several uploads that have either not been fully uploaded or have not been processed:

❯ ls -l ~/.opencloud/storage/users/uploads
insgesamt 251468
-rw-r--r-- 1 jfd jfd 10318088 24. Okt 14:20 01502061-747e-4362-8a77-e0428a77c313
-rw------- 1 jfd jfd      914 24. Okt 14:20 01502061-747e-4362-8a77-e0428a77c313.info
-rw-r--r-- 1 jfd jfd 10318088 24. Okt 14:02 11abced4-e0ab-44ec-bd64-796de76b7a00
-rw------- 1 jfd jfd      806 24. Okt 14:02 11abced4-e0ab-44ec-bd64-796de76b7a00.info
-rw-r--r-- 1 jfd jfd 10318088 24. Okt 14:02 16aad981-8c1b-4435-8941-954bd0b95076
-rw------- 1 jfd jfd      806 24. Okt 14:02 16aad981-8c1b-4435-8941-954bd0b95076.info
-rw-r--r-- 1 jfd jfd  9555687 24. Okt 14:02 1dc5b17e-cdf6-49dc-a7fe-4e2f4eabecba
-rw------- 1 jfd jfd      805 24. Okt 14:02 1dc5b17e-cdf6-49dc-a7fe-4e2f4eabecba.info
-rw-r--r-- 1 jfd jfd 10318088 24. Okt 14:02 2ad12eb3-0732-43d9-8726-3dfec50f4c8d
-rw------- 1 jfd jfd      786 24. Okt 14:02 2ad12eb3-0732-43d9-8726-3dfec50f4c8d.info
-rw-r--r-- 1 jfd jfd 10318088 24. Okt 14:20 367127ed-351d-4f0f-9150-081d4a7eaf9e
-rw------- 1 jfd jfd      806 24. Okt 14:20 367127ed-351d-4f0f-9150-081d4a7eaf9e.info
-rw-r--r-- 1 jfd jfd 10318088 24. Okt 14:02 3d93092a-830d-4dc8-a4b5-7b7aa479844b
-rw------- 1 jfd jfd      806 24. Okt 14:02 3d93092a-830d-4dc8-a4b5-7b7aa479844b.info
-rw-r--r-- 1 jfd jfd 10318088 24. Okt 14:20 4454fbb4-3fee-42b1-b99e-eca0cb49b5e2
-rw------- 1 jfd jfd      806 24. Okt 14:20 4454fbb4-3fee-42b1-b99e-eca0cb49b5e2.info
-rw-r--r-- 1 jfd jfd 10318088 24. Okt 14:02 47902c5b-0251-475e-b27c-7c5cd69866d5
-rw------- 1 jfd jfd      786 24. Okt 14:02 47902c5b-0251-475e-b27c-7c5cd69866d5.info
-rw-r--r-- 1 jfd jfd 10318088 24. Okt 14:02 47ea1e00-2760-4d53-beaf-4528e5423deb
-rw------- 1 jfd jfd      786 24. Okt 14:02 47ea1e00-2760-4d53-beaf-4528e5423deb.info
-rw-r--r-- 1 jfd jfd 10318088 24. Okt 14:20 52cacb5e-773a-446e-ad38-90964d3502c0
-rw------- 1 jfd jfd      806 24. Okt 14:20 52cacb5e-773a-446e-ad38-90964d3502c0.info
-rw-r--r-- 1 jfd jfd 10318088 24. Okt 14:02 53449afc-c3ac-4f09-99b8-c238d6c74e33
-rw------- 1 jfd jfd      786 24. Okt 14:02 53449afc-c3ac-4f09-99b8-c238d6c74e33.info
-rw-r--r-- 1 jfd jfd 10318088 24. Okt 14:16 6adb1b9e-3c89-4e8a-908e-a8fb85056a46
-rw------- 1 jfd jfd      806 24. Okt 14:16 6adb1b9e-3c89-4e8a-908e-a8fb85056a46.info
-rw-r--r-- 1 jfd jfd        0 24. Okt 14:16 7764ccfa-6d37-4fb3-8691-89f5498ebd4a
-rw------- 1 jfd jfd      806 24. Okt 14:16 7764ccfa-6d37-4fb3-8691-89f5498ebd4a.info
-rw-r--r-- 1 jfd jfd 10318088 24. Okt 14:02 7d839d6d-d29e-49cb-8632-3dbbcc649778
-rw------- 1 jfd jfd      786 24. Okt 14:02 7d839d6d-d29e-49cb-8632-3dbbcc649778.info
-rw-r--r-- 1 jfd jfd 10318088 24. Okt 14:20 7f945acb-91ac-43a3-92b7-c0df7701bebd
-rw------- 1 jfd jfd      806 24. Okt 14:20 7f945acb-91ac-43a3-92b7-c0df7701bebd.info
-rw-r--r-- 1 jfd jfd  7360231 24. Okt 14:02 815b7242-c917-43a6-96bc-8b2a2f33c30b
-rw------- 1 jfd jfd      806 24. Okt 14:02 815b7242-c917-43a6-96bc-8b2a2f33c30b.info
-rw-r--r-- 1 jfd jfd        0 24. Okt 14:20 90461db0-b056-4a62-b8a7-90c81d7ad4b5
-rw-r--r-- 1 jfd jfd 10318088 24. Okt 14:02 92859a93-084f-4db5-9e7c-499c7933cf10
-rw------- 1 jfd jfd      786 24. Okt 14:02 92859a93-084f-4db5-9e7c-499c7933cf10.info
-rw-r--r-- 1 jfd jfd        0 24. Okt 14:20 a28b6895-b017-4ed2-a400-65c181fe03bb
-rw-r--r-- 1 jfd jfd 10318088 24. Okt 14:02 af929e73-3438-4ec3-a6d3-ac156c7ef9c3
-rw------- 1 jfd jfd      786 24. Okt 14:02 af929e73-3438-4ec3-a6d3-ac156c7ef9c3.info
-rw-r--r-- 1 jfd jfd 10318088 24. Okt 14:20 b2292d58-4b13-4e43-b819-1f695d3bd68e
-rw------- 1 jfd jfd      914 24. Okt 14:20 b2292d58-4b13-4e43-b819-1f695d3bd68e.info
-rw-r--r-- 1 jfd jfd 10318088 24. Okt 14:16 baa000b6-afc3-477e-8b7f-630f11cdfea7
-rw------- 1 jfd jfd      806 24. Okt 14:16 baa000b6-afc3-477e-8b7f-630f11cdfea7.info
-rw-r--r-- 1 jfd jfd  4181735 24. Okt 14:02 bc98d141-e627-438d-a73f-b439491ae979
-rw------- 1 jfd jfd      806 24. Okt 14:02 bc98d141-e627-438d-a73f-b439491ae979.info
-rw-r--r-- 1 jfd jfd 10318088 24. Okt 14:02 c27c4ba0-2e51-4604-ba66-97fd32c9fbbf
-rw------- 1 jfd jfd      786 24. Okt 14:02 c27c4ba0-2e51-4604-ba66-97fd32c9fbbf.info
-rw-r--r-- 1 jfd jfd 10318088 24. Okt 14:02 cb8be57b-0d82-44d6-ac25-d899ccce1541
-rw------- 1 jfd jfd      786 24. Okt 14:02 cb8be57b-0d82-44d6-ac25-d899ccce1541.info
-rw-r--r-- 1 jfd jfd  9211623 24. Okt 14:02 d55a2343-b0e0-4e6e-b36b-55c5bf58e48d
-rw------- 1 jfd jfd      805 24. Okt 14:02 d55a2343-b0e0-4e6e-b36b-55c5bf58e48d.info
-rw-r--r-- 1 jfd jfd 10318088 24. Okt 14:20 dcf1e074-06ad-4887-88b0-e1d2755c0086
-rw------- 1 jfd jfd      806 24. Okt 14:20 dcf1e074-06ad-4887-88b0-e1d2755c0086.info
-rw-r--r-- 1 jfd jfd 10318088 24. Okt 14:20 f8ac12e7-496f-429f-8f5a-ecdf98c02463
-rw------- 1 jfd jfd      914 24. Okt 14:20 f8ac12e7-496f-429f-8f5a-ecdf98c02463.info

The revision in the log above seems to have been written with colliding filename (hence the .1 suffix), but actually has a different mtime:

❯ ls -l ./-e134-45aa-b82e-c2d77d70191f.REV.2025-10-24T12:02:41.153848281Z
-rw------- 1 jfd jfd 10318088 24. Okt 14:02 ./-e134-45aa-b82e-c2d77d70191f.REV.2025-10-24T12:02:41.153848281Z
❯ ls -l ./-e134-45aa-b82e-c2d77d70191f.REV.2025-10-24T12:02:41.153848281Z.1
-rw------- 1 jfd jfd 10318088 24. Okt 14:02 ./-e134-45aa-b82e-c2d77d70191f.REV.2025-10-24T12:02:41.153848281Z.1
❯ getfattr -d ./-e134-45aa-b82e-c2d77d70191f.REV.2025-10-24T12:02:41.153848281Z
# file: -e134-45aa-b82e-c2d77d70191f.REV.2025-10-24T12:02:41.153848281Z
user.oc.blobid="7a5d4011-06a8-46bf-ad3e-db2f54ee38a3"
user.oc.blobsize="10318088"
user.oc.cs.adler32=0stzzxpw==
user.oc.cs.md5=0sM+DkwBI7ZyFmAdjghWKGmA==
user.oc.cs.sha1=0sKHJPgxzuFqXWsq2Lv/CsNbvxc5Y=
user.oc.mtime="2025-10-24T12:02:41.176592038Z"
user.oc.type="1"

❯ getfattr -d ./-e134-45aa-b82e-c2d77d70191f.REV.2025-10-24T12:02:41.153848281Z.1
# file: -e134-45aa-b82e-c2d77d70191f.REV.2025-10-24T12:02:41.153848281Z.1
user.oc.blobid="0efa1eda-88d0-46de-aa9d-6213f8ba6aeb"
user.oc.blobsize="10318088"
user.oc.cs.adler32=0stzzxpw==
user.oc.cs.md5=0sM+DkwBI7ZyFmAdjghWKGmA==
user.oc.cs.sha1=0sKHJPgxzuFqXWsq2Lv/CsNbvxc5Y=
user.oc.mtime="2025-10-24T12:02:41.153848281Z"
user.oc.type="1"

trying to fix the uploads

the uploads are visible in the sessions:

❯ go run ./opencloud/cmd/opencloud storage-users uploads sessions
Sessions:
┌──────────────────────────────────────┬──────────────────────────────────────┬──────────┬──────────┬──────────┬──────────────────────────────────────┬──────────────────────────────────────┬───────────────────────────┬────────────┬──────────────────────┬─────────────┐
│                Space                 │              Upload Id               │   Name   │  Offset  │   Size   │              Executant               │                Owner                 │          Expires          │ Processing │      Scan Date       │ Scan Result │
├──────────────────────────────────────┼──────────────────────────────────────┼──────────┼──────────┼──────────┼──────────────────────────────────────┼──────────────────────────────────────┼───────────────────────────┼────────────┼──────────────────────┼─────────────┤
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 07b98e11-ca0d-4b2d-bdb0-886c74b6a2a5 │ test.txt │ 1117927  │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:56:14+02:00 │ false      │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 11abced4-e0ab-44ec-bd64-796de76b7a00 │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:30+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 16aad981-8c1b-4435-8941-954bd0b95076 │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:41+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 1dc5b17e-cdf6-49dc-a7fe-4e2f4eabecba │ test.txt │ 9555687  │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:57+02:00 │ false      │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 2ad12eb3-0732-43d9-8726-3dfec50f4c8d │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:30+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 3d93092a-830d-4dc8-a4b5-7b7aa479844b │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:48+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 4454fbb4-3fee-42b1-b99e-eca0cb49b5e2 │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:20:54+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 47902c5b-0251-475e-b27c-7c5cd69866d5 │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:30+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 47ea1e00-2760-4d53-beaf-4528e5423deb │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:30+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 53449afc-c3ac-4f09-99b8-c238d6c74e33 │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:30+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 6adb1b9e-3c89-4e8a-908e-a8fb85056a46 │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:16:08+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 7764ccfa-6d37-4fb3-8691-89f5498ebd4a │ test.txt │ 0        │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:16:08+02:00 │ false      │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 7d839d6d-d29e-49cb-8632-3dbbcc649778 │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:30+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 815b7242-c917-43a6-96bc-8b2a2f33c30b │ test.txt │ 7360231  │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:57+02:00 │ false      │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 90461db0-b056-4a62-b8a7-90c81d7ad4b5 │ test.txt │ 4984551  │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:20:54+02:00 │ false      │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 92859a93-084f-4db5-9e7c-499c7933cf10 │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:30+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ a28b6895-b017-4ed2-a400-65c181fe03bb │ test.txt │ 6541031  │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:20:54+02:00 │ false      │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ af929e73-3438-4ec3-a6d3-ac156c7ef9c3 │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:30+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ baa000b6-afc3-477e-8b7f-630f11cdfea7 │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:16:08+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ bc98d141-e627-438d-a73f-b439491ae979 │ test.txt │ 4181735  │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:57+02:00 │ false      │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ c27c4ba0-2e51-4604-ba66-97fd32c9fbbf │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:30+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ cb8be57b-0d82-44d6-ac25-d899ccce1541 │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:30+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ d55a2343-b0e0-4e6e-b36b-55c5bf58e48d │ test.txt │ 9211623  │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:57+02:00 │ false      │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ e063ab90-e562-4a2c-add9-5009b0b34f89 │ test.txt │ 1216231  │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:56:14+02:00 │ false      │ 0001-01-01T00:00:00Z │             │
└──────────────────────────────────────┴──────────────────────────────────────┴──────────┴──────────┴──────────┴──────────────────────────────────────┴──────────────────────────────────────┴───────────────────────────┴────────────┴──────────────────────┴─────────────┘

we can try to resume them:

❯ go run ./opencloud/cmd/opencloud storage-users uploads sessions --resume
Sessions:
┌──────────────────────────────────────┬──────────────────────────────────────┬──────────┬──────────┬──────────┬──────────────────────────────────────┬──────────────────────────────────────┬───────────────────────────┬────────────┬──────────────────────┬─────────────┐
│                Space                 │              Upload Id               │   Name   │  Offset  │   Size   │              Executant               │                Owner                 │          Expires          │ Processing │      Scan Date       │ Scan Result │
├──────────────────────────────────────┼──────────────────────────────────────┼──────────┼──────────┼──────────┼──────────────────────────────────────┼──────────────────────────────────────┼───────────────────────────┼────────────┼──────────────────────┼─────────────┤
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 07b98e11-ca0d-4b2d-bdb0-886c74b6a2a5 │ test.txt │ 1117927  │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:56:14+02:00 │ false      │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 11abced4-e0ab-44ec-bd64-796de76b7a00 │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:30+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 16aad981-8c1b-4435-8941-954bd0b95076 │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:41+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 1dc5b17e-cdf6-49dc-a7fe-4e2f4eabecba │ test.txt │ 9555687  │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:57+02:00 │ false      │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 2ad12eb3-0732-43d9-8726-3dfec50f4c8d │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:30+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 3d93092a-830d-4dc8-a4b5-7b7aa479844b │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:48+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 4454fbb4-3fee-42b1-b99e-eca0cb49b5e2 │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:20:54+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 47902c5b-0251-475e-b27c-7c5cd69866d5 │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:30+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 47ea1e00-2760-4d53-beaf-4528e5423deb │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:30+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 53449afc-c3ac-4f09-99b8-c238d6c74e33 │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:30+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 6adb1b9e-3c89-4e8a-908e-a8fb85056a46 │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:16:08+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 7764ccfa-6d37-4fb3-8691-89f5498ebd4a │ test.txt │ 0        │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:16:08+02:00 │ false      │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 7d839d6d-d29e-49cb-8632-3dbbcc649778 │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:30+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 815b7242-c917-43a6-96bc-8b2a2f33c30b │ test.txt │ 7360231  │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:57+02:00 │ false      │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 90461db0-b056-4a62-b8a7-90c81d7ad4b5 │ test.txt │ 4984551  │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:20:54+02:00 │ false      │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 92859a93-084f-4db5-9e7c-499c7933cf10 │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:30+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ a28b6895-b017-4ed2-a400-65c181fe03bb │ test.txt │ 6541031  │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:20:54+02:00 │ false      │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ af929e73-3438-4ec3-a6d3-ac156c7ef9c3 │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:30+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ baa000b6-afc3-477e-8b7f-630f11cdfea7 │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:16:08+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ bc98d141-e627-438d-a73f-b439491ae979 │ test.txt │ 4181735  │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:57+02:00 │ false      │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ c27c4ba0-2e51-4604-ba66-97fd32c9fbbf │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:30+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ cb8be57b-0d82-44d6-ac25-d899ccce1541 │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:30+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ d55a2343-b0e0-4e6e-b36b-55c5bf58e48d │ test.txt │ 9211623  │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:57+02:00 │ false      │ 0001-01-01T00:00:00Z │             │
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ e063ab90-e562-4a2c-add9-5009b0b34f89 │ test.txt │ 1216231  │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:56:14+02:00 │ false      │ 0001-01-01T00:00:00Z │             │
└──────────────────────────────────────┴──────────────────────────────────────┴──────────┴──────────┴──────────┴──────────────────────────────────────┴──────────────────────────────────────┴───────────────────────────┴────────────┴──────────────────────┴─────────────┘

which leaves only one:

 go run ./opencloud/cmd/opencloud storage-users uploads sessions
Sessions:
┌──────────────────────────────────────┬──────────────────────────────────────┬──────────┬──────────┬──────────┬──────────────────────────────────────┬──────────────────────────────────────┬───────────────────────────┬────────────┬──────────────────────┬─────────────┐
│                Space                 │              Upload Id               │   Name   │  Offset  │   Size   │              Executant               │                Owner                 │          Expires          │ Processing │      Scan Date       │ Scan Result │
├──────────────────────────────────────┼──────────────────────────────────────┼──────────┼──────────┼──────────┼──────────────────────────────────────┼──────────────────────────────────────┼───────────────────────────┼────────────┼──────────────────────┼─────────────┤
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 7d839d6d-d29e-49cb-8632-3dbbcc649778 │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:30+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
└──────────────────────────────────────┴──────────────────────────────────────┴──────────┴──────────┴──────────┴──────────────────────────────────────┴──────────────────────────────────────┴───────────────────────────┴────────────┴──────────────────────┴─────────────┘

using restart we can fix that as well:

❯ go run ./opencloud/cmd/opencloud storage-users uploads sessions --resume
Sessions:
┌──────────────────────────────────────┬──────────────────────────────────────┬──────────┬──────────┬──────────┬──────────────────────────────────────┬──────────────────────────────────────┬───────────────────────────┬────────────┬──────────────────────┬─────────────┐
│                Space                 │              Upload Id               │   Name   │  Offset  │   Size   │              Executant               │                Owner                 │          Expires          │ Processing │      Scan Date       │ Scan Result │
├──────────────────────────────────────┼──────────────────────────────────────┼──────────┼──────────┼──────────┼──────────────────────────────────────┼──────────────────────────────────────┼───────────────────────────┼────────────┼──────────────────────┼─────────────┤
│ d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 │ 7d839d6d-d29e-49cb-8632-3dbbcc649778 │ test.txt │ 10318088 │ 10318088 │ some-admin-user-id-0000-000000000000 │ some-admin-user-id-0000-000000000000 │ 2025-10-25T14:02:30+02:00 │ true       │ 0001-01-01T00:00:00Z │             │
└──────────────────────────────────────┴──────────────────────────────────────┴──────────┴──────────┴──────────┴──────────────────────────────────────┴──────────────────────────────────────┴───────────────────────────┴────────────┴──────────────────────┴─────────────┘
❯ go run ./opencloud/cmd/opencloud storage-users uploads sessions --restart
Sessions:
┌───────┬───────────┬──────┬────────┬──────┬───────────┬───────┬─────────┬────────────┬───────────┬─────────────┐
│ Space │ Upload Id │ Name │ Offset │ Size │ Executant │ Owner │ Expires │ Processing │ Scan Date │ Scan Result │
└───────┴───────────┴──────┴────────┴──────┴───────────┴───────┴─────────┴────────────┴───────────┴─────────────┘
❯ go run ./opencloud/cmd/opencloud storage-users uploads sessions
Sessions:
┌───────┬───────────┬──────┬────────┬──────┬───────────┬───────┬─────────┬────────────┬───────────┬─────────────┐
│ Space │ Upload Id │ Name │ Offset │ Size │ Executant │ Owner │ Expires │ Processing │ Scan Date │ Scan Result │
└───────┴───────────┴──────┴────────┴──────┴───────────┴───────┴─────────┴────────────┴───────────┴─────────────┘

hm now I still have an info file here, but it does not seem to have an upload blob next to it:

❯ ls -l ~/.opencloud/storage/users/uploads
insgesamt 4
-rw------- 1 jfd jfd 805 24. Okt 14:02 1dc5b17e-cdf6-49dc-a7fe-4e2f4eabecba.info

[rhafer]

I updated the benchmark cli to add more options

@butonic Is there a pull request for this already somewhere? This might be handy for testing the PR I am currently working on.

[rhafer]

This will hopefully fixed along with: #414 , haven't tried in depth yet.

[fschade]

finding: it only happens if the initial node does not exist!
idea: the node is not there, the 10 concurrent put requests try to create that node which fails

[fschade]

the first upload batches may fail with status 412 (Precondition Failed) due to the unfinished upload of the base node (first upload)! Every put request AFTER the node is created (first put request) is successful!

the uploaded data is not purged automatically from the uploads storage-users location to prevent data loss,
the storage-users uploads sessions cmd can be used for cleanup (e.g. using a cron) or restoring.

2025-10-24T14:02:41+02:00 ERR failed to update existing node error="Failed to set all xattrs: xattr.Set /home/jfd/.opencloud/storage/users/users/some-admin-user-id-0000-000000000000/.oc-nodes/09/85/63/ff/-e134-45aa-b82e-c2d77d70191f.REV.2025-10-24T12:02:41.153848281Z user.oc.blobsize: no such file or directory" driver=posix host.name=wrecker pid=204432 pkg=rhttp request-id=wrecker/5WfkpmfQgS-003028 service=storage-users
2025-10-24T14:02:41+02:00 ERR error uploading file error="Failed to set all xattrs: xattr.Set /home/jfd/.opencloud/storage/users/users/some-admin-user-id-0000-000000000000/.oc-nodes/09/85/63/ff/-e134-45aa-b82e-c2d77d70191f.REV.2025-10-24T12:02:41.153848281Z user.oc.blobsize: no such file or directory" datatx=simple host.name=wrecker path=/16aad981-8c1b-4435-8941-954bd0b95076 pkg=rhttp service=storage-users
2025-10-24T14:02:41+02:00 ERR http end="24/Oct/2025:14:02:41 +0200" host=127.0.0.1 host.name=wrecker method=PUT pkg=rhttp proto=HTTP/1.1 service=storage-users size=0 start="24/Oct/2025:14:02:41 +0200" status=500 time_ns=104049581 traceid=82f09122932b60bb2db28c1727095d1e uri=/data/simple/16aad981-8c1b-4435-8941-954bd0b95076 url=/16aad981-8c1b-4435-8941-954bd0b95076
2025-10-24T14:02:41+02:00 ERR http end="24/Oct/2025:14:02:41 +0200" host=127.0.0.1 host.name=wrecker method=PUT pkg=rhttp proto=HTTP/1.1 service=frontend size=0 start="24/Oct/2025:14:02:41 +0200" status=500 time_ns=106043922 traceid=633b8c07dd83f50f86100c9723966e2d uri=/data url=/
2025-10-24T14:02:41+02:00 ERR PUT request to data server failed name=eu.opencloud.web.ocdav path=/test.txt request-id=wrecker/5WfkpmfQgS-003011 service=ocdav spaceid=storage-users-1$d0e6c8e8-deef-4a6f-945e-75aba6a2f5a8 traceid=aae0674bc10f8708a01476a5383daf16

does not exist anymore (tested multiple times with up to 20k uploads), as mentioned by @rhafer, that seems to be fixed as a side-effect in one of the other upload related fixes!

i created a k6 test as reference which can be found here.

closing

FYI: @dragotin, @rhafer, @aduffeck and @butonic