--mode verify: Two changes to dandisets without update to timestamps #90
Description
Followup to prior older
full log with pointers to more logs
dandi@drogon:/mnt/backup/dandi/dandisets$ duct flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron-nonzarr.lock bash -c '/mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron --mode verify'
2025-10-07T16:09:43-0400 [INFO ] con-duct: duct is executing 'flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron-nonzarr.lock bash -c /mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron --mode verify'...
2025-10-07T16:09:43-0400 [INFO ] con-duct: Log files will be written to .duct/logs/2025.10.07T16.09.43-64178_
2025-10-07T16:09:51-0400 [WARNING ] dandi: A newer version (0.71.4) of dandi/dandi-cli is available. You are using 0.66.3
2025-10-07T16:12:30-0400 [ERROR ] backups2datalad: Job failed on input <Dandiset 000363/draft>:
Traceback (most recent call last):
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/aioutil.py", line 177, in dowork
outp = await func(inp)
^^^^^^^^^^^^^^^
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/datasetter.py", line 206, in update_dandiset
changed = await self.sync_dataset(
^^^^^^^^^^^^^^^^^^^^^^^^
dandiset, ds, dmanager, error_on_change=True
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/datasetter.py", line 251, in sync_dataset
await syncer.prune_deleted()
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/syncer.py", line 93, in prune_deleted
raise UnexpectedChangeError(
...<3 lines>...
)
backups2datalad.util.UnexpectedChangeError: Dandiset 000363: Asset '.duct/logs/2025.03.06T17.17.16-3058713_stderr' deleted from Dandiset but timestamp was not updated on server
2025-10-07T16:29:08-0400 [ERROR ] backups2datalad: Job failed on input <Dandiset 001246/draft>:
+ Exception Group Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/aioutil.py", line 177, in dowork
| outp = await func(inp)
| ^^^^^^^^^^^^^^^
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/datasetter.py", line 206, in update_dandiset
| changed = await self.sync_dataset(
| ^^^^^^^^^^^^^^^^^^^^^^^^
| dandiset, ds, dmanager, error_on_change=True
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| )
| ^
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/datasetter.py", line 250, in sync_dataset
| await syncer.sync_assets()
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/syncer.py", line 77, in sync_assets
| report = await async_assets(
| ^^^^^^^^^^^^^^^^^^^
| self.dandiset, self.ds, self.manager, self.tracker, self.error_on_change
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| )
| ^
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/asyncer.py", line 509, in async_assets
| anyio.create_task_group() as nursery,
| ~~~~~~~~~~~~~~~~~~~~~~~^^
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/anyio/_backends/_asyncio.py", line 815, in __aexit__
| raise BaseExceptionGroup(
| "unhandled errors in a TaskGroup", self._exceptions
| )
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/zarr.py", line 373, in needs_sync
| local_paths.remove(path)
| ~~~~~~~~~~~~~~~~~~^^^^^^
| KeyError: '.zattrs'
|
| During handling of the above exception, another exception occurred:
|
| Traceback (most recent call last):
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/zarr.py", line 567, in sync_zarr
| await zsync.run()
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/zarr.py", line 152, in run
| if not await self.needs_sync(client, last_sync, to_delete):
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/zarr.py", line 375, in needs_sync
| self.check_change(f"entry {path!r} added")
| ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/zarr.py", line 497, in check_change
| raise UnexpectedChangeError(
| ...<3 lines>...
| )
| backups2datalad.util.UnexpectedChangeError: Dandiset 001246: Zarr 1c0e023e-150d-43a7-948b-dc46dd514ca6: entry '.zattrs' added, but Dandiset draft timestamp was not updated on server
+------------------------------------
2025-10-07T16:31:45-0400 [ERROR ] backups2datalad: An error occurred:
Traceback (most recent call last):
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/__main__.py", line 119, in wrapped
await f(datasetter, *args, **kwargs)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/__main__.py", line 229, in update_from_backup
await datasetter.update_from_backup(dandisets, exclude=exclude)
File "/home/dandi/miniconda3/envs/dandisets-2/lib/python3.13/site-packages/backups2datalad/datasetter.py", line 141, in update_from_backup
raise RuntimeError(
f"Backups for {quantify(len(report.failed), 'Dandiset')} failed"
)
RuntimeError: Backups for 2 Dandisets failed
Logs saved to /mnt/backup/dandi/dandisets/.git/dandi/backups2datalad/2025/10/2025.10.07.20.09.50Z.log
action summary:
publish (notneeded: 2)
2025-10-07T16:31:45-0400 [INFO ] con-duct: Summary:
Exit Code: 1
Command: flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron-nonzarr.lock bash -c /mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron --mode verify
Log files location: .duct/logs/2025.10.07T16.09.43-64178_
Wall Clock Time: 1322.390 sec
Memory Peak Usage (RSS): 1.4 GB
Memory Average Usage (RSS): 581.5 MB
Virtual Memory Peak Usage (VSZ): 8.8 TB
Virtual Memory Average Usage (VSZ): 2.0 TB
Memory Peak Percentage: 1.8%
Memory Average Percentage: 0.6166279969064191%
CPU Peak Usage: 236.4%
Average CPU Usage: 35.82204176334106%
so we have two issues
1 - deleted file without timestamp update
backups2datalad.util.UnexpectedChangeError: Dandiset 000363: Asset '.duct/logs/2025.03.06T17.17.16-3058713_stderr' deleted from Dandiset but timestamp was not updated on server
2 - added to zarr file without timestamp update
backups2datalad.util.UnexpectedChangeError: Dandiset 001246: Zarr 1c0e023e-150d-43a7-948b-dc46dd514ca6: entry '.zattrs' added, but Dandiset draft timestamp was not updated on server