assets can still be missing sha256, may be due to "unembargo"

[yarikoptic]

originally observed/reported in dandi/dandisets#257 (comment) but likely better reside in a dedicated report here since might require some fixing of code. Assets in 000235 are lacking sha256 despite them being out there for quite awhile. I believe the same problem is overall in 00023[5678] which we had to exclude from backup for now since we do need sha256. Since those "appeared" only recently but are "low" in their id (we have 000337 ATM) I think they are unembargoed and iirc @AlmightyYakob suggested that there could be remaining issues in unemabrgoe'ing process. So - the code should be fixed and assets in these dandisets have their sha256 computed.

related question: Do we have some way to figure out that a dandiset was unemabrgoed by looking at metadata?

[jjnesbitt]

I can make a fix to the unembargo process today. Regarding the existing null sha256 asset checksums, I could manually re-run checksums on those assets, and update here once that's completed. Would that be sufficient?

[yarikoptic]

sure, a fix to code + manual trigger of re-run is what I think too is needed.

[dandibot]

🚀 Issue was released in v0.2.50 🚀

[jjnesbitt]

@yarikoptic these assets have been fix and now there are no asset blobs with missing sha256s.

[yarikoptic]

eh, unfortunately it is not yet the case! I reenable backup of those 4 dandisets only for that to error out since there is still no sha256 in their assets' metadata. @AlmightyYakob please double check. Here is crude "proof" - you can see them having only dandi:dandi-etag :

(git)lena:~/proj/dandi/dandi-cli-master[master]git
$> dandi ls -r --metadata all -f json_pp dandi://dandi/000235 | grep -A2 -e path -e digest
2022-09-23 10:54:52,051 [ WARNING] A newer version (0.46.3) of dandi/dandi-cli is available. You are using 0.46.2+4.gae35684
    "path": "000235",
    "size": 45887231195,
    "version": "draft"
--
      "digest": {
        "dandi:dandi-etag": "5f16079a446ba716499c3d3491cd0ef4-43"
      },
--
      "path": "sub-Fish40-GCaMP-vlgut-FBd-7dpf-RandomWave/sub-Fish40-GCaMP-vlgut-FBd-7dpf-RandomWave_ses-20210929T112126_ophys.nwb",
      "repository": "https://dandiarchive.org/",
      "schemaKey": "Asset",
--
    "path": "sub-Fish40-GCaMP-vlgut-FBd-7dpf-RandomWave/sub-Fish40-GCaMP-vlgut-FBd-7dpf-RandomWave_ses-20210929T112126_ophys.nwb",
    "size": 2850110801
  },
--
      "digest": {
        "dandi:dandi-etag": "83a0cefed2c1bf31379a4e1464c5f3c4-47"
      },
--

[jjnesbitt]

eh, unfortunately it is not yet the case! I reenable backup of those 4 dandisets only for that to error out since there is still no sha256 in their assets' metadata. @AlmightyYakob please double check. Here is crude "proof" - you can see them having only dandi:dandi-etag :

(git)lena:~/proj/dandi/dandi-cli-master[master]git
$> dandi ls -r --metadata all -f json_pp dandi://dandi/000235 | grep -A2 -e path -e digest
2022-09-23 10:54:52,051 [ WARNING] A newer version (0.46.3) of dandi/dandi-cli is available. You are using 0.46.2+4.gae35684
    "path": "000235",
    "size": 45887231195,
    "version": "draft"
--
      "digest": {
        "dandi:dandi-etag": "5f16079a446ba716499c3d3491cd0ef4-43"
      },
--
      "path": "sub-Fish40-GCaMP-vlgut-FBd-7dpf-RandomWave/sub-Fish40-GCaMP-vlgut-FBd-7dpf-RandomWave_ses-20210929T112126_ophys.nwb",
      "repository": "https://dandiarchive.org/[](https://dandiarchive.org/)",
      "schemaKey": "Asset",
--
    "path": "sub-Fish40-GCaMP-vlgut-FBd-7dpf-RandomWave/sub-Fish40-GCaMP-vlgut-FBd-7dpf-RandomWave_ses-20210929T112126_ophys.nwb",
    "size": 2850110801
  },
--
      "digest": {
        "dandi:dandi-etag": "83a0cefed2c1bf31379a4e1464c5f3c4-47"
      },
--

Ahh, I messed up and didn't save the assets that point to the underlying asset blobs. I've re-saved them, can you check now?

[yarikoptic]

I've rechecked with the above dandi ls command across those dandisets -- seems to be kosher. I will try now to backup again. thanks!