Disallow direct access of orphaned assets

[jjnesbitt]

Originally, we used to require that you access an asset through the dandiset and version it belonged to. For a while now, we've had an endpoint for accessing assets directly, via their asset ID. However, this leads to a situation where you can easily access an orphaned asset, which can be misleading, as you might assume it's an "active" asset. We also intend to garbage collect orphaned assets, which would prevent access to that asset anyways.

I think we should either update the direct asset endpoint to only return assets that have an associated version (and thus would 404 on orphaned asset access), or return some other 4xx level response.

@yarikoptic @satra Do you know of anyone that make significant use of orphaned assets?

[jjnesbitt]

FWIW, my motivation for this is here in #2731. Generally speaking though, orphaned assets could easily cause other issues when it comes to metadata that is derived from the version(s) an asset is associated with.

[satra]

i hope not, but i do know that some people have shared asset links from draft dandisets that they then updated, that others could then not replicate. i'm not sure i have seen people share that direct api link for assets.

[waxlamp]

We should return 404 for orphaned assets. We rely on GC to clean up after ourselves, but that in itself indicates that we don't believe garbage-collectible assets are valid for use (since they could disappear at literally any time, so we can adopt the legal fiction that they have already disappeared).

Obiter dicta: this points to a weakness in our API, where a PUT to an asset generates a new asset and leaves a dead one in the system. The property that the URL for the asset actually violates the expected semantics for PUT, and exposes details of our garbage collection based implementation in a way that complicates the codebase and the UX.

[yarikoptic]

For 'significant use' it would require some analysis along the machinery which @CodyCBakerPhD developed for https://usage.dandiarchive.org/ .

But as for extra ways on how orphaned assets could still be accessed -- we do have a full history of the draft versions preserved as part of the https://github.com/dandisets repos. So if any user reuses SPECIFIC git checkout of the draft version, that user can potentially access outdated draft version assets (good for reproducibility!). Sure thing we do not have to guarantee such access and it would be ok if access to outdated never released draft asset results in some 4xx prompting them to "git pull --ff-only" ;-)

[CodyCBakerPhD]

but that in itself indicates that we don't believe garbage-collectible assets are valid for use (since they could disappear at literally any time, so we can adopt the legal fiction that they have already disappeared).

That statement applies equally to any non-orphaned draft assets

For 'significant use' it would require some analysis along the machinery which @CodyCBakerPhD developed for https://usage.dandiarchive.org/ .

It absolutely happens, 000409 was/is the best example since it was popular and in draft state for years, and most/all assets were replaced prior to recent publication

However, this leads to a situation where you can easily access an orphaned asset, which can be misleading, as you might assume it's an "active" asset.

I will just point out that some users are aware that contents can be directly accessed from the S3 bucket (e.g., via s5cmd or the aws s3 CLI) - even if you blocked the DANDI API from allowing access to orphaned access, significant machinery along the lines of embargo would be require to prevent direct S3 access, so IDK how significant the overall benefit is

[jjnesbitt]

I will just point out that some users are aware that contents can be directly accessed from the S3 bucket (e.g., via s5cmd or the aws s3 CLI) - even if you blocked the DANDI API from allowing access to orphaned access, significant machinery along the lines of embargo would be require to prevent direct S3 access, so IDK how significant the overall benefit is

This isn't coming from the perspective of hiding any data in the bucket, embargoed or not. Once any unembargoed data is placed into the public bucket, we happily allow any public access. It's more about the overall consistency of data in the archive. Orphaned assets break some of the assumptions we make about assets, and may lead to weird situations, as mentioned in the issue description.

As Roni pointed out, this is also due to the fact that an update to an asset creates another asset. It would be nice if an update to a draft asset would simply update the asset, leaving the asset ID unchanged. But that's a separate problem.

[CodyCBakerPhD]

Once any unembargoed data is placed into the public bucket

Embargoed data is already on the public bucket. Access is denied through a tagging/iam system. That is what I refer to here; to block direct S3 access to an orphaned asset, but prior to garbage collection, it would need that kind of setup. Not worth it in my opinion (but neither is the API proposal; also, the title of this issue is ambiguous to the 'type' of direct access - you seem to mean API, and I am commenting on non-API), just pointing it out.

TBH if garbage collection was turned on and ran often enough, that would solve both issues

[jjnesbitt]

Embargoed data is already on the public bucket. Access is denied through a tagging/iam system. That is what I refer to here; to block direct S3 access to an orphaned asset, but prior to garbage collection, it would need that kind of setup. Not worth it in my opinion (but neither is the API proposal; also, the title of this issue is ambiguous to the 'type' of direct access - you seem to mean API, and I am commenting on non-API), just pointing it out.

My prior statement was an enumeration over both cases of uploading open data directly, or embargoed data being unembargoed and having its tag removed, thus being open.

Again, I'm not proposing we block S3 access to orphaned assets. As you point out, I'm strictly referring to accessing orphaned assets from the API, via the direct asset endpoints (/assets/{asset_id}/). Strictly speaking, accessing assets via S3 is nonsensical, as the data stored in S3 are referred to as asset blobs. So yes, by "direct" access I'm referring to API access. The reason I use the term "direct" is to contrast the other mode of access, which is via the dandiset version (/dandiset/{dandiset_id}/versions/{version}/assets/{asset_id}/).