object_store: range request with suffix

[clbarnes]

We need to read the last n bytes of a file whose size we do not know. This is supported by HTTP range requests, and access to local files (via Seek), can object_store support it?

object_store::GetOptions::range could take, instead of a core::ops::Range, something like this: https://github.com/clbarnes/byteranges-rs/blob/0a953e7c580e96b65fe28e61ed460d6e221dcd8d/src/request.rs#L6-L51 . So long as From<RangeBounds> is supported this may not have to break any existing code.

The alternative is a HEAD request to find the length and then a range request using the offset from 0, which is twice the round trips.

[tustvold]

I seem to remember GCS being the only cloud store that supports this, but I could be mistaken. Have you done any research into this?

[clbarnes]

The S3 docs say they support the Range header, and specifically states that they don't support multipart/byteranges responses, but doesn't specifically say they don't support suffixes https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html#API_GetObject_RequestSyntax

[tustvold]

Have you tested this, I seem to remember it did not but my memory is hazy

[normanrz]

Suffix range requests are supported by all major cloud providers. Here is an example against S3 (proxied via Cloudfront):

curl -H 'Range: bytes=-524292' https://static.webknossos.org/data/zarr_v3/l4_sample/color/1/c/0/3/3/1 | wc -c

[tustvold]

I would be happy to review a PR making this change. The use of From<RangeBounds> is clever and I like it, although I suspect many use-cases have hard-coded Range<usize> which will then require modification to support this.

Suffix range requests are supported by all major cloud providers

Good to know, I'm not sure where I got the impression otherwise. Ultimately it isn't a massive problem if they don't support them, we just report this as an error to the user.

[clbarnes]

we just report this as an error to the user.

Sounds like users might consider that a problem 😁

[tustvold]

Well yes, but if the store itself doesn't support them there isn't really all that much we can do 😄

[tustvold]

It would appear that Azure Blob Storage does not support suffix range headers, although it does support prefix range headers

https://learn.microsoft.com/en-us/rest/api/storageservices/specifying-the-range-header-for-blob-service-operations

[clbarnes]

I think I'm closing in on an implementation here. Allowing suffixes could mean that some bytes are fetched twice but I don't think that's too much of a problem, as it's on the user to have knowledge of the resource they're looking into. I've been using HttpRange as the type name (and in all the functions which use it) - would be nice to have something more terse but it's the best description I think, even for stores which aren't HTTP-backed.

I think that replacing references to Range<usize> in the API with T: Into<HttpRange> + Copy should make for a seamless upgrade path, but the range field on the GetOptions struct would need to be deprecated and a different field of type HttpRange (called http_range?), probably raising an error if both are defined.

What we do with suffixes on azure is another question - probably just a log_once::warn_once!() when a suffix is requested, and then a HEAD request to find the length and regular range request.

[tustvold]

In the API with T: Into + Copy

This would still be a breaking API change, something I am rather hesitant about. It would also not be object safe, which is likely more problematic.

Did you see, it would appear at least one of the major cloud providers do not support this, and I would not be surprised if there are other implementations that do not - #4611 (comment)