Proposal: Pushing blob chunks could have a max chunk size hint

[gabivlj]

Description

Hi! We've encountered a use-case we'd love to discuss over here. We're proposing a new header that could be useful for registries that are behind proxies. Those proxies usually buffer requests so for big layers it degrades performance and memory usage of the server. It would be nice if the registry, either on upload creation (POST /v2/:name/blobs/uploads/) or in another way to hint the client the preferred max chunk size.

For example, I've seen places that document the header OCI-Chunk-Min-Length, the purpose is hinting a minimum chunk size for clients. It would be amazing to see the counter-part OCI-Chunk-Max-Length.

Is it something interesting to see in https://distribution.github.io/distribution/spec/api/?

I feel like this would be incredibly useful for registries that have limited resources.

[sudo-bmitch]

Does this affect Docker's streaming method for a blob push (not defined by OCI yet, but since it's used by docker it's well supported by registries), non-chunked pushes (used by a lot of other tools), or the final chunk size?

[gabivlj]

This affects mainly the POST and PATCH method on chunked uploads. It would work the same as OCI-Chunk-Min-Length, but by defining a maximum chunk size to send in PATCH to not overwhelm the server.

I sent a PR to the distribution repository showcasing how this change would look like.

[sudo-bmitch]

Docker's streaming API uses a patch request without a content-length value. It's effectively the same as a post/put, but the digest is not known in advance, so they push the full blob with a single patch.

How do proxies deal with very large blobs uploaded with a post/put?

[gabivlj]

Docker's streaming API uses a patch request without a content-length value. It's effectively the same as a post/put, but the digest is not known in advance, so they push the full blob with a single patch.

How do proxies deal with very large blobs uploaded with a post/put?

For proxies or platforms like Workers they buffer the request body. So it would be nice to tell the client (like Docker) to limit the request body to a certain size, as there might be a limit in the server on how many bytes it can buffer.

[sudo-bmitch]

For proxies or platforms like Workers they buffer the request body. So it would be nice to tell the client (like Docker) to limit the request body to a certain size, as there might be a limit in the server on how many bytes it can buffer.

I think that makes my point. The proposed change fixes one of 4 blob upload methods, only 3 of which are defined by OCI, and it's the least used method due to poor support. So I'm unsure how this will solve the underlying issue for proxies.

[gabivlj]

For proxies or platforms like Workers they buffer the request body. So it would be nice to tell the client (like Docker) to limit the request body to a certain size, as there might be a limit in the server on how many bytes it can buffer.

I think that makes my point. The proposed change fixes one of 4 blob upload methods, only 3 of which are defined by OCI, and it's the least used method due to poor support. So I'm unsure how this will solve the underlying issue for proxies.

Please correct me if my understanding here is incorrect, but what I'm proposing is that we do this for all workflows that involve the POST and PATCH and then PUT uploads. I think that involves the chunked upload and the streaming way of Docker as we would also implement in Docker a way for it to respect the limit.

Monolithic uploads and POST PUT only uploads are the only ones that we are not trying to support here.

It would benefit the services that have this limitation as we would atleast have a way to upload layers that surpass this limit.

[sudo-bmitch]

I believe ordered by popularity, there's:

1. POST/PUT
2. Docker's streaming PATCH
3. Chunked POST/PATCH/PUT
4. Single POST

The only method covered by this PR would be 3. OCI hasn't standardized 2 so I don't see how this would apply to that. And if we did define the streaming PATCH, I believe it would have to be excluded from this by its definition.

And 1, being one of the more popular methods, would continue to be an issue for proxies. Given the number of issues I had with registries when I implemented 3, I suspect it's a very small percentage of blob uploads (I'd guess less than 1%). I never tried to implement 4, but I'm guessing a lot of registries using an external blob store would have support issues with that.

Given that close to 100% of blob uploads wouldn't be affected by this change, I'm worried the change to the spec won't have the desired result for your use case.

[gabivlj]

I believe ordered by popularity, there's:

1. POST/PUT

2. Docker's streaming PATCH

3. Chunked POST/PATCH/PUT

4. Single POST

The only method covered by this PR would be 3. OCI hasn't standardized 2 so I don't see how this would apply to that. And if we did define the streaming PATCH, I believe it would have to be excluded from this by its definition.

And 1, being one of the more popular methods, would continue to be an issue for proxies. Given the number of issues I had with registries when I implemented 3, I suspect it's a very small percentage of blob uploads (I'd guess less than 1%). I never tried to implement 4, but I'm guessing a lot of registries using an external blob store would have support issues with that.

Given that close to 100% of blob uploads wouldn't be affected by this change, I'm worried the change to the spec won't have the desired result for your use case.

Interesting. I think this would benefit us as we would have a justification for a Docker PR where we support this functionality for the 2nd use case (see PR mentioned on top). We are also fine supporting only 3rd case outside of Docker. We are not targeting 1st and 4th at the moment.

[sudo-bmitch]

For reference, the streaming patch is defined in the distribution project. In practice, this was implemented as a POST / single PATCH (no length or range header) / PUT. I believe this is done for packing the layer content on the fly, where the output of the tar + compress of the files is not known in advance and they didn't want to write to a temp file or memory.

Supporting more than one PATCH in a stream request is interesting, and looks like it's technically allowed in the distribution project spec, even if it was not previously implemented that way in practice. Since OCI hasn't defined that API yet, I'd be curious to hear from registry operators if they have any concerns with supporting multiple PATCH requests on that API (trying to think ahead for if/when we do add that API).

[gabivlj]

To clarify nomenclature, stream request you mean the way docker pushes a layer to PATCH?

From what I've seen in the code, I think it's doable to have multiple PATCH calls from Docker if we add a limit reader like here:
distribution/distribution#4144

What I want to get here is just have a way for clients/registries to agree on a max request body size for more constrained environments, seems like a good place to put this is what I'm proposing here:
#486

[sudo-bmitch]

Yes, I believe docker was using the API that I linked above. Changing the behavior of the docker client or the distribution registry should be easy. Making sure that such a change doesn't impact any other registry implementations (ecr, gar, acr, quay, harbor, zot, etc) is where I'm looking for feedback from registry operators. It also means OCI, if/when we added that API, would need to support a looser implementation where multiple PATCH requests without a range or length would be allowed.

[gabivlj]

I'm looking to document this and making it an optional field for registries operators to implement, I think it wouldn't be a breaking change for existing registries AFAIK?

[gabivlj]

I also want to add that this seems similar to #391. Who can chime in this thread that could review the change and bring more to the discussion?

Thanks for the comments on this @sudo-bmitch btw, I appreciate the input on this!