Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for signing chunked requests #20

Open
dblock opened this issue Jun 27, 2022 · 0 comments
Open

Add support for signing chunked requests #20

dblock opened this issue Jun 27, 2022 · 0 comments
Labels
bug Something isn't working enhancement New feature or request help wanted Extra attention is needed

Comments

@dblock
Copy link
Collaborator

dblock commented Jun 27, 2022

It looks like none of the implementations of the signing interceptor I was able to find supports signing chunked requests with Sigv4. Documentation states that we need a few things:

  1. Specify the Transfer-Encoding HTTP header. If you include the Transfer-Encoding header and specify any value other than identity, you must omit the Content-Length header. This is easy and what happens when entity#setChunked(false).
  2. For all requests, you must include the x-amz-decoded-content-length header, specifying the size of the object in bytes. This is easy to do, but doesn't actually seem to be necessary for non-chunked requests that come with a Content-Length.
  3. Each chunk signature calculation includes the signature of the previous chunk. To begin, you create a seed signature using only the headers. You use the seed signature in the signature calculation of the first chunk. For each subsequent chunk, you create a chunk signature that includes the signature of the previous chunk. I wasn't able to successfully implement this.

Repro in https://github.com/dblock/aws-request-signing-apache-interceptor/tree/repro-20

Long story:

We started from a bug, opensearch-project/OpenSearch#3640, then @jiten1551 reproduced it in https://github.com/jiten1551/Java-Client/tree/master. I forked that to https://github.com/dblock/opensearch-java-client-demo and added code to attempt supporting compressed requests, which seemed to work for a moment. I PRed kumjiten/Java-Client#1, but the next day the same code didn't work, which was also verified by @pgtgrly. I reached out to the AWS SDK team, and followed their recommendation to upgrade to AWS SDK 2.0 with the same results. Then I found this fork. With #31 it now supports Sigv4 signing GET and POST requests, but not chunked requests. We've also added a client setChunkedEnabled setting to explicitly control chunked transfer encoding in opensearch-project/OpenSearch#3864

@dblock dblock changed the title Add support for gzip compressed, chunked requests Add support for chunked requests Jul 5, 2022
dblock added a commit to dblock/aws-request-signing-apache-interceptor that referenced this issue Jul 5, 2022
Signed-off-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>
@dblock dblock changed the title Add support for chunked requests Add support for signing chunked requests Jul 5, 2022
@dblock dblock added bug Something isn't working enhancement New feature or request help wanted Extra attention is needed labels Jul 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

1 participant