Minio does not: implement Region in config.json / Retry when http response is AuthorizationHeaderMalformed

[ghost]

Hi, I'm a developper from Scaleway and we release our object storage. We love use minio client but we can't use it with a signature V4.

Expected behaviour

$> ./mc mb sis/test-minio -debug
mc: <DEBUG> PUT / HTTP/1.1
Host: test-minio.s3.nl-ams.scw.cloud
User-Agent: Minio (darwin; amd64) minio-go/v6.0.8 mc/2018-10-15T15:09:26Z
Content-Length: 150
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20181015/nl-ams/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
Content-Md5: 8p7tByecW7sUlTxMbWAA/A==
X-Amz-Content-Sha256: b86017d61d4ad22a838e75e918325b623687c3e84d631bc2b6435970bf12473d
X-Amz-Date: 20181015T151557Z
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 0
Content-Type: text/html; charset=UTF-8
Date: Mon, 15 Oct 2018 15:15:57 GMT
Location: http://test-minio.s3.nl-ams.scw.cloud/test-minio
X-Amz-Id-2: txb6b044c543ba4564a9a2f-005bc4af2d
X-Amz-Request-Id: txb6b044c543ba4564a9a2f-005bc4af2d

mc: <DEBUG> Response Time:  159.607086ms
Bucket created successfully `sis/test-minio`.

Actual behaviour

$> ./mc ls sis --debug
mc: <DEBUG> GET / HTTP/1.1
Host: s3.nl-ams.scw.cloud
User-Agent: Minio (darwin; amd64) minio-go/5.0.1 mc/2018-03-25T01:22:22Z
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20181015/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20181015T095239Z
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Mon, 15 Oct 2018 09:52:39 GMT
X-Amz-Id-2: txbc616aca981c49e7a5d11-005bc46367
X-Amz-Request-Id: txbc616aca981c49e7a5d11-005bc46367

115
<?xml version='1.0' encoding='UTF-8'?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><RequestId>txbc616aca981c49e7a5d11-005bc46367</RequestId></Error>
0

As you can see minio handle by default us-east-1, the only way to change region on your current version is with mc mb sis/test -region nl-ams but it is impossible with other option and/or set it globally.

Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20181015/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**

Steps to reproduce the behaviour

Use a object storage solution with other endpoint than us-east-1.

mc version

$> ./mc version
Version: 2018-10-15T15:09:26Z
Release-tag: DEVELOPMENT.2018-10-15T15-09-26Z
Commit-id: 7bbc81ffa4c49aabb9e2d62fa0c4d704e066400c

System information

OS:darwin - Arch:amd64 - Lang:go1.11

[ghost]

I made a Pull Request to fix this issue: #2570

[kannappanr]

@quentinselle Thanks for the PR. In s3, region doesn't make much sense when listing buckets. In the case of minio server, it lists the buckets in a minio instance even if the region is set to us-east-1

In the case of AWS S3, I tried to list buckets by connecting to s3.us-west-2.amazonaws.com
Following is the output that I got.

GET
/

host:s3.us-west-2.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20181016T062834Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2018-10-15 23:28:34,352 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20181016T062834Z
20181016/us-east-1/s3/aws4_request
5b1233fff7e7db4779fb0d45dfa06a26b08580392d0eb5881352b476a097a087
2018-10-15 23:28:34,352 - MainThread - botocore.auth - DEBUG - Signature:
be4c74f78bf2de853521bb1aebdf7a8966f076d540c81713807e02d55203c3a0
2018-10-15 23:28:34,352 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://s3.us-west-2.amazonaws.com/, headers={'User-Agent': b'aws-cli/1.11.139 Python/3.6.3 Linux/4.13.0-46-generic botocore/1.12.19', 'X-Amz-Date': b'20181016T062834Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=REDACTED/20181016/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=be4c74f78bf2de853521bb1aebdf7a8966f076d540c81713807e02d55203c3a0'}>
2018-10-15 23:28:34,570 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-request-id': '4451412E55F5BA94', 'x-amz-id-2': 'RPfeqUpKrXa0ZQq/2iVWAmU4sBPzLElLIkPoEOxl93Pax7YjepiCarlf4H0OPLtBb3pu35C0VYo=', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Date': 'Tue, 16 Oct 2018 06:28:33 GMT', 'Connection': 'close', 'Server': 'AmazonS3'}
2018-10-15 23:28:34,570 - MainThread - botocore.parsers - DEBUG - Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region \'us-east-1\' is wrong; expecting \'us-west-2\'</Message><Region>us-west-2</Region><RequestId>4451412E55F5BA94</RequestId><HostId>RPfeqUpKrXa0ZQq/2iVWAmU4sBPzLElLIkPoEOxl93Pax7YjepiCarlf4H0OPLtBb3pu35C0VYo=</HostId></Error>'
2018-10-15 23:28:34,572 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListBuckets: calling handler <botocore.retryhandler.RetryHandler object at 0x7f1ccb3f9c88>
2018-10-15 23:28:34,572 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2018-10-15 23:28:34,572 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListBuckets: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7f1ccb3186a0>>
2018-10-15 23:28:34,573 - MainThread - botocore.utils - DEBUG - S3 client configured for region us-east-1 but the bucket None is in region us-west-2; Please configure the proper region to avoid multiple unnecessary redirects and signing attempts.
2018-10-15 23:28:34,573 - MainThread - botocore.utils - DEBUG - Updating URI from https://s3.us-west-2.amazonaws.com/ to https://s3.us-west-2.amazonaws.com/
2018-10-15 23:28:34,573 - MainThread - botocore.endpoint - DEBUG - Response received to retry, sleeping for 0 seconds
2018-10-15 23:28:34,574 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.ListBuckets: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f1ccb3182e8>>
2018-10-15 23:28:34,574 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.ListBuckets: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7f1ccb3a6eb8>>
2018-10-15 23:28:34,575 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.ListBuckets: calling handler <function set_operation_specific_signer at 0x7f1ccc7106a8>
2018-10-15 23:28:34,575 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2018-10-15 23:28:34,576 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
GET
/

host:s3.us-west-2.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20181016T062834Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2018-10-15 23:28:34,576 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20181016T062834Z
20181016/us-west-2/s3/aws4_request
5b1233fff7e7db4779fb0d45dfa06a26b08580392d0eb5881352b476a097a087
2018-10-15 23:28:34,576 - MainThread - botocore.auth - DEBUG - Signature:
8a0b0fba2cd16e81be78720d6090d332f583b46b4774fb59aa4d2997ded5d774
2018-10-15 23:28:34,577 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://s3.us-west-2.amazonaws.com/, headers={'User-Agent': b'aws-cli/1.11.139 Python/3.6.3 Linux/4.13.0-46-generic botocore/1.12.19', 'X-Amz-Date': b'20181016T062834Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=REDACTED/20181016/us-west-2/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=8a0b0fba2cd16e81be78720d6090d332f583b46b4774fb59aa4d2997ded5d774'}>
2018-10-15 23:28:35,119 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': 'tpy9yGQPfm7l6IgIP9RyS5HQ4QFpTNgZI7GIsWarmFh0omQzwlEeJHyk6CAMB4b++mhyGuHQvsI=', 'x-amz-request-id': 'DAB5CA3C60BAB343', 'Date': 'Tue, 16 Oct 2018 06:28:35 GMT', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Server': 'AmazonS3'}

In the output of the first GET command, it returns a AuthorizationHeaderMalformed saying that the region expected is us-west-2.

If scaleway, can return an error like that instead of signature malformed error, then minio-go SDK code can be modified to use that error to retry the API again with the correct region set.

[ghost]

Hi, thanks for your comment, I update the Scaleway backend (in local env) to deal with the AuthorizationHeaderMalformed response when region is not good.

As it is necessary to update the minio-go to implement the retry with the good endpoint I test my patch with aws-cli (as you did, I put a bad region in the ~/.aws/config file) and much to my surprise I get:

GET
/

host:localhost:6007
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20181016T160353Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2018-10-16 18:03:53,416 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20181016T160353Z
20181016/TestTwo/s3/aws4_request
4a28687c12ed91cae4e02ac882af040a2a7ba724a061a31d2a81d812c42780f1
2018-10-16 18:03:53,416 - MainThread - botocore.auth - DEBUG - Signature:
3bcb46244cbd3144a715e295a6316e826c8551a37d1da0614075c42e34aa7a1b
2018-10-16 18:03:53,416 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=http://localhost:6007/, headers={'User-Agent': b'aws-cli/1.16.8 Python/3.7.0 Darwin/17.7.0 botocore/1.11.8', 'X-Amz-Date': b'20181016T160353Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=**REDACTED**/20181016/TestTwo/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=3bcb46244cbd3144a715e295a6316e826c8551a37d1da0614075c42e34aa7a1b'}>
2018-10-16 18:03:53,424 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': 'tx9c4b0d2f824b4b4ab36b2-005bc60be9', 'Connection': 'Close', 'x-amz-request-id': 'tx9c4b0d2f824b4b4ab36b2-005bc60be9', 'Content-Type': 'application/xml', 'Date': 'Tue, 16 Oct 2018 16:03:53 GMT', 'Transfer-Encoding': 'chunked'}
2018-10-16 18:03:53,424 - MainThread - botocore.parsers - DEBUG - Response body:
b"<?xml version='1.0' encoding='UTF-8'?>\n<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region 'TestTwo' is wrong; expecting 'TestOne'</Message><RequestId>tx9c4b0d2f824b4b4ab36b2-005bc60be9</RequestId><Region>TestOne</Region></Error>"
2018-10-16 18:03:53,424 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListBuckets: calling handler <botocore.retryhandler.RetryHandler object at 0x10c212c50>
2018-10-16 18:03:53,425 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2018-10-16 18:03:53,425 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListBuckets: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x10c212fd0>>
2018-10-16 18:03:53,425 - MainThread - botocore.utils - DEBUG - S3 client configured for region TestTwo but the bucket None is in region TestOne; Please configure the proper region to avoid multiple unnecessary redirects and signing attempts.
2018-10-16 18:03:53,425 - MainThread - botocore.client - DEBUG - Assuming an endpoint for s3, TestOne: https://s3.TestOne.amazonaws.com
2018-10-16 18:03:53,425 - MainThread - botocore.utils - DEBUG - Updating URI from http://localhost:6007/ to http://s3.TestOne.amazonaws.com/
2018-10-16 18:03:53,425 - MainThread - botocore.endpoint - DEBUG - Response received to retry, sleeping for 0 seconds
2018-10-16 18:03:53,425 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.ListBuckets: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x10c2128d0>>
2018-10-16 18:03:53,425 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.ListBuckets: calling handler <function set_operation_specific_signer at 0x10b4a2a60>
2018-10-16 18:03:53,425 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2018-10-16 18:03:53,426 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
GET
/

host:s3.TestOne.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20181016T160353Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2018-10-16 18:03:53,494 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20181016T160353Z
20181016/TestOne/s3/aws4_request
e6b0873e226e9774770ea48c7fe16c1f2b526a012a9426f55e57263a851b01a7
2018-10-16 18:03:53,494 - MainThread - botocore.auth - DEBUG - Signature:
4b0c6f8b71cc34e4b5aa82ff941112a6c77f5561a6e7854c2fba58eea360b8f1
2018-10-16 18:03:53,495 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=http://s3.TestOne.amazonaws.com/, headers={'User-Agent': b'aws-cli/1.16.8 Python/3.7.0 Darwin/17.7.0 botocore/1.11.8', 'X-Amz-Date': b'20181016T160353Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=**REDACTED**/20181016/TestOne/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=4b0c6f8b71cc34e4b5aa82ff941112a6c77f5561a6e7854c2fba58eea360b8f2}>
2018-10-16 18:03:53,501 - MainThread - botocore.endpoint - DEBUG - Exception received when sending HTTP request.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 171, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 56, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socket.py", line 748, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known

-> url=http://s3.TestOne.amazonaws.com/,

Indeed, boto retry with the new Region but not with the endpoint from conf... it retry with a amazonaws.com hardcoded endpoint instead of retry with the precedent request and change it with the new region and the recalculate signature...

Have you got any suggestion for the implementation of this retry ?

[ghost]

This is harcoded here: https://github.com/boto/botocore/blob/develop/botocore/client.py#L349
I made a patch who works.
I prepare a PR for botocore :)

[kannappanr]

Pasting comment from @poornas in #2571 here

@quentinselle, minio-go already retries with region when response code is InvalidRegion. response message for AuthorizationHeaderMalformed needs to be parsed to get the correct region and set as errResp.region in https://github.com/minio/minio-go/blob/master/api-error-response.go#L179 and
https://github.com/minio/minio-go/blob/master/api.go#L630 to use the errResp.Region in case of AuthorizationHeaderMalformed

[Praveenrajmani]

mc works as intended. Regions are bucket level concepts. Please refer to this comment here minio/minio-go#1074 (comment) for more info.

This is not an issue w.r.t mc. It matters on server implementation. The compatible server should honour all the regions in ListBuckets() as minio does.