Signature mismatch with s3 path style requests for signature v4. #675
Description
$ s3cmd -c .s3cfg-creds --debug ls s3://miniocloud --region us-east-1
DEBUG: get_hostname(miniocloud): s3.amazonaws.com
DEBUG: canonical_headers = host:s3.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20151209T234532Z
DEBUG: Canonical Request:
GET
/
delimiter=%2F
host:s3.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20151209T234532Z
host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
----------------------
DEBUG: signature-v4 headers: {'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': 'AWS4-HMAC-SHA256 Credential=AKIAJVA5BMMU2RHO6IOQ/20151209/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=ea1e3faf470b17550c97ef0dde53bebaf72ae2a4653c1fa50b916ddd712b8429', 'x-amz-date': '20151209T234532Z'}
DEBUG: Processing request, please wait...
DEBUG: get_hostname(miniocloud): s3.amazonaws.com
DEBUG: ConnMan.get(): creating new connection: https://s3.amazonaws.com
DEBUG: Using ca_certs_file None
DEBUG: non-proxied HTTPSConnection(s3.amazonaws.com)
DEBUG: format_uri(): /miniocloud/?delimiter=/
DEBUG: Sending request method_string='GET', uri='/miniocloud/?delimiter=/', headers={'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': 'AWS4-HMAC-SHA256 Credential=AKIAJVA5BMMU2RHO6IOQ/20151209/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=ea1e3faf470b17550c97ef0dde53bebaf72ae2a4653c1fa50b916ddd712b8429', 'x-amz-date': '20151209T234532Z'}, body=(0 bytes)
DEBUG: Response: {'status': 403, 'headers': {'x-amz-bucket-region': 'us-east-1', 'x-amz-id-2': 'hpGlzSi4Ph1HvcX48CIyQerCvvPhcVeQNq9tJzzAWlZ4ZErRXdXmL3yfiGw+08hoeEcjBpJdVDA=', 'server': 'AmazonS3', 'transfer-encoding': 'chunked', 'x-amz-request-id': 'DD308505E5A4E04A', 'date': 'Wed, 09 Dec 2015 23:45:32 GMT', 'content-type': 'application/xml'}, 'reason': 'Forbidden', 'data': '<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>AKIAJVA5BMMU2RHO6IOQ</AWSAccessKeyId><StringToSign>AWS4-HMAC-SHA256\n20151209T234532Z\n20151209/us-east-1/s3/aws4_request\n59d0c5a9e51be0cb5d9f15ac59d5332fde6a231fb804b4e149cbb2c8e7283473</StringToSign><SignatureProvided>ea1e3faf470b17550c97ef0dde53bebaf72ae2a4653c1fa50b916ddd712b8429</SignatureProvided><StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 31 35 31 32 30 39 54 32 33 34 35 33 32 5a 0a 32 30 31 35 31 32 30 39 2f 75 73 2d 65 61 73 74 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 35 39 64 30 63 35 61 39 65 35 31 62 65 30 63 62 35 64 39 66 31 35 61 63 35 39 64 35 33 33 32 66 64 65 36 61 32 33 31 66 62 38 30 34 62 34 65 31 34 39 63 62 62 32 63 38 65 37 32 38 33 34 37 33</StringToSignBytes><CanonicalRequest>GET\n/miniocloud/\ndelimiter=%2F\nhost:s3.amazonaws.com\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20151209T234532Z\n\nhost;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855</CanonicalRequest><CanonicalRequestBytes>47 45 54 0a 2f 6d 69 6e 69 6f 63 6c 6f 75 64 2f 0a 64 65 6c 69 6d 69 74 65 72 3d 25 32 46 0a 68 6f 73 74 3a 73 33 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 31 35 31 32 30 39 54 32 33 34 35 33 32 5a 0a 0a 68 6f 73 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 0a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35</CanonicalRequestBytes><RequestId>DD308505E5A4E04A</RequestId><HostId>hpGlzSi4Ph1HvcX48CIyQerCvvPhcVeQNq9tJzzAWlZ4ZErRXdXmL3yfiGw+08hoeEcjBpJdVDA=</HostId></Error>'}
DEBUG: ConnMan.put(): connection put back to pool (https://s3.amazonaws.com#1)
DEBUG: S3Error: 403 (Forbidden)
DEBUG: HttpHeader: x-amz-bucket-region: us-east-1
DEBUG: HttpHeader: x-amz-id-2: hpGlzSi4Ph1HvcX48CIyQerCvvPhcVeQNq9tJzzAWlZ4ZErRXdXmL3yfiGw+08hoeEcjBpJdVDA=
DEBUG: HttpHeader: server: AmazonS3
DEBUG: HttpHeader: transfer-encoding: chunked
DEBUG: HttpHeader: x-amz-request-id: DD308505E5A4E04A
DEBUG: HttpHeader: date: Wed, 09 Dec 2015 23:45:32 GMT
DEBUG: HttpHeader: content-type: application/xml
DEBUG: ErrorXML: Code: 'SignatureDoesNotMatch'
DEBUG: ErrorXML: Message: 'The request signature we calculated does not match the signature you provided. Check your key and signing method.'
DEBUG: ErrorXML: AWSAccessKeyId: 'AKIAJVA5BMMU2RHO6IOQ'
DEBUG: ErrorXML: StringToSign: 'AWS4-HMAC-SHA256\n20151209T234532Z\n20151209/us-east-1/s3/aws4_request\n59d0c5a9e51be0cb5d9f15ac59d5332fde6a231fb804b4e149cbb2c8e7283473'
DEBUG: ErrorXML: SignatureProvided: 'ea1e3faf470b17550c97ef0dde53bebaf72ae2a4653c1fa50b916ddd712b8429'
DEBUG: ErrorXML: StringToSignBytes: '41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 31 35 31 32 30 39 54 32 33 34 35 33 32 5a 0a 32 30 31 35 31 32 30 39 2f 75 73 2d 65 61 73 74 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 35 39 64 30 63 35 61 39 65 35 31 62 65 30 63 62 35 64 39 66 31 35 61 63 35 39 64 35 33 33 32 66 64 65 36 61 32 33 31 66 62 38 30 34 62 34 65 31 34 39 63 62 62 32 63 38 65 37 32 38 33 34 37 33'
DEBUG: ErrorXML: CanonicalRequest: 'GET\n/miniocloud/\ndelimiter=%2F\nhost:s3.amazonaws.com\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20151209T234532Z\n\nhost;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'
DEBUG: ErrorXML: CanonicalRequestBytes: '47 45 54 0a 2f 6d 69 6e 69 6f 63 6c 6f 75 64 2f 0a 64 65 6c 69 6d 69 74 65 72 3d 25 32 46 0a 68 6f 73 74 3a 73 33 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 31 35 31 32 30 39 54 32 33 34 35 33 32 5a 0a 0a 68 6f 73 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 0a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35'
DEBUG: ErrorXML: RequestId: 'DD308505E5A4E04A'
DEBUG: ErrorXML: HostId: 'hpGlzSi4Ph1HvcX48CIyQerCvvPhcVeQNq9tJzzAWlZ4ZErRXdXmL3yfiGw+08hoeEcjBpJdVDA='
ERROR: S3 error: 403 (SignatureDoesNotMatch): The request signature we calculated does not match the signature you provided. Check your key and signing method.
The bug is in canonical request. which needs to keep a copy of the path not just "/" in case of path style requests.