AWS SDK upload fails with Azure backend

[ritazh]

Using this sample code here:
https://github.com/aws/aws-sdk-java/blob/master/src/samples/AmazonS3/S3Sample.java#L119

Error:
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1398): Input is expected to be encoded in multiple of 2 bytes but found: 17

File was uploaded successfully, but ends in exception.

Same test runs successfully against AWS S3.

I see this PR added etag support for AzureBlob. https://github.com/jclouds/jclouds/pull/716/files

[ritazh]

Adding more info, in com.amazonaws.util, Base16Codec is expecting an even length.

public byte[] decode(byte[] src, final int length)
    {
        if (length % 2 != 0) {
            throw new IllegalArgumentException(
                "Input is expected to be encoded in multiple of 2 bytes but found: "
                + length
            );
        }
        final byte[] dest = new byte[length / 2];

[kahing]

s3 expects a regular PUT to return etags as the md5sum of the blob encoded in base16, and the AmazonS3Client is probably verifying the md5sum from the response. Maybe S3Proxy can return Content-MD5 as etags instead?

[ritazh]

Yes I think that would work.

[kahing]

@andrewgaul pointed out to me that munging with etags will cause problems with conditional GETs. And the option in the sdk to disable the check is not implemented correctly: https://github.com/aws/aws-sdk-java/blob/bb9e0be441afb5658f0edbbd19bcf27e2fac4105/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/internal/ServiceUtils.java#L452

[ritazh]

I'm trying to figure out why jcloud returns an ETag of 17 bytes for Azure blob. Any suggestions?

[kahing]

According to https://github.com/jclouds/jclouds/pull/716/files#diff-e724f00ae88c3a1a9ec1ddbe252a3bccL10 Azure etags looks like 0x8CACB9BD7C1EEEC which is 17 bytes.

[gaul]

Azure returns a non-MD5 ETag. We should not change the meaning of this since ETag are used for conditional GETs. We could elide non-MD5 ETag but I do not believe that we should recalculate the MD5 in-flight to fake the ETag. I opened a pull request against aws-sdk-java to control their behavior.

[gaul]

After offline discussion with @kahing, we believe that S3Proxy could mimic S3 more closely when using the Azure backend. All of the relevant Azure operations, e.g., put blob, get blob, head blob, list blob, return both Content-MD5 and ETag. Thus we could map the former to the latter for all operations except for conditional GETs where we must issue our own HEAD request before the conditional GET. Note that some of these require specifying a recent Azure API version which jclouds already does.

@ritazh Would you like to try implementing this? Note that we want to use the server values and not calculate these on the proxy side.

[ritazh]

Sure thing. Let me take a closer look this weekend and get back to you.