POST uploads

[gaul]

S3Proxy does not currently support POST uploads, triggering many s3-tests failures:

ERROR: s3tests.functional.test_s3.test_post_object_anonymous_request
ERROR: s3tests.functional.test_s3.test_post_object_authenticated_request_bad_access_key
ERROR: s3tests.functional.test_s3.test_post_object_set_success_code
ERROR: s3tests.functional.test_s3.test_post_object_set_invalid_success_code
FAIL: s3tests.functional.test_s3.test_post_object_authenticated_request
FAIL: s3tests.functional.test_s3.test_post_object_upload_larger_than_chunk
FAIL: s3tests.functional.test_s3.test_post_object_set_key_from_filename
FAIL: s3tests.functional.test_s3.test_post_object_ignored_header
FAIL: s3tests.functional.test_s3.test_post_object_case_insensitive_condition_fields
FAIL: s3tests.functional.test_s3.test_post_object_escaped_field_values
FAIL: s3tests.functional.test_s3.test_post_object_success_redirect_action
FAIL: s3tests.functional.test_s3.test_post_object_invalid_date_format
FAIL: s3tests.functional.test_s3.test_post_object_no_key_specified
FAIL: s3tests.functional.test_s3.test_post_object_missing_signature
FAIL: s3tests.functional.test_s3.test_post_object_user_specified_header
FAIL: s3tests.functional.test_s3.test_post_object_condition_is_case_sensitive
FAIL: s3tests.functional.test_s3.test_post_object_expires_is_case_sensitive
FAIL: s3tests.functional.test_s3.test_post_object_missing_expires_condition
FAIL: s3tests.functional.test_s3.test_post_object_missing_conditions_list
FAIL: s3tests.functional.test_s3.test_post_object_upload_size_limit_exceeded
FAIL: s3tests.functional.test_s3.test_post_object_missing_content_length_argument
FAIL: s3tests.functional.test_s3.test_post_object_invalid_content_length_argument
FAIL: s3tests.functional.test_s3.test_post_object_upload_size_below_minimum

[gaul]

@auguster I implemented basic support for POST uploads -- can you test to see if it resolves your issue? Note that S3Proxy still lacks support for policies.

[auguster]

Thanks you for the news !
Unfortunatly I'll be away for the whole coming week, so I might not be able to test that before the week after...

[ck86]

Is it planned to be merged?

[gaul]

S3Proxy has partial POST upload support although there are a lot of extra bits. Do you need one feature in particular?

[ck86]

We currently migrating from S3 to Azure Blob Storage and to upload files from the client, our API creates a PostObject, which seems to work with the proxy (at least I get similar "form fields"). But when I submit the form I get a "Not Implemented" error.

[gaul]

Could you share some example source code? Also maybe dump logs via -DLOG_LEVEL=trace although it might not capture enough information to help.

[ck86]

How I use the php aws sdk:

$options = [
    ['acl' => 'public-read'],
    ['bucket' => $bucket],
    ['starts-with', '$key', $key],
    ['content-length-range', 0, 1024 * 1024 * 1024],
];

/** @var \Aws\S3\S3Client $s3 */
$s3 = app('s3');
$postObject = new PostObjectV4($s3, $bucket, $inputs, $options);
$attributes = $postObject->getFormAttributes();
$fields = $postObject->getFormInputs();

this is an example response:

{
    "attributes": {
        "action": "http://localhost:8080",
        "method": "POST",
        "enctype": "multipart/form-data"
    },
    "fields": {
        "acl": "public-read",
        "key": "filename.ext",
        "X-Amz-Credential": "*******",
        "X-Amz-Algorithm": "AWS4-HMAC-SHA256",
        "X-Amz-Date": "20180418T063726Z",
        "Policy": "**************",
        "X-Amz-Signature": "************"
    }
}

The above mentions field are used to submit the form with a file called filename.ext.

This is the exact error message:

<?xml version='1.0' encoding='UTF-8'?>
<Error>
    <Code>NotImplemented</Code>
    <Message>A header you provided implies functionality that is not implemented.</Message>
    <RequestId>4442587FB7D0A2F9</RequestId>
</Error>

Unfortunately I can't or I don't know how to set the DLOG_LEVEL. I am using docker and if I am adding it after ENTRYPOINT ./target/s3proxy --properties ./s3proxy.conf the proxy will not start because of the unknown option.

[ck86]

any updates? :/

[gaul]

The Docker image should take a LOG_LEVEL environment variable. I really need a self-contained test case to make progress on this; otherwise I would have to implement the entire POST handling functionality which I would like to avoid for now.

[simonbowen]

@ck86 I'm facing this issue at the moment, did you manage to find a work around?

[ck86]

@simonbowen unfortunately not. I switched to another proxy and used the signed URLs to upload a file.

[simonbowen]

@ck86 Can you tell me the name of the other proxy you used? I'm looking at using LocalStack instead at the moment.

[ck86]

minio