AWS CLI file uploads using HTTP are failing from 4.0.0: Connection was closed before we received a valid response from endpoint URL

[mathbruyen]

Server fails to map AWS CLI request to on aws s3 cp:

o.s.web.servlet.DispatcherServlet        : Exiting from "ERROR" dispatch, status 400
o.s.web.servlet.DispatcherServlet        : PUT "/backups/README.md", parameters={}
s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to com.adobe.testing.s3mock.ObjectController#putObject(String, ObjectKey, List, String, String, StorageClass, HttpHeaders, InputStream)
o.s.w.s.m.m.a.HttpEntityMethodProcessor  : Found 'Content-Type:application/xml' in response
o.s.w.s.m.m.a.HttpEntityMethodProcessor  : Writing [ErrorResponse[code=UnexpectedContent, message=This request contains unsupported content., resource=n (truncated)...]

The request is logged as:

Making request for OperationModel(name=PutObject) with params: {'url_path': '/README.md', 'query_string': {}, 'method': 'PUT', 'headers': {'Content-Type': 'text/markdown', 'User-Agent': 'aws-cli/2.15.37 Python/3.11.8 Linux/6.1.0-31-amd64 exe/x86_64.debian.12 prompt/off command/s3.cp', 'Content-MD5': 'rLtqOfcZH0Ny51fAf5RNrg==', 'Expect': '100-continue'}, 'body': <s3transfer.utils.ReadFileChunk object at 0x7f0a3bd34f90>, 'auth_path': '/backups/README.md', 'url': 'http://localhost:9098/backups/README.md', 'context': {'client_region': 'eu-west-3', 'client_config': <botocore.config.Config object at 0x7f0a3c1ca410>, 'has_streaming_input': True, 'auth_type': 'v4', 'signing': {'region': 'eu-west-3', 'signing_name': 's3', 'disableDoubleEncoding': True}, 'endpoint_properties': {'authSchemes': [{'disableDoubleEncoding': True, 'name': 'sigv4', 'signingName': 's3', 'signingRegion': 'eu-west-3'}]}, 's3_redirect': {'redirected': False, 'bucket': 'backups', 'params': {'Bucket': 'backups', 'Key': 'README.md', 'Body': <s3transfer.utils.ReadFileChunk object at 0x7f0a3bd34f90>, 'ContentType': 'text/markdown'}}, 'S3Express': {'bucket_name': 'backups'}}}

Git bisect points at commit "Use Tomcat instead of Jetty": a327414

Works with Docker Hub image 3.12.0 and fails with Docker hub image 4.0.0. Replicated locally by building commits with ./mvnw clean package -pl server,docker -am -DskipTests -Dcheckstyle.skip and starting server with docker run --rm -p 9098:9090 -e initialBuckets=backups -e debug=true adobe/s3mock:latest (not changing port mapping and using -p 9090:9090 yields to the same results): server.log.

CLI executed with aws --debug --endpoint-url http://localhost:9098 s3 cp README.md s3://backups/README.md (from the root of this project, hence project's README being uploaded): cli.log

Fails with AWS CLI version 2.15.37 installed from Debian packages and also with version 2.22.10 running in an Alpine Docker container (FROM alpine:3.21.3 + apk add aws-cli).

aws --version
aws-cli/2.15.37 Python/3.11.8 Linux/6.1.0-31-amd64 exe/x86_64.debian.12 prompt/off

Other endpoints work fine:

aws --endpoint http://localhost:9098 s3api list-buckets
{
    "Buckets": [
        {
            "Name": "backups",
            "CreationDate": "2025-04-09T09:13:16.392000+00:00"
        }
    ],
    "Owner": {
        "DisplayName": "s3-mock-file-store",
        "ID": "79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be"
    }
}

Writing files works from curl:

curl -v --request PUT --upload-file ./README.md http://localhost:9098/backups/README.md
> PUT /backups/README.md HTTP/1.1
> Host: localhost:9098
> User-Agent: curl/7.88.1
> Accept: */*
> Content-Length: 42443
> Expect: 100-continue
> 
< HTTP/1.1 100 
* We are completely uploaded and fine
< HTTP/1.1 200 
< Vary: Origin
< Vary: Access-Control-Request-Method
< Vary: Access-Control-Request-Headers
< Last-Modified: Wed, 09 Apr 2025 09:13:22 GMT
< ETag: "acbb6a39f7191f4372e757c07f944dae"
< Content-Length: 0
< Date: Wed, 09 Apr 2025 09:13:22 GMT

[sanjayts-dv]

I'm not sure if it's related but we had our CI integration test builds suddenly failing with the below error when using 4.0.0 docker tag:

upload failed: /some/local/path to s3://some/s3/path Connection was closed before we received a valid response from endpoint URL: "http://s3:9090/some/s3/path.

Downgrading to the previous version made it go away, just FYI.

[afranken]

thanks for reporting this.
We have integration tests with the Kotlin SDK and the Java SDK V1 and V2, which all passed the Jetty to Tomcat migration.

I'll look into the aws-cli errors.

@sanjayts-dv how are you accessing S3Mock? Also through aws-cli?

[mathbruyen]

upload failed: /some/local/path to s3://some/s3/path Connection was closed before we received a valid response from endpoint URL: "http://s3:9090/some/s3/path.

That is the same error message. Updated issue title to be clearer.

[bigboynaruto]

Hi @afranken

I have the same issue with boto3 1.35.40 in Python.

The usage looks like this:

import boto3

session = boto3.session.Session()
client = session.client("s3", endpoint_url="http://localhost:9090")
client.create_bucket(Bucket="my-bucket")
with open("file.txt", "rb") as data:
    client.upload_fileobj(data, "my-bucket", "my-key")

Bucket is successfully getting created, but the last command raises an exception:

botocore.exceptions.ConnectionClosedError: Connection was closed before we received a valid response from endpoint URL: "http://localhost:9090/my-bucket/my-key".

Similar to others, it works on 3.12.0.

[sanjayts-dv]

@afranken I'm uploading objects to the mock S3 instance as part of a one-off docker-compose task. And yes, I'm using the aws-cli for uploading these objects. The task looks like:

  init-s3:
    image: amazon/aws-cli:latest
    env_file: .env.integration-tests
    volumes:
      - ./integration/s3_init_data:/s3_init_data
    entrypoint: ["/bin/sh", "-c"]
    command:
      - |
        aws s3 cp --endpoint-url=$${S3_ENDPOINT} /s3_init_data/somefile ://my_custom_bucket/somefile
        // and so on

[afranken]

I found the cause: for as of yet unknown reasons, the Tomcat HTTP connector does not respond correctly to the "100 continue" challenge of the boto3 client (it's used in aws cli as well).

Example output from aws cli with --debug and S3Mock 3.12.0:

2025-04-10 01:08:14,598 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.PutObject: calling handler <bound method UserAgentString.rebuild_and_replace_user_agent_handler of <botocore.useragent.UserAgentString object at 0x104d1f410>>
2025-04-10 01:08:14,598 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=http://localhost:9090/x/y.txt, headers={'Content-Type': b'text/plain', 'x-amz-sdk-checksum-algorithm': b'CRC32', 'User-Agent': b'aws-cli/2.25.13 md/awscrt#0.25.4 ua/2.1 os/macos#24.4.0 md/arch#arm64 lang/python#3.12.10 md/pyimpl#CPython m/N cfg/retry-mode#standard md/installer#source md/prompt#off md/command#s3api.put-object', 'Expect': b'100-continue', 'x-amz-checksum-crc32': b'NGSOug==', 'X-Amz-Date': b'20250409T230814Z', 'X-Amz-Content-SHA256': b'65dbc69741c062892a2842cc5d6099bbf02eefd4628d352d446c6c37cb0967da', 'Authorization': b'AWS4-HMAC-SHA256 Credential=NONE/20250409/us-east-1/s3/aws4_request, SignedHeaders=content-type;host;x-amz-checksum-crc32;x-amz-content-sha256;x-amz-date;x-amz-sdk-checksum-algorithm, Signature=37285ce19b27e82759690e90862796326813642064fbc021088488d477bf2f1e', 'Content-Length': '42443'}>
2025-04-10 01:08:14,598 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): localhost:9090
2025-04-10 01:08:14,599 - MainThread - botocore.awsrequest - DEBUG - Waiting for 100 Continue response.
2025-04-10 01:08:14,613 - MainThread - botocore.awsrequest - DEBUG - 100 Continue response seen, now sending request body.
2025-04-10 01:08:14,675 - MainThread - urllib3.connectionpool - DEBUG - http://localhost:9090 "PUT /x/y.txt HTTP/1.1" 200 0
2025-04-10 01:08:14,675 - MainThread - botocore.hooks - DEBUG - Event before-parse.s3.PutObject: calling handler <function _handle_200_error at 0x1029d4180>
2025-04-10 01:08:14,675 - MainThread - botocore.hooks - DEBUG - Event before-parse.s3.PutObject: calling handler <function handle_expires_header at 0x1029bff60>
2025-04-10 01:08:14,675 - MainThread - botocore.parsers - DEBUG - Response headers: {'Server': 'Jetty(12.0.12)', 'Date': 'Wed, 09 Apr 2025 23:08:14 GMT', 'Vary': 'Origin, Access-Control-Request-Method, Access-Control-Request-Headers', 'x-amz-checksum-crc32': 'NGSOug==', 'Last-Modified': 'Wed, 09 Apr 2025 23:08:14 GMT', 'ETag': '"44a97f35fa58f36daa1cb42955a2bea0"', 'Content-Length': '0'}
2025-04-10 01:08:14,675 - MainThread - botocore.parsers - DEBUG - Response body:
b''

Same with S3Mock 4.0.0:

2025-04-10 01:41:22,320 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.PutObject: calling handler <bound method UserAgentString.rebuild_and_replace_user_agent_handler of <botocore.useragent.UserAgentString object at 0x1052779b0>>
2025-04-10 01:41:22,320 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=http://localhost:9090/x/y.txt, headers={'Content-Type': b'text/plain', 'x-amz-sdk-checksum-algorithm': b'CRC32', 'User-Agent': b'aws-cli/2.25.13 md/awscrt#0.25.4 ua/2.1 os/macos#24.4.0 md/arch#arm64 lang/python#3.12.10 md/pyimpl#CPython m/N cfg/retry-mode#standard md/installer#source md/prompt#off md/command#s3api.put-object', 'Expect': b'100-continue', 'x-amz-checksum-crc32': b'NGSOug==', 'X-Amz-Date': b'20250409T234122Z', 'X-Amz-Content-SHA256': b'65dbc69741c062892a2842cc5d6099bbf02eefd4628d352d446c6c37cb0967da', 'Authorization': b'AWS4-HMAC-SHA256 Credential=NONE/20250409/us-east-1/s3/aws4_request, SignedHeaders=content-type;host;x-amz-checksum-crc32;x-amz-content-sha256;x-amz-date;x-amz-sdk-checksum-algorithm, Signature=e6a74e4aafab3b1c03e282cf2064d65dbb22af4396cb34460f69439d180f36d3', 'Content-Length': '42443'}>
2025-04-10 01:41:22,320 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): localhost:9090
2025-04-10 01:41:22,321 - MainThread - botocore.awsrequest - DEBUG - Waiting for 100 Continue response.
2025-04-10 01:41:22,324 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.PutObject: calling handler <function _update_status_code at 0x101d1c2c0>
2025-04-10 01:41:22,324 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.PutObject: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x105277c80>>
2025-04-10 01:41:22,324 - MainThread - botocore.retries.standard - DEBUG - Retry needed, retrying request after delay of: 0.2322557477907542

A simple workaround is to use HTTPS, Tomcat is correctly responding and all requests go through successfully.

Example for aws cli:

aws s3api put-object \
    --bucket x \
    --key y.txt \
    --content-type text/plain \
    --endpoint https://localhost:9191 \
    --body README.md \
    --no-verify-ssl \
    --debug

[sanjayts-dv]

Thank you for the clarification @afranken. Just to understand -- is this the recommendation (using https) going forward or are there any plans to make the new major release work with http endpoints like it used to?

[BenRichey3]

Experiencing this issue as well with python 3.11.11 and boto3==1.37.31. Calls to Object.put are failing. Hoping for a fix soon!