s3 output for GCS repeat uploading same object

[bananatoast]

Bug Report

Describe the bug
I am testing s3 output plugin with GCS using HMAC key authentication. First lines of log could be uploaded to GCS bucket, but then fluent-bit keeps putting same contents as another object. Then, no more logs will be uploaded.

To Reproduce

• Steps to reproduce the problem:
  Here is my fluent-bit.conf

[SERVICE]
  Flush         5
  Daemon        off
  Log_Level     debug
[INPUT]
  Name          tail
  Path          /tmp/app.log
[OUTPUT]
  Name          s3
  Match         *
  endpoint      storage.googleapis.com
  bucket        my-bucket
  region        asia-northeast1
  store_dir     /tmp/buffer
  s3_key_format /%Y/%m/%d/%H
  tls.debug     4

And run with this command

AWS_ACCESS_KEY_ID=<my_aws_key_id> AWS_SECRET_ACCESS_KEY=<my_secret_key> fluent-bit -c fluent-bit.conf

Here is log messages at errors:

...
[2024/02/26 16:08:45] [error] [http_client] broken connection to storage.googleapis.com:443 ?
[2024/02/26 16:08:45] [debug] [aws_client] storage.googleapis.com: http_do=-1, HTTP Status: 200
[2024/02/26 16:08:45] [debug] [upstream] KA connection #64 to storage.googleapis.com:443 is now available
[2024/02/26 16:08:45] [debug] [aws_client] auto-retrying
[2024/02/26 16:08:45] [debug] [upstream] KA connection #64 to storage.googleapis.com:443 has been assigned (recycled)
[2024/02/26 16:08:45] [debug] [http_client] not using http_proxy for header
[2024/02/26 16:08:45] [debug] [aws_credentials] Requesting credentials from the env provider..
[2024/02/26 16:08:45] [debug] [aws_client] storage.googleapis.com: http_do=-1, HTTP Status: 0
[2024/02/26 16:08:45] [error] [output:s3:s3.1] PutObject request failed
[2024/02/26 16:08:45] [error] [output:s3:s3.1] Failed to send locally buffered data left over from previous executions; will retry. Buffer=/tmp/buffer/my-bucket
[2024/02/26 16:08:45] [debug] [retry] new retry created for task_id=0 attempts=1
[2024/02/26 16:08:45] [ warn] [engine] failed to flush chunk '67996-1708931081.288602000.flb', retry in 8 seconds: task_id=0, input=tail.0 > [2024/02/26 16:08:45] [debug] [out flush] cb_destroy coro_id=0
output=s3.1 (out_id=1)
[2024/02/26 16:08:53] [debug] [output:s3:s3.1] task_id=0 assigned to thread #0
[2024/02/26 16:08:53] [ info] [output:s3:s3.1] Sending locally buffered data from previous executions to S3; buffer=/tmp/buffer/my-bucket
[2024/02/26 16:08:53] [debug] [upstream] KA connection #64 to storage.googleapis.com:443 is connected
[2024/02/26 16:08:53] [debug] [http_client] not using http_proxy for header
[2024/02/26 16:08:53] [debug] [aws_credentials] Requesting credentials from the env provider..

We found the following objects in GCS, and repeat uploading same objects every 4 minutes.

Name	Size	Created
05-objectBLNYuRB7	748B	Feb 26, 2024, 4:04:45 PM
05-objectidG5ufxX	748B	Feb 26, 2024, 4:08:53 PM
...	748B	...

Expected behavior
When I configure bucket, region and AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY pair with AWS one, logs are uploaded to AWS s3 as usual and not repeating. The same behavior is expected with GCS.

Your Environment

• Version used: 2.2.2
• Environment: tested on kubernets 1.20 (sidecar) and standalone on MacOS 13.6
• Filters and plugins: out_s3

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.

[univbyn-blip]

Root cause identified in #12267 — Fluent Bit misparses GCS's x-goog-stored-content-length response header as Content-Length and waits for a body that never arrives, so every successful upload is judged as failed and retried.