Skip to content

out_s3: current_buffer_size underflows after deleting buffer files recovered from a previous run — every new chunk is then rejected with "Buffer is full" #12270

Description

@univbyn-blip

Describe the bug

The s3 output tracks store_dir disk usage in ctx->current_buffer_size
(a size_t). The accounting is asymmetric across restarts:

  • buffer files left behind by a previous run (e.g. after a crash or a
    pod restart with pending uploads) are adopted at startup and uploaded,
    but their sizes are never added to current_buffer_size;
  • when such a file is deleted after upload, s3_store_file_delete()
    subtracts its size unconditionally:
/* plugins/out_s3/s3_store.c */
int s3_store_file_delete(struct flb_s3 *ctx, struct s3_file *s3_file)
{
    ...
    ctx->current_buffer_size -= s3_file->size;

Subtracting more than was ever added wraps the unsigned counter around
to ~2^64. From that point on, the store_dir_limit_size check in
s3_store_buffer_put() considers the buffer permanently full and
rejects every new chunk:

[error] [output:s3:out_gwlog_gcs] Buffer is full: current_buffer_size=18446744073709314048, new_data=4931, store_dir_limit_size=20000000000 bytes
[ warn] [output:s3:out_gwlog_gcs] Could not buffer chunk. Data order preservation will be compromised
[error] [engine] chunk '1-1786521063.962669295.flb' cannot be retried: task_id=1, input=re_gwlog > output=out_gwlog_gcs

18446744073709314048 == 2^64 - 237568, i.e. the counter is at
"-237568 bytes" — exactly the size of the leftover files that were
uploaded and deleted. The counter keeps drifting further "negative" as
more recovered files are deleted, and it can never recover, so the
output effectively stops accepting data until the process is restarted
(and the same thing can happen again on the next restart). This results
in permanent data loss ("chunk cannot be retried").

Note that another code path already guards against exactly this: the
quarantine path clamps the subtraction at zero. s3_store_file_delete()
is missing the same guard.

To Reproduce

  1. Configure an s3 output with store_dir on a persistent path (e.g. a
    hostPath volume in Kubernetes) and store_dir_limit_size set (e.g.
    20G).
  2. Let Fluent Bit buffer some data, then kill it before the upload
    completes, leaving files in store_dir.
  3. Start Fluent Bit again. It adopts and uploads the leftover files.
  4. As soon as the recovered files are uploaded and deleted,
    current_buffer_size wraps to ~2^64 and every subsequent chunk is
    rejected with "Buffer is full" even though the disk is empty.

Expected behavior

Deleting a buffer file must not wrap the usage counter. Either the
subtraction should be clamped at zero (defensive fix, same pattern as
the quarantine path), or files adopted at startup should be accounted
into current_buffer_size (root-cause fix), or both.

Suggested fix (minimal, defensive)

    if (ctx->current_buffer_size >= s3_file->size) {
        ctx->current_buffer_size -= s3_file->size;
    }
    else {
        ctx->current_buffer_size = 0;
    }

I'm happy to submit a PR with this change.

Workaround

Unset store_dir_limit_size (default 0 = unlimited): the broken check
only runs when a limit is configured.

Your Environment

  • Version used: 5.0.9 (the unguarded subtraction is also present on current master)
  • Configuration: s3 output → GCS S3-compatible endpoint, use_put_object On, store_dir on a Kubernetes hostPath volume, store_dir_limit_size 20G
  • Environment: GKE (containerd), DaemonSet tailing container logs

PR Title

out_s3: guard current_buffer_size subtraction against underflow

PR Body

Fixes #<이슈번호>

s3_store_file_delete() subtracted s3_file->size from
ctx->current_buffer_size unconditionally. Buffer files recovered from
a previous run are not accounted into current_buffer_size at startup,
so deleting them after upload wraps the unsigned counter around to
~2^64. The store_dir_limit_size check then treats the buffer as
permanently full and rejects every new chunk ("Buffer is full",
"chunk cannot be retried"), causing data loss until restart.

Clamp the subtraction at zero, using the same pattern already applied
on the quarantine accounting path in s3.c.

Observed in production on 5.0.9 (Kubernetes DaemonSet, store_dir on a
hostPath volume surviving pod restarts):

[error] Buffer is full: current_buffer_size=18446744073709314048, new_data=4931, store_dir_limit_size=20000000000 bytes

18446744073709314048 == 2^64 - 237568 — the counter went "negative"
by exactly the size of the recovered-and-deleted files.


Testing

  • Example configuration file for the change
  • Debug log output from testing the change

Documentation

  • N/A — bug fix, no user-facing configuration change

Backporting

  • Backport to latest stable release.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions