Describe the bug
The s3 output tracks store_dir disk usage in ctx->current_buffer_size
(a size_t). The accounting is asymmetric across restarts:
- buffer files left behind by a previous run (e.g. after a crash or a
pod restart with pending uploads) are adopted at startup and uploaded,
but their sizes are never added to current_buffer_size;
- when such a file is deleted after upload,
s3_store_file_delete()
subtracts its size unconditionally:
/* plugins/out_s3/s3_store.c */
int s3_store_file_delete(struct flb_s3 *ctx, struct s3_file *s3_file)
{
...
ctx->current_buffer_size -= s3_file->size;
Subtracting more than was ever added wraps the unsigned counter around
to ~2^64. From that point on, the store_dir_limit_size check in
s3_store_buffer_put() considers the buffer permanently full and
rejects every new chunk:
[error] [output:s3:out_gwlog_gcs] Buffer is full: current_buffer_size=18446744073709314048, new_data=4931, store_dir_limit_size=20000000000 bytes
[ warn] [output:s3:out_gwlog_gcs] Could not buffer chunk. Data order preservation will be compromised
[error] [engine] chunk '1-1786521063.962669295.flb' cannot be retried: task_id=1, input=re_gwlog > output=out_gwlog_gcs
18446744073709314048 == 2^64 - 237568, i.e. the counter is at
"-237568 bytes" — exactly the size of the leftover files that were
uploaded and deleted. The counter keeps drifting further "negative" as
more recovered files are deleted, and it can never recover, so the
output effectively stops accepting data until the process is restarted
(and the same thing can happen again on the next restart). This results
in permanent data loss ("chunk cannot be retried").
Note that another code path already guards against exactly this: the
quarantine path clamps the subtraction at zero. s3_store_file_delete()
is missing the same guard.
To Reproduce
- Configure an s3 output with
store_dir on a persistent path (e.g. a
hostPath volume in Kubernetes) and store_dir_limit_size set (e.g.
20G).
- Let Fluent Bit buffer some data, then kill it before the upload
completes, leaving files in store_dir.
- Start Fluent Bit again. It adopts and uploads the leftover files.
- As soon as the recovered files are uploaded and deleted,
current_buffer_size wraps to ~2^64 and every subsequent chunk is
rejected with "Buffer is full" even though the disk is empty.
Expected behavior
Deleting a buffer file must not wrap the usage counter. Either the
subtraction should be clamped at zero (defensive fix, same pattern as
the quarantine path), or files adopted at startup should be accounted
into current_buffer_size (root-cause fix), or both.
Suggested fix (minimal, defensive)
if (ctx->current_buffer_size >= s3_file->size) {
ctx->current_buffer_size -= s3_file->size;
}
else {
ctx->current_buffer_size = 0;
}
I'm happy to submit a PR with this change.
Workaround
Unset store_dir_limit_size (default 0 = unlimited): the broken check
only runs when a limit is configured.
Your Environment
- Version used: 5.0.9 (the unguarded subtraction is also present on current master)
- Configuration: s3 output → GCS S3-compatible endpoint,
use_put_object On, store_dir on a Kubernetes hostPath volume, store_dir_limit_size 20G
- Environment: GKE (containerd), DaemonSet tailing container logs
PR Title
out_s3: guard current_buffer_size subtraction against underflow
PR Body
Fixes #<이슈번호>
s3_store_file_delete() subtracted s3_file->size from
ctx->current_buffer_size unconditionally. Buffer files recovered from
a previous run are not accounted into current_buffer_size at startup,
so deleting them after upload wraps the unsigned counter around to
~2^64. The store_dir_limit_size check then treats the buffer as
permanently full and rejects every new chunk ("Buffer is full",
"chunk cannot be retried"), causing data loss until restart.
Clamp the subtraction at zero, using the same pattern already applied
on the quarantine accounting path in s3.c.
Observed in production on 5.0.9 (Kubernetes DaemonSet, store_dir on a
hostPath volume surviving pod restarts):
[error] Buffer is full: current_buffer_size=18446744073709314048, new_data=4931, store_dir_limit_size=20000000000 bytes
18446744073709314048 == 2^64 - 237568 — the counter went "negative"
by exactly the size of the recovered-and-deleted files.
Testing
Documentation
Backporting
Describe the bug
The s3 output tracks store_dir disk usage in
ctx->current_buffer_size(a
size_t). The accounting is asymmetric across restarts:pod restart with pending uploads) are adopted at startup and uploaded,
but their sizes are never added to
current_buffer_size;s3_store_file_delete()subtracts its size unconditionally:
Subtracting more than was ever added wraps the unsigned counter around
to ~2^64. From that point on, the
store_dir_limit_sizecheck ins3_store_buffer_put()considers the buffer permanently full andrejects every new chunk:
18446744073709314048 == 2^64 - 237568, i.e. the counter is at"-237568 bytes" — exactly the size of the leftover files that were
uploaded and deleted. The counter keeps drifting further "negative" as
more recovered files are deleted, and it can never recover, so the
output effectively stops accepting data until the process is restarted
(and the same thing can happen again on the next restart). This results
in permanent data loss ("chunk cannot be retried").
Note that another code path already guards against exactly this: the
quarantine path clamps the subtraction at zero.
s3_store_file_delete()is missing the same guard.
To Reproduce
store_diron a persistent path (e.g. ahostPath volume in Kubernetes) and
store_dir_limit_sizeset (e.g.20G).completes, leaving files in
store_dir.current_buffer_sizewraps to ~2^64 and every subsequent chunk isrejected with "Buffer is full" even though the disk is empty.
Expected behavior
Deleting a buffer file must not wrap the usage counter. Either the
subtraction should be clamped at zero (defensive fix, same pattern as
the quarantine path), or files adopted at startup should be accounted
into
current_buffer_size(root-cause fix), or both.Suggested fix (minimal, defensive)
I'm happy to submit a PR with this change.
Workaround
Unset
store_dir_limit_size(default 0 = unlimited): the broken checkonly runs when a limit is configured.
Your Environment
use_put_object On,store_diron a Kubernetes hostPath volume,store_dir_limit_size 20GPR Title
out_s3: guard current_buffer_size subtraction against underflow
PR Body
Fixes #<이슈번호>
s3_store_file_delete()subtracteds3_file->sizefromctx->current_buffer_sizeunconditionally. Buffer files recovered froma previous run are not accounted into
current_buffer_sizeat startup,so deleting them after upload wraps the unsigned counter around to
~2^64. The
store_dir_limit_sizecheck then treats the buffer aspermanently full and rejects every new chunk ("Buffer is full",
"chunk cannot be retried"), causing data loss until restart.
Clamp the subtraction at zero, using the same pattern already applied
on the quarantine accounting path in
s3.c.Observed in production on 5.0.9 (Kubernetes DaemonSet,
store_diron ahostPath volume surviving pod restarts):
18446744073709314048 == 2^64 - 237568— the counter went "negative"by exactly the size of the recovered-and-deleted files.
Testing
Documentation
Backporting