Skip to content

Inconsistent information about loss of data on buffering for tail input plugin #1529

Closed as not planned
@MarcioJales

Description

@MarcioJales

Hi!

I've found what I think it is some inconsistency about the possible data loss on the tail input plugin in respect of buffering.

In the "backpressure" documentation, it is stated that "The tail plugin can handle pauses without data
ingloss (sic), storing its current file offset and resuming reading later":

(for example, TCP input plugin). The tail plugin can handle pauses without data

On the other hand, the "Buffering and Storage" documentation says the opposite, "For some inputs, such as TCP and tail, pausing the input will almost certainly lead to log loss":

In a high load environment with backpressure the risks of having high memory usage is the chance of getting killed by the Kernel \(OOM Killer\). A workaround for this backpressure scenario is to limit the amount of memory in records that an input plugin can register, this configuration property is called `mem_buf_limit`. If a plugin has enqueued more than the `mem_buf_limit`, it won't be able to ingest more until that data can be delivered or flushed properly. In this scenario the input plugin in question is paused. When the input is paused, records will not be ingested until it is resumed. For some inputs, such as TCP and tail, pausing the input will almost certainly lead to log loss. For the tail input, Fluent Bit can save its current offset in the current file it is reading, and pick back up when the input is resumed.

I'd like to know, then, what is the behavior expected in this realm?

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions