-
Notifications
You must be signed in to change notification settings - Fork 539
Closed as not planned
Closed as not planned
Copy link
Labels
Description
Hi!
I've found what I think it is some inconsistency about the possible data loss on the tail
input plugin in respect of buffering.
In the "backpressure" documentation, it is stated that "The tail plugin can handle pauses without data
ingloss (sic), storing its current file offset and resuming reading later":
(for example, TCP input plugin). The tail plugin can handle pauses without data |
On the other hand, the "Buffering and Storage" documentation says the opposite, "For some inputs, such as TCP and tail, pausing the input will almost certainly lead to log loss":
In a high load environment with backpressure the risks of having high memory usage is the chance of getting killed by the Kernel \(OOM Killer\). A workaround for this backpressure scenario is to limit the amount of memory in records that an input plugin can register, this configuration property is called `mem_buf_limit`. If a plugin has enqueued more than the `mem_buf_limit`, it won't be able to ingest more until that data can be delivered or flushed properly. In this scenario the input plugin in question is paused. When the input is paused, records will not be ingested until it is resumed. For some inputs, such as TCP and tail, pausing the input will almost certainly lead to log loss. For the tail input, Fluent Bit can save its current offset in the current file it is reading, and pick back up when the input is resumed. |
I'd like to know, then, what is the behavior expected in this realm?