Skip to content

admin: memory: updating for style #1505

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 30 additions & 15 deletions administration/memory-management.md
Original file line number Diff line number Diff line change
@@ -1,30 +1,46 @@
# Memory Management
# Memory management

<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=5cc3ce54-e910-4ebf-85f5-f02530b3e11b" />

In certain scenarios it would be ideal to estimate how much memory Fluent Bit could be using, this is very useful for containerized environments where memory limits are a must.
You might need to estimate how much memory Fluent Bit could be using in scenarios
like containerized environments where memory limits are essential.

In order to that we will assume that the input plugins have set the **Mem\_Buf\_Limit** option \(you can learn more about it in the [Backpressure](backpressure.md) section\).
To make an estimate, in-use input plugins must set the `Mem_Buf_Limit`option.
Learn more about it in [Backpressure](backpressure.md).

## Estimating

Input plugins append data independently, so in order to do an estimation, a limit should be imposed through the **Mem\_Buf\_Limit** option. If the limit was set to _10MB_ we need to estimate that in the worse case, the output plugin likely could use _20MB_.
Input plugins append data independently. To make an estimation, impose a limit with
the `Mem_Buf_Limit` option. If the limit was set to `10MB`, you can estimate that in
the worst case, the output plugin likely could use `20MB`.

Fluent Bit has an internal binary representation for the data being processed, but when this data reaches an output plugin, it will likely create its own representation in a new memory buffer for processing.
The best examples are the [InfluxDB](../pipeline/outputs/influxdb.md) and [Elasticsearch](../pipeline/outputs/elasticsearch.md) output plugins, both need to convert the binary representation to their respective custom JSON formats before it can be sent to the backend servers.
Fluent Bit has an internal binary representation for the data being processed. When
this data reaches an output plugin, it can create its own representation in a new
memory buffer for processing. The best examples are the
[InfluxDB](../pipeline/outputs/influxdb.md) and
[Elasticsearch](../pipeline/outputs/elasticsearch.md) output plugins, which need to
convert the binary representation to their respective custom JSON formats before
sending data to the backend servers.

So, if we impose a limit of _10MB_ for the input plugins and consider the worse case scenario of the output plugin consuming _20MB_ extra, as a minimum we need \(_30MB_ x 1.2\) = **36MB**.
When imposing a limit of `10MB` for the input plugins, and a worst case scenario of
the output plugin consuming `20MB`, you need to allocate a minimum (`30MB` x 1.2) =
`36MB`.

## Glibc and Memory Fragmentation
## Glibc and memory fragmentation

It is well known that in intensive environments where memory allocations happen in the orders of magnitude, the default memory allocator provided by Glibc could lead to high fragmentation, reporting a high memory usage by the service.
In intensive environments where memory allocations happen in the orders of magnitude,
the default memory allocator provided by Glibc could lead to high fragmentation,
reporting a high memory usage by the service.

It's strongly suggested that in any production environment, Fluent Bit should be built with [jemalloc](http://jemalloc.net/) enabled \(e.g. `-DFLB_JEMALLOC=On`\). Jemalloc is an alternative memory allocator that can reduce fragmentation \(among others things\) resulting in better performance.
It's strongly suggested that in any production environment, Fluent Bit should be
built with [jemalloc](http://jemalloc.net/) enabled (`-DFLB_JEMALLOC=On`).
The jemalloc implementation of malloc is an alternative memory allocator that can
reduce fragmentation, resulting in better performance.

You can check if Fluent Bit has been built with Jemalloc using the following command:
Use the following command to determine if Fluent Bit has been built with jemalloc:

```text
$ bin/fluent-bit -h | grep JEMALLOC
```bash
bin/fluent-bit -h | grep JEMALLOC
```

The output should look like:
Expand All @@ -36,5 +52,4 @@ FLB_HAVE_PROXY_GO FLB_HAVE_JEMALLOC JEMALLOC_MANGLE FLB_HAVE_REGEX
FLB_HAVE_C_TLS FLB_HAVE_SETJMP FLB_HAVE_ACCEPT4 FLB_HAVE_INOTIFY
```

If the FLB\_HAVE\_JEMALLOC option is listed in _Build Flags_, everything will be fine.

If the `FLB_HAVE_JEMALLOC` option is listed in `Build Flags`, jemalloc is enabled.
2 changes: 2 additions & 0 deletions vale-styles/FluentBit/Spelling-exceptions.txt
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ Fargate
Firehose
FluentBit
Fluentd
glibc
Golang
golib
Grafana
Expand All @@ -72,6 +73,7 @@ Hostname
Ingester
Keepalive
Istio
jemalloc
keepalive
Kinesis
kubectl
Expand Down