Skip to content

Backport(v1.16): formatter_csv: fix memory leak (#4864) #4920

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 24, 2025

Conversation

kenhys
Copy link
Contributor

@kenhys kenhys commented Apr 23, 2025

Which issue(s) this PR fixes:
Backport #4864
Fixes #3627

What this PR does / why we need it:
Seems in_exec plugin creats Thread object each interval and it invoke formatter_csv on the thread.

If it use a Thread object as hash key when caching with formatter_csv, that thread will be permanently referenceable and will not be collected by the GC.

Here is memory usage of Ruby's process.
chart

Reproduce

See #3627

<system>
  process_name fluentd_monitor
</system>

<source>
  @type exec
  tag hardware.memory
  run_interval 0.1
  command /home/watson/prj/sandbox/fluentd/get_memory_metrics.sh
  <parse>
    @type json
    types mem_available:float,swap_total:float,swap_free:float
  </parse>
</source>

<match **>
  @type exec
  command /home/watson/prj/sandbox/fluentd/psql_memory.sh
  <format>
    @type csv
    force_quotes false
    delimiter |
    fields time_local,mem_available,swap_total,swap_free
  </format>
  <buffer time>
    @type file
    path /home/watson/prj/sandbox/fluentd/log
    timekey 50s
    timekey_wait 0s
    flush_mode lazy
    total_limit_size 200MB
    retry_type periodic  # exponential backoff is broken https://github.com/fluent/fluentd/issues/3609
    retry_max_times 0  # need to go to secondary as soon as possible https://github.com/fluent/fluentd/issues/3610
  </buffer>
</match>

Docs Changes:

Release Note:
formatter_csv: fix memory leak

**Which issue(s) this PR fixes**:
Fixes fluent#3627

**What this PR does / why we need it**:
Seems `in_exec` plugin creats Thread object each interval and it invoke
formatter_csv on the thread.

If it use a Thread object as hash key when caching with formatter_csv,
that thread will be permanently referenceable and will not be collected
by the GC.

Here is memory usage of Ruby's process.

![chart](https://github.com/user-attachments/assets/4e74adda-154b-4c00-9cb4-3b1a0e8745fd)

### Reproduce
See fluent#3627
```
<system>
  process_name fluentd_monitor
</system>

<source>
  @type exec
  tag hardware.memory
  run_interval 0.1
  command /home/watson/prj/sandbox/fluentd/get_memory_metrics.sh
  <parse>
    @type json
    types mem_available:float,swap_total:float,swap_free:float
  </parse>
</source>

<match **>
  @type exec
  command /home/watson/prj/sandbox/fluentd/psql_memory.sh
  <format>
    @type csv
    force_quotes false
    delimiter |
    fields time_local,mem_available,swap_total,swap_free
  </format>
  <buffer time>
    @type file
    path /home/watson/prj/sandbox/fluentd/log
    timekey 50s
    timekey_wait 0s
    flush_mode lazy
    total_limit_size 200MB
    retry_type periodic  # exponential backoff is broken fluent#3609
    retry_max_times 0  # need to go to secondary as soon as possible fluent#3610
  </buffer>
</match>
```

**Docs Changes**:

**Release Note**:
formatter_csv: fix memory leak

---------

Signed-off-by: Shizuo Fujita <fujita@clear-code.com>
Signed-off-by: Daijiro Fukuda <fukuda@clear-code.com>
Co-authored-by: Daijiro Fukuda <fukuda@clear-code.com>
Signed-off-by: Kentaro Hayashi <hayashi@clear-code.com>
@daipom daipom merged commit fe47c82 into fluent:v1.16 Apr 24, 2025
15 of 16 checks passed
@kenhys kenhys deleted the backport-4864 branch April 24, 2025 07:34
@daipom daipom added this to the v1.16.8 milestone Apr 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants