Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compressable#decompress Improve performance #2031

Merged
merged 1 commit into from
Jun 25, 2018

Conversation

ksss
Copy link
Contributor

@ksss ksss commented Jun 24, 2018

This patch expect to reduce copy string object.

benchmark with minor revisions in https://github.com/fluent/fluentd-benchmark/tree/182f773eb2e2f0ba96520d80cd2a1a8541e05875/one_forward

agent.conf

<source>
  @type tail
  path dummy.log
  pos_file /var/tmp/_var_log_dummy.pos
  format none
  tag dummy
</source>
<match dummy>
  @type copy
  <store>
    @type forward
    flush_interval 0
    try_flush_interval 1
    buffer_chunk_limit 1m
    buffer_queue_limit 64
    num_threads 32
    compress gzip
    # phi_threshold 35
    <server>
      host 127.0.0.1 # FIX ME
      port 24224
    </server>
  </store>
</match>

receiver.conf

<source>
  @type forward
  port 24224
</source>
<match dummy>
  @type flowcounter_simple
  unit minute
</match>

dummer.conf

configure 'sample' do
  output "dummy.log"
  message "time:2013-11-25 00:23:52 +0900\tlevel:ERROR\tmethod:POST\turi:/api/v1/people\treqtime:3.1983877060667103"
end
$ bundle exec dummer -c dummer.conf -r 300000

before

2018-06-24 11:29:23 +0900 [info]: #0 plugin:out_flowcounter_simple	count:5423147	indicator:num	unit:minute
2018-06-24 11:30:23 +0900 [info]: #0 plugin:out_flowcounter_simple	count:5456601	indicator:num	unit:minute
2018-06-24 11:31:23 +0900 [info]: #0 plugin:out_flowcounter_simple	count:5406923	indicator:num	unit:minute
2018-06-24 11:32:23 +0900 [info]: #0 plugin:out_flowcounter_simple	count:5449504	indicator:num	unit:minute
2018-06-24 11:33:23 +0900 [info]: #0 plugin:out_flowcounter_simple	count:5381575	indicator:num	unit:minute

after

2018-06-24 11:37:55 +0900 [info]: #0 plugin:out_flowcounter_simple	count:5918921	indicator:num	unit:minute
2018-06-24 11:38:55 +0900 [info]: #0 plugin:out_flowcounter_simple	count:6396454	indicator:num	unit:minute
2018-06-24 11:39:55 +0900 [info]: #0 plugin:out_flowcounter_simple	count:6604293	indicator:num	unit:minute
2018-06-24 11:40:55 +0900 [info]: #0 plugin:out_flowcounter_simple	count:5642139	indicator:num	unit:minute
2018-06-24 11:41:55 +0900 [info]: #0 plugin:out_flowcounter_simple	count:6128793	indicator:num	unit:minute

(about 13% increase)

fluentd commit: 84c2194
dummer (0.4.0)
fluent-plugin-flowcounter-simple (0.0.4)

@repeatedly repeatedly merged commit b593e6f into fluent:master Jun 25, 2018
@repeatedly
Copy link
Member

Good catch! Thanks!

@ksss ksss deleted the decompress branch June 25, 2018 03:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants