Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

in_tcp: Add message_length_limit to drop large incoming data #4137

Merged
merged 1 commit into from
Apr 17, 2023

Conversation

daipom
Copy link
Contributor

@daipom daipom commented Apr 10, 2023

Which issue(s) this PR fixes:
None.

What this PR does / why we need it:
Add the same feature to in_tcp as message_length_limit in in_udp, to drop too large incoming data.

Docs Changes:
fluent/fluentd-docs-gitbook#452

Release Note:
Same as the title.

TODO:

  • Add tests
  • Add document

@daipom
Copy link
Contributor Author

daipom commented Apr 10, 2023

Behavior of message_length_limit

  • When the byte-size of the buffer exceeds the message_length_limit, the buffer will be cleared.
  • If there is subsequent data, that part is discarded so that a warning of parsing failure doesn't occur.

Run Fluentd with the following setting.

<source>
  @type tcp
  tag test
  message_length_limit 12
  delimiter ";"
  <parse>
    @type json
  </parse>
</source>

<match test.**>
  @type stdout
</match>

Send data as follows.

$ netcat 0.0.0.0 5170
{"message":
"hello
world!"};
{"k":"v"};

Result:

[info]: #0 The buffer size exceeds 'message_length_limit', cleared: limit=12 size=19 head="{\"message\":\n\"hello\n"
(datetime) test: {"k":"v"}

Normal behavior

Run Fluentd with the following setting.

<source>
  @type tcp
  tag test
  delimiter ";"
  <parse>
    @type json
  </parse>
</source>

<match test.**>
  @type stdout
</match>

Send data as follows.

$ netcat 0.0.0.0 5170
{"message":
"hello
world!"};
{"k":"v"};

Result:

(datetime) test: {"message":"hello\nworld!"}
(datetime) test: {"k":"v"}

@daipom daipom changed the title in_tcp: Add message_length_limit to drop large incoming data in_tcp: Add message_length_limit to drop large incoming data Apr 10, 2023
@daipom daipom force-pushed the in_tcp-add-message_length_limit branch from 47463b6 to 283a3b3 Compare April 10, 2023 16:06
@daipom
Copy link
Contributor Author

daipom commented Apr 10, 2023

Writing to a socket at high speed can cause multiple writes to be stored in the buffer before the server reads them.
The new tests are exactly those.

It is undesirable that high-speed writing causes the total buffer size to exceed the limit and drop the records even if each record size does not exceed the limit.

So I fixed to check the size at 2 points: delimited one message size and total remaining buffer size.

One concern is whether this fix will affect the performance improvement of the next fix.

@daipom daipom marked this pull request as ready for review April 11, 2023 00:44
@daipom daipom requested a review from ashie April 11, 2023 00:44
@ashie ashie added this to the v1.16.1 milestone Apr 13, 2023
test/plugin/test_in_tcp.rb Outdated Show resolved Hide resolved
Add the same feature to `in_tcp` as `message_length_limit` in
`in_udp`, to drop too large incoming data.

Signed-off-by: Daijiro Fukuda <fukuda@clear-code.com>
Co-authored-by: Takuro Ashie <ashie@clear-code.com>
@daipom daipom force-pushed the in_tcp-add-message_length_limit branch from 283a3b3 to 0d9c320 Compare April 14, 2023 10:27
@ashie ashie merged commit 11c9550 into fluent:master Apr 17, 2023
@daipom daipom deleted the in_tcp-add-message_length_limit branch April 17, 2023 01:06
@daipom
Copy link
Contributor Author

daipom commented Apr 17, 2023

Thanks for your review!

daipom added a commit to daipom/fluentd-docs-gitbook that referenced this pull request Apr 20, 2023
fluent/fluentd#4137

Signed-off-by: Daijiro Fukuda <fukuda@clear-code.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants