Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

in_syslog: add option 'original_message_key' #2500

Closed

Conversation

candlerb
Copy link
Contributor

(Note: this applies on top of PR #2499 )

Which issue(s) this PR fixes:

N/A

What this PR does / why we need it:

Some log systems like loki natively work on raw text lines, and are more natural to use in this form as opposed to a JSON collection of parsed fields.

Loki can accept certain parsed fields out-of-band as tags, but only if low-cardinality. syslog parsing can generate high cardinality values, in particular pid, so all these values cannot be mapped to tags. But if they are not mapped to tags, and the original line is not kept, then these values are lost.

I think it would also be useful with out_file to record the entire original syslog line.

Docs Changes:

Add new setting to https://github.com/fluent/fluentd-docs-gitbook/blob/1.0/plugins/input/syslog.md

Release Note:

### Enhancement

in_syslog: add `original_message_key` to retain the entire original message

Fixes fluent#2497

Signed-off-by: Brian Candler <b.candler@pobox.com>
Allows the entire message to be retained, as well as the parsed components

Signed-off-by: Brian Candler <b.candler@pobox.com>
@repeatedly
Copy link
Member

This is done by meta parser like multi-format-parser and it is more general approach.

<parse>
  @type keep_original  # wrap other parser and assign original `text` into parsed result.
  <parser>
    @type json
  </parser>
</parse>

@candlerb
Copy link
Contributor Author

For benefit of readers here is a link to multi-format-parser

@candlerb candlerb closed this Jul 16, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants