Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add some format string support to libbeat #2065

Merged
merged 1 commit into from
Jul 20, 2016
Merged

Conversation

urso
Copy link

@urso urso commented Jul 19, 2016

This PR adds some customizable format-string support to libbeat. This feature is not used anywhere yet, but lays some ground-work for features/improvements requiring format-string support dynamically constructing strings right from events.

EventFormatString implements format string support on events
of type common.MapStr.

The concrete event expansion requires the field name enclosed by brackets.
For example: '%{[field.name]}'. Field names can be separated by points or
multiple braces. This format %{[field.name]} is equivalent to %{[field][name]}.

Default values are given by the colon operator. For example:
%{[field.name]:default value}.

EventFormatString is build on top of generic StringFormatter Compile function.

Compile compiles an input format string into a StringFormatter. The variable
compiler vc is invoked for every variable expansion found in the input format
string.

Variable expansion are enclosed in expansion braces %{<expansion>}.
The <expansion> can contain additional parameters separated by ops
introduced by collons ':'. For example the format string %{value:v1:?v2}
will be parsed into variable expansion on value with variable ops
[(":", "v1"), (":?", "v2")]. It's up to the variable compiler to interpret
content and variable ops.

The back-slash character \ acts as escape character so the sequence '%{' will be written to the resulting string is \ is used.

return path, nil
}

// TODO: move to libbeat/common?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done :)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not done. common.MapStr does not directly support querying a field as string-value. Same method might become use-full when extracting compound keys from events (e.g. think lookup-cache)

@urso
Copy link
Author

urso commented Jul 19, 2016

$ go test -cover
PASS
coverage: 87.0% of statements
ok      github.com/elastic/beats/libbeat/common/fmtstr  0.017s

@tsg
Copy link
Contributor

tsg commented Jul 19, 2016

Will the %{ .. } parser be able to be used for other uses cases? I have one specific use case in mind:

For the index pattern, ideally we'd support the logstash syntax filebeat-%{+YYYY.MM.dd}, meaning that + introduces a timestamp formatter.

Ideally this would be combinable with the event fields, so can e.g. split in different indices by metricset: metricbeat-%{[metricset.module]}-%{[metricset.name]}-%{+YYYY.MM.dd}

@urso
Copy link
Author

urso commented Jul 19, 2016

Yeah, I have something similar in mind. This particular use-case is not fully supported yet, but I've got some ideas for future PRs to make supported expressions-types more flexible.

@ruflin
Copy link
Collaborator

ruflin commented Jul 20, 2016

LGTM

@tsg tsg merged commit 08be317 into elastic:master Jul 20, 2016
@urso urso deleted the enh/formatstr branch February 19, 2019 18:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants