-
Notifications
You must be signed in to change notification settings - Fork 5.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Logfmt parser #4539
Logfmt parser #4539
Conversation
…between Parse and ParseLine.
plugins/parsers/logfmt/parser.go
Outdated
|
||
//add default tags | ||
metrics = append(metrics, m) | ||
p.applyDefaultTags(metrics) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Call only once at end of function, since this applies the tags to all metrics.
plugins/parsers/logfmt/parser.go
Outdated
//attempt type conversions | ||
value := string(decoder.Value()) | ||
if iValue, err := strconv.Atoi(value); err == nil { | ||
//log.Printf("Print Atoi Value Here:", iValue) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Clean up (delete) log messages.
plugins/parsers/logfmt/parser.go
Outdated
@@ -0,0 +1,115 @@ | |||
// Package logfmt converts logfmt data into metrics. New comment |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Take a pass through the file and make sure comments are all useful and well written.
plugins/parsers/logfmt/parser.go
Outdated
"strconv" | ||
"time" | ||
|
||
glogfmt "github.com/go-logfmt/logfmt" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No reason to alias the import path since there is no conflict.
plugins/parsers/logfmt/parser.go
Outdated
|
||
//attempt type conversions | ||
value := string(decoder.Value()) | ||
if iValue, err := strconv.Atoi(value); err == nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use strconv.ParseInt
here because it handles 64-bit ints correctly.
docs/DATA_FORMATS_INPUT.md
Outdated
# Logfmt | ||
This parser implements the logfmt format by extracting key-value pairs from log text in the form `<key>=<value>`. | ||
At the moment, the plugin will produce one metric per line and all keys | ||
are added as fields. Values are left as strings (for now). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Values are left as strings
The code below doesn't do this.
docs/DATA_FORMATS_INPUT.md
Outdated
A typical log | ||
``` | ||
method=GET host=influxdata.org ts=2018-07-24T19:43:40.275Z | ||
connect=4ms service=8ms status=200 bytes=1653 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This example should be a single line
docs/DATA_FORMATS_INPUT.md
Outdated
"connect": "4ms", | ||
"service": "8ms", | ||
"status": 200, | ||
"bytes": 1653, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Show the converted version as line protocol.
name: "Log parser only returns metrics from first string", | ||
now: func() time.Time { return time.Unix(0, 0) }, | ||
measurement: "testlog", | ||
s: `ts=2018-07-24T19:43:35.207268Z lvl=5 msg="Write failed" log_id=09R4e4Rl000 "/n" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fix the newline test here \n
, like up on line 89
docs/DATA_FORMATS_INPUT.md
Outdated
# Logfmt | ||
This parser implements the logfmt format by extracting key-value pairs from log text in the form `<key>=<value>`. | ||
At the moment, the plugin will produce one metric per line and all keys | ||
are added as fields. Values are left as strings (for now). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Values are left as strings (for now).
This isn't true, we are doing auto conversion to other types.
// NewParser creates a parser. | ||
func NewParser(metricName string, defaultTags map[string]string) *Parser { | ||
return &Parser{ | ||
MetricName: metricName, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What happens if this is not set?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
By default, doesn't telegraf use the plugin name of the input/processor that uses it?
docs/DATA_FORMATS_INPUT.md
Outdated
``` | ||
will be converted into | ||
``` | ||
method=GET, host=influxdata.org, ts=2018-07-24T19:43:40.275Z, connect=4ms, service=8ms, status=200, bytes=1653, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Turn this into valid line protocol:
logfmt method="GET",host="influxdata.org",ts="2018-07-24T19:43:40.275Z",connect="4ms",service="8ms",status=200i,bytes=1653i
Required for all PRs:
closes #4427