Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logfmt parser #4539

Merged
merged 22 commits into from
Aug 22, 2018
Merged

Logfmt parser #4539

merged 22 commits into from
Aug 22, 2018

Conversation

Ayrdrie
Copy link
Contributor

@Ayrdrie Ayrdrie commented Aug 9, 2018

Required for all PRs:

closes #4427

  • Signed CLA.
  • Associated README.md updated.
  • Has appropriate unit tests.


//add default tags
metrics = append(metrics, m)
p.applyDefaultTags(metrics)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Call only once at end of function, since this applies the tags to all metrics.

//attempt type conversions
value := string(decoder.Value())
if iValue, err := strconv.Atoi(value); err == nil {
//log.Printf("Print Atoi Value Here:", iValue)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean up (delete) log messages.

@@ -0,0 +1,115 @@
// Package logfmt converts logfmt data into metrics. New comment
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Take a pass through the file and make sure comments are all useful and well written.

"strconv"
"time"

glogfmt "github.com/go-logfmt/logfmt"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No reason to alias the import path since there is no conflict.


//attempt type conversions
value := string(decoder.Value())
if iValue, err := strconv.Atoi(value); err == nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use strconv.ParseInt here because it handles 64-bit ints correctly.

# Logfmt
This parser implements the logfmt format by extracting key-value pairs from log text in the form `<key>=<value>`.
At the moment, the plugin will produce one metric per line and all keys
are added as fields. Values are left as strings (for now).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Values are left as strings

The code below doesn't do this.

A typical log
```
method=GET host=influxdata.org ts=2018-07-24T19:43:40.275Z
connect=4ms service=8ms status=200 bytes=1653
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This example should be a single line

"connect": "4ms",
"service": "8ms",
"status": 200,
"bytes": 1653,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Show the converted version as line protocol.

name: "Log parser only returns metrics from first string",
now: func() time.Time { return time.Unix(0, 0) },
measurement: "testlog",
s: `ts=2018-07-24T19:43:35.207268Z lvl=5 msg="Write failed" log_id=09R4e4Rl000 "/n"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix the newline test here \n, like up on line 89

# Logfmt
This parser implements the logfmt format by extracting key-value pairs from log text in the form `<key>=<value>`.
At the moment, the plugin will produce one metric per line and all keys
are added as fields. Values are left as strings (for now).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Values are left as strings (for now).

This isn't true, we are doing auto conversion to other types.

// NewParser creates a parser.
func NewParser(metricName string, defaultTags map[string]string) *Parser {
return &Parser{
MetricName: metricName,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if this is not set?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By default, doesn't telegraf use the plugin name of the input/processor that uses it?

```
will be converted into
```
method=GET, host=influxdata.org, ts=2018-07-24T19:43:40.275Z, connect=4ms, service=8ms, status=200, bytes=1653,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Turn this into valid line protocol:

logfmt method="GET",host="influxdata.org",ts="2018-07-24T19:43:40.275Z",connect="4ms",service="8ms",status=200i,bytes=1653i

@danielnelson danielnelson added this to the 1.8.0 milestone Aug 20, 2018
@danielnelson danielnelson merged commit e893dc3 into master Aug 22, 2018
@danielnelson danielnelson deleted the logfmt-parser branch August 22, 2018 20:55
rgitzel pushed a commit to rgitzel/telegraf that referenced this pull request Oct 17, 2018
otherpirate pushed a commit to otherpirate/telegraf that referenced this pull request Mar 15, 2019
otherpirate pushed a commit to otherpirate/telegraf that referenced this pull request Mar 15, 2019
dupondje pushed a commit to dupondje/telegraf that referenced this pull request Apr 22, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add logfmt Parser Plugin
3 participants