Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add special syslog timestamp parser that uses current year #4190

Merged
merged 1 commit into from
May 23, 2018
Merged

Add special syslog timestamp parser that uses current year #4190

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions plugins/inputs/logparser/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,7 @@ You must capture at least one field per line.
- ts-httpd ("02/Jan/2006:15:04:05 -0700")
- ts-epoch (seconds since unix epoch, may contain decimal)
- ts-epochnano (nanoseconds since unix epoch)
- ts-syslog ("Jan 02 15:04:05", parsed time is set to the current year)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@danielnelson this is the BSD syslog timestamp format (RFC3164) not the syslog RFC5424 timestamp (basically a RFC3339 micro timestamp).
So probably you want to call it ts-bsd-syslog, and maybe also add another format for ts-syslog.
Just my 2 cents.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding is that both of these RFCs describe the wire format for syslog messages, and not how the time might be written to logfiles. Is there any documention that describes how the timestamp in, for instance, /var/log/syslog are written?

Another concern I had, and maybe you can help with, is if these dates are commonly localized?

Copy link
Contributor

@leodido leodido May 25, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The timestamp depends on the format (so RFC) chosen.
For example you can configure RSYSLOG to use RSYSLOG_SyslogProtocol23Format format, which means RFC5424. With this setting /var/log/* will contains RFC3339MICRO timestamps (as per RFC5424). Without this setting it will default to the old (RFC3164) BSD syslog (and timestamp) format.

Copy link
Contributor

@leodido leodido May 25, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding localization. RFC5425 with RFC3339MICRO timestamps is not affected.
Rather, timestamps of syslog messages following the old RFC3164 are not localized (must be always in english) (ref).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I'll leave this as ts-syslog since it pairs with the built in grok pattern SYSLOGTIMESTAMP and there isn't any reason to add a special pattern for rfc5424 style timestamps since we already have ts-rfc3339 and ts-rfc3339nano. We only need a special type here to inject the year.

- ts-"CUSTOM"

CUSTOM time layouts must be within quotes and be the representation of the
Expand Down
17 changes: 17 additions & 0 deletions plugins/inputs/logparser/grok/grok.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ var timeLayouts = map[string]string{
// will get handled in the ParseLine function.
"ts-epoch": "EPOCH",
"ts-epochnano": "EPOCH_NANO",
"ts-syslog": "SYSLOG_TIMESTAMP",
"ts": "GENERIC_TIMESTAMP", // try parsing all known timestamp layouts.
}

Expand All @@ -44,6 +45,7 @@ const (
DROP = "drop"
EPOCH = "EPOCH"
EPOCH_NANO = "EPOCH_NANO"
SYSLOG_TIMESTAMP = "SYSLOG_TIMESTAMP"
GENERIC_TIMESTAMP = "GENERIC_TIMESTAMP"
)

Expand Down Expand Up @@ -112,6 +114,7 @@ type Parser struct {
// layouts.
foundTsLayouts []string

timeFunc func() time.Time
g *grok.Grok
tsModder *tsModder
}
Expand Down Expand Up @@ -174,6 +177,10 @@ func (p *Parser) Compile() error {
p.loc, _ = time.LoadLocation("UTC")
}

if p.timeFunc == nil {
p.timeFunc = time.Now
}

return p.compileCustomPatterns()
}

Expand Down Expand Up @@ -285,6 +292,16 @@ func (p *Parser) ParseLine(line string) (telegraf.Metric, error) {
} else {
timestamp = time.Unix(0, iv)
}
case SYSLOG_TIMESTAMP:
ts, err := time.ParseInLocation("Jan 02 15:04:05", v, p.loc)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @danielnelson
Should we use "Jan 2 15:04:05" instead of "Jan 02 15:04:05" ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think maybe it should be Jan _2 15:04:05, but I can't seem to create a testcase that fails. Do you have a example that is failing?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we try to parse "Sep 2 09:01:55 value=42" in test function TestSyslogTimestampParser(),
the test will show the following:
=== RUN TestSyslogTimestampParser 2018/06/22 09:36:41 E! Error parsing Sep 2 09:01:55 to time layout [SYSLOG_TIMESTAMP]: parsing time "Sep 2 09:01:55" as "Jan 02 15:04:05": cannot parse "2 09:01:55" as "02"

Does this the example count?
Thanks.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh, the test was "passing" even though the parsing failed. Here is the fix: #4334

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That fix looks nice. Thank you.

if err == nil {
if ts.Year() == 0 {
ts = ts.AddDate(timestamp.Year(), 0, 0)
}
timestamp = ts
} else {
log.Printf("E! Error parsing %s to time layout [%s]: %s", v, t, err)
}
case GENERIC_TIMESTAMP:
var foundTs bool
// first try timestamp layouts that we've already found
Expand Down
12 changes: 12 additions & 0 deletions plugins/inputs/logparser/grok/grok_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -970,3 +970,15 @@ func TestNewlineInPatterns(t *testing.T) {
require.NoError(t, err)
require.NotNil(t, m)
}

func TestSyslogTimestampParser(t *testing.T) {
p := &Parser{
Patterns: []string{`%{SYSLOGTIMESTAMP:timestamp:ts-syslog} value=%{NUMBER:value:int}`},
timeFunc: func() time.Time { return time.Date(2018, time.April, 1, 0, 0, 0, 0, nil) },
}
require.NoError(t, p.Compile())
m, err := p.ParseLine("Sep 25 09:01:55 value=42")
require.NoError(t, err)
require.NotNil(t, m)
require.Equal(t, 2018, m.Time().Year())
}