Skip to content

Commit

Permalink
Add additional time formats to decode_cef (elastic#19346)
Browse files Browse the repository at this point in the history
The date formats in the CEF guide describe the time formats in terms of Java's SimpleTimeFormat class.
The `zzz` specifier covers a few additional formats than what are covered by `MST` in Go's time format.
Namely on the Go side it was missing support for offsets (e.g. +04, +0400, +04:00). This change additional
adds support for the ISO8601 `Z` time zone value (this does not strictly match the CEF guide's format).

These are the Java SimpleDateFormats in the CEF guide:

MMM dd HH:mm:ss.SSS zzz
MMM dd HH:mm:sss.SSS
MMM dd HH:mm:ss zzz
MMM dd HH:mm:ss
MMM dd yyyy HH:mm:ss.SSS zzz
MMM dd yyyy HH:mm:ss.SSS
MMM dd yyyy HH:mm:ss zzz
MMM dd yyyy HH:mm:ss
  • Loading branch information
andrewkroh authored Jun 25, 2020
1 parent b5fe0f7 commit b82829b
Show file tree
Hide file tree
Showing 3 changed files with 79 additions and 0 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -413,6 +413,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Explicitly set ECS version in all Filebeat modules. {pull}19198[19198]
- Add new mode to multiline reader to aggregate constant number of lines {pull}18352[18352]
- Add automatic retries and exponential backoff to httpjson input. {pull}18956[18956]
- Add support for timezone offsets and `Z` to decode_cef timestamp parser. {pull}19346[19346]
- Improve ECS categorization field mappings in traefik module. {issue}16183[16183] {pull}19379[19379]

*Heartbeat*
Expand Down
15 changes: 15 additions & 0 deletions x-pack/filebeat/processors/decode_cef/cef/types.go
Original file line number Diff line number Diff line change
Expand Up @@ -103,18 +103,33 @@ func toMACAddress(v string) (string, error) {
var timeLayouts = []string{
// MMM dd HH:mm:ss.SSS zzz
"Jan _2 15:04:05.000 MST",
"Jan _2 15:04:05.000 Z0700",
"Jan _2 15:04:05.000 Z07:00",

// MMM dd HH:mm:sss.SSS
"Jan _2 15:04:05.000",

// MMM dd HH:mm:ss zzz
"Jan _2 15:04:05 MST",
"Jan _2 15:04:05 Z0700",
"Jan _2 15:04:05 Z07:00",

// MMM dd HH:mm:ss
"Jan _2 15:04:05",

// MMM dd yyyy HH:mm:ss.SSS zzz
"Jan _2 2006 15:04:05.000 MST",
"Jan _2 2006 15:04:05.000 Z0700",
"Jan _2 2006 15:04:05.000 Z07:00",

// MMM dd yyyy HH:mm:ss.SSS
"Jan _2 2006 15:04:05.000",

// MMM dd yyyy HH:mm:ss zzz
"Jan _2 2006 15:04:05 MST",
"Jan _2 2006 15:04:05 Z0700",
"Jan _2 2006 15:04:05 Z07:00",

// MMM dd yyyy HH:mm:ss
"Jan _2 2006 15:04:05",
}
Expand Down
63 changes: 63 additions & 0 deletions x-pack/filebeat/processors/decode_cef/cef/types_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
// or more contributor license agreements. Licensed under the Elastic License;
// you may not use this file except in compliance with the Elastic License.

package cef

import (
"testing"

"github.com/stretchr/testify/assert"
)

func TestToTimestamp(t *testing.T) {
var times = []string{
// Unix epoch in milliseconds.
"1322004689000",

// MMM dd HH:mm:ss.SSS zzz
"Jun 23 17:37:24.000 Z",
"Jun 23 17:37:24.000 EST",
"Jun 23 17:37:24.000 +05",
"Jun 23 17:37:24.000 +0500",
"Jun 23 17:37:24.000 +05:00",

// MMM dd HH:mm:sss.SSS
"Jun 23 17:37:24.000",

// MMM dd HH:mm:ss zzz
"Jun 23 17:37:24 Z",
"Jun 23 17:37:24 EST",
"Jun 23 17:37:24 +05",
"Jun 23 17:37:24 +0500",
"Jun 23 17:37:24 +05:00",

// MMM dd HH:mm:ss
"Jun 23 17:37:24",

// MMM dd yyyy HH:mm:ss.SSS zzz
"Jun 23 2020 17:37:24.000 Z",
"Jun 23 2020 17:37:24.000 EST",
"Jun 23 2020 17:37:24.000 +05",
"Jun 23 2020 17:37:24.000 +0500",
"Jun 23 2020 17:37:24.000 +05:00",

// MMM dd yyyy HH:mm:ss.SSS
"Jun 23 2020 17:37:24.000",

// MMM dd yyyy HH:mm:ss zzz
"Jun 23 2020 17:37:24 Z",
"Jun 23 2020 17:37:24 EST",
"Jun 23 2020 17:37:24 +05",
"Jun 23 2020 17:37:24 +0500",
"Jun 23 2020 17:37:24 +05:00",

// MMM dd yyyy HH:mm:ss
"Jun 23 2020 17:37:24",
}

for _, timeValue := range times {
_, err := toTimestamp(timeValue)
assert.NoError(t, err, timeValue)
}
}

0 comments on commit b82829b

Please sign in to comment.