Skip to content

Commit

Permalink
Add additional time formats to decode_cef (elastic#19346)
Browse files Browse the repository at this point in the history
The date formats in the CEF guide describe the time formats in terms of Java's SimpleTimeFormat class.
The `zzz` specifier covers a few additional formats than what are covered by `MST` in Go's time format.
Namely on the Go side it was missing support for offsets (e.g. +04, +0400, +04:00). This change additional
adds support for the ISO8601 `Z` time zone value (this does not strictly match the CEF guide's format).

These are the Java SimpleDateFormats in the CEF guide:

MMM dd HH:mm:ss.SSS zzz
MMM dd HH:mm:sss.SSS
MMM dd HH:mm:ss zzz
MMM dd HH:mm:ss
MMM dd yyyy HH:mm:ss.SSS zzz
MMM dd yyyy HH:mm:ss.SSS
MMM dd yyyy HH:mm:ss zzz
MMM dd yyyy HH:mm:ss

(cherry picked from commit b82829b)
  • Loading branch information
andrewkroh committed Jun 25, 2020
1 parent 0cb01b7 commit bd201c9
Show file tree
Hide file tree
Showing 3 changed files with 79 additions and 0 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -276,6 +276,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Add support for array parsing in azure-eventhub input. {pull}18585[18585]
from being added to events by default. {pull}18159[18159]
- Improved performance of PANW sample dashboards. {issue}19031[19031] {pull}19032[19032]
- Add support for timezone offsets and `Z` to decode_cef timestamp parser. {pull}19346[19346]

*Heartbeat*

Expand Down
15 changes: 15 additions & 0 deletions x-pack/filebeat/processors/decode_cef/cef/types.go
Original file line number Diff line number Diff line change
Expand Up @@ -103,18 +103,33 @@ func toMACAddress(v string) (string, error) {
var timeLayouts = []string{
// MMM dd HH:mm:ss.SSS zzz
"Jan _2 15:04:05.000 MST",
"Jan _2 15:04:05.000 Z0700",
"Jan _2 15:04:05.000 Z07:00",

// MMM dd HH:mm:sss.SSS
"Jan _2 15:04:05.000",

// MMM dd HH:mm:ss zzz
"Jan _2 15:04:05 MST",
"Jan _2 15:04:05 Z0700",
"Jan _2 15:04:05 Z07:00",

// MMM dd HH:mm:ss
"Jan _2 15:04:05",

// MMM dd yyyy HH:mm:ss.SSS zzz
"Jan _2 2006 15:04:05.000 MST",
"Jan _2 2006 15:04:05.000 Z0700",
"Jan _2 2006 15:04:05.000 Z07:00",

// MMM dd yyyy HH:mm:ss.SSS
"Jan _2 2006 15:04:05.000",

// MMM dd yyyy HH:mm:ss zzz
"Jan _2 2006 15:04:05 MST",
"Jan _2 2006 15:04:05 Z0700",
"Jan _2 2006 15:04:05 Z07:00",

// MMM dd yyyy HH:mm:ss
"Jan _2 2006 15:04:05",
}
Expand Down
63 changes: 63 additions & 0 deletions x-pack/filebeat/processors/decode_cef/cef/types_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
// or more contributor license agreements. Licensed under the Elastic License;
// you may not use this file except in compliance with the Elastic License.

package cef

import (
"testing"

"github.com/stretchr/testify/assert"
)

func TestToTimestamp(t *testing.T) {
var times = []string{
// Unix epoch in milliseconds.
"1322004689000",

// MMM dd HH:mm:ss.SSS zzz
"Jun 23 17:37:24.000 Z",
"Jun 23 17:37:24.000 EST",
"Jun 23 17:37:24.000 +05",
"Jun 23 17:37:24.000 +0500",
"Jun 23 17:37:24.000 +05:00",

// MMM dd HH:mm:sss.SSS
"Jun 23 17:37:24.000",

// MMM dd HH:mm:ss zzz
"Jun 23 17:37:24 Z",
"Jun 23 17:37:24 EST",
"Jun 23 17:37:24 +05",
"Jun 23 17:37:24 +0500",
"Jun 23 17:37:24 +05:00",

// MMM dd HH:mm:ss
"Jun 23 17:37:24",

// MMM dd yyyy HH:mm:ss.SSS zzz
"Jun 23 2020 17:37:24.000 Z",
"Jun 23 2020 17:37:24.000 EST",
"Jun 23 2020 17:37:24.000 +05",
"Jun 23 2020 17:37:24.000 +0500",
"Jun 23 2020 17:37:24.000 +05:00",

// MMM dd yyyy HH:mm:ss.SSS
"Jun 23 2020 17:37:24.000",

// MMM dd yyyy HH:mm:ss zzz
"Jun 23 2020 17:37:24 Z",
"Jun 23 2020 17:37:24 EST",
"Jun 23 2020 17:37:24 +05",
"Jun 23 2020 17:37:24 +0500",
"Jun 23 2020 17:37:24 +05:00",

// MMM dd yyyy HH:mm:ss
"Jun 23 2020 17:37:24",
}

for _, timeValue := range times {
_, err := toTimestamp(timeValue)
assert.NoError(t, err, timeValue)
}
}

0 comments on commit bd201c9

Please sign in to comment.