Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Teach elasticsearch/audit fileset to handle timestamps correctly #15942

Merged
merged 6 commits into from
Jan 30, 2020
Merged

Teach elasticsearch/audit fileset to handle timestamps correctly #15942

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
f06e72b
Add samples from ES 7.1.1 audit log
ycombinator Jan 29, 2020
8105784
When JSON logs contain @timestamp and TZ is detected, use detected TZ
ycombinator Jan 29, 2020
1bace41
Add CHANGELOG entry
ycombinator Jan 29, 2020
7ee2045
Remove redundant guard
ycombinator Jan 29, 2020
b1c9d27
Revert unrelated change
ycombinator Jan 29, 2020
741e892
s/teach/improve/ in changelog entry
ycombinator Jan 30, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Fix s3 input hanging with GetObjectRequest API call by adding context_timeout config. {issue}15502[15502] {pull}15590[15590]
- Add shared_credential_file to cloudtrail config {issue}15652[15652] {pull}15656[15656]
- Fix typos in zeek notice fileset config file. {issue}15764[15764] {pull}15765[15765]
- Teach `elasticsearch/audit` fileset to handle timestamps correctly. {pull}15942[15942]
ycombinator marked this conversation as resolved.
Show resolved Hide resolved

*Heartbeat*

Expand Down
3 changes: 1 addition & 2 deletions filebeat/module/elasticsearch/audit/config/audit.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,4 @@ paths:
exclude_files: [".gz$"]

processors:
# Locale for timezone is only needed in non-json logs
- add_locale.when.not.regexp.message: "^{"
- add_locale: ~
Copy link
Contributor Author

@ycombinator ycombinator Jan 29, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We add this processor for JSON logs as well because some versions of ES (e.g. 7.1.1) generate JSON audit logs with the @timestamp field which does not contain a time zone.

Later versions of ES will generate JSON audit logs with a timestamp field which does contain the time zone. In such cases, we will remove the event.timezone field in the ingest pipeline to avoid confusion about which timezone is correct.

Related: #13918.

18 changes: 17 additions & 1 deletion filebeat/module/elasticsearch/audit/ingest/pipeline-json.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,23 @@
"remove": {
"field": "elasticsearch.audit.type",
"ignore_missing": true

}
},
{
"date": {
"if": "ctx.elasticsearch.audit['@timestamp'] != null && ctx.event.timezone != null",
"field": "elasticsearch.audit.@timestamp",
"target_field": "elasticsearch.audit.@timestamp",
"formats": [
"yyyy-MM-dd'T'HH:mm:ss,SSS"
],
"timezone": "{{ event.timezone }}"
}
},
{
"remove": {
"if": "ctx.elasticsearch.audit['@timestamp'] == null && ctx.event.timezone != null",
"field": "event.timezone"
}
},
{
Expand Down
3 changes: 3 additions & 0 deletions filebeat/module/elasticsearch/audit/test/test-audit-711.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{"@timestamp":"2019-09-05T14:02:37,921", "node.id":"UwRu4mReRtyJO1-FWAPvIQ", "event.type":"transport", "event.action":"authentication_success", "user.name":"_system", "origin.type":"local_node", "origin.address":"127.0.0.1:9300", "realm":"__fallback", "request.id":"474ZciqtQteOhjLO3OdZIw", "action":"indices:monitor/stats", "request.name":"IndicesStatsRequest"}
{"@timestamp":"2020-01-29T09:41:10,856", "node.id":"DJKjhISiTzy-JY5nCU8h3Q", "event.type":"transport", "event.action":"access_granted", "user.name":"_xpack_security", "user.realm":"__attach", "user.roles":["superuser"], "origin.type":"local_node", "origin.address":"127.0.0.1:9300", "request.id":"I9bQCw28Qfe4HWtIJHgoAg", "action":"cluster:admin/xpack/security/realm/cache/clear", "request.name":"ClearRealmCacheRequest"}
{"@timestamp":"2020-01-29T09:41:10,859", "node.id":"DJKjhISiTzy-JY5nCU8h3Q", "event.type":"transport", "event.action":"access_granted", "user.name":"_xpack_security", "user.realm":"__attach", "user.roles":["superuser"], "origin.type":"local_node", "origin.address":"127.0.0.1:9300", "request.id":"I9bQCw28Qfe4HWtIJHgoAg", "action":"cluster:admin/xpack/security/realm/cache/clear[n]", "request.name":"Node"}
77 changes: 77 additions & 0 deletions filebeat/module/elasticsearch/audit/test/test-audit-711.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
[
{
"@timestamp": "2019-09-05T16:02:37.921Z",
"elasticsearch.audit.action": "indices:monitor/stats",
"elasticsearch.audit.layer": "transport",
"elasticsearch.audit.origin.type": "local_node",
"elasticsearch.audit.realm": "__fallback",
"elasticsearch.audit.request.id": "474ZciqtQteOhjLO3OdZIw",
"elasticsearch.audit.request.name": "IndicesStatsRequest",
"elasticsearch.node.id": "UwRu4mReRtyJO1-FWAPvIQ",
"event.action": "authentication_success",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"input.type": "log",
"log.offset": 0,
"message": "{\"@timestamp\":\"2019-09-05T14:02:37,921\", \"node.id\":\"UwRu4mReRtyJO1-FWAPvIQ\", \"event.type\":\"transport\", \"event.action\":\"authentication_success\", \"user.name\":\"_system\", \"origin.type\":\"local_node\", \"origin.address\":\"127.0.0.1:9300\", \"realm\":\"__fallback\", \"request.id\":\"474ZciqtQteOhjLO3OdZIw\", \"action\":\"indices:monitor/stats\", \"request.name\":\"IndicesStatsRequest\"}",
"service.type": "elasticsearch",
"source.address": "127.0.0.1:9300",
"source.ip": "127.0.0.1",
"source.port": 9300,
"user.name": "_system"
},
{
"@timestamp": "2020-01-29T11:41:10.856Z",
"elasticsearch.audit.action": "cluster:admin/xpack/security/realm/cache/clear",
"elasticsearch.audit.layer": "transport",
"elasticsearch.audit.origin.type": "local_node",
"elasticsearch.audit.request.id": "I9bQCw28Qfe4HWtIJHgoAg",
"elasticsearch.audit.request.name": "ClearRealmCacheRequest",
"elasticsearch.audit.user.realm": "__attach",
"elasticsearch.audit.user.roles": [
"superuser"
],
"elasticsearch.node.id": "DJKjhISiTzy-JY5nCU8h3Q",
"event.action": "access_granted",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"input.type": "log",
"log.offset": 363,
"message": "{\"@timestamp\":\"2020-01-29T09:41:10,856\", \"node.id\":\"DJKjhISiTzy-JY5nCU8h3Q\", \"event.type\":\"transport\", \"event.action\":\"access_granted\", \"user.name\":\"_xpack_security\", \"user.realm\":\"__attach\", \"user.roles\":[\"superuser\"], \"origin.type\":\"local_node\", \"origin.address\":\"127.0.0.1:9300\", \"request.id\":\"I9bQCw28Qfe4HWtIJHgoAg\", \"action\":\"cluster:admin/xpack/security/realm/cache/clear\", \"request.name\":\"ClearRealmCacheRequest\"}",
"service.type": "elasticsearch",
"source.address": "127.0.0.1:9300",
"source.ip": "127.0.0.1",
"source.port": 9300,
"user.name": "_xpack_security"
},
{
"@timestamp": "2020-01-29T11:41:10.859Z",
"elasticsearch.audit.action": "cluster:admin/xpack/security/realm/cache/clear[n]",
"elasticsearch.audit.layer": "transport",
"elasticsearch.audit.origin.type": "local_node",
"elasticsearch.audit.request.id": "I9bQCw28Qfe4HWtIJHgoAg",
"elasticsearch.audit.request.name": "Node",
"elasticsearch.audit.user.realm": "__attach",
"elasticsearch.audit.user.roles": [
"superuser"
],
"elasticsearch.node.id": "DJKjhISiTzy-JY5nCU8h3Q",
"event.action": "access_granted",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"input.type": "log",
"log.offset": 785,
"message": "{\"@timestamp\":\"2020-01-29T09:41:10,859\", \"node.id\":\"DJKjhISiTzy-JY5nCU8h3Q\", \"event.type\":\"transport\", \"event.action\":\"access_granted\", \"user.name\":\"_xpack_security\", \"user.realm\":\"__attach\", \"user.roles\":[\"superuser\"], \"origin.type\":\"local_node\", \"origin.address\":\"127.0.0.1:9300\", \"request.id\":\"I9bQCw28Qfe4HWtIJHgoAg\", \"action\":\"cluster:admin/xpack/security/realm/cache/clear[n]\", \"request.name\":\"Node\"}",
"service.type": "elasticsearch",
"source.address": "127.0.0.1:9300",
"source.ip": "127.0.0.1",
"source.port": 9300,
"user.name": "_xpack_security"
}
]
21 changes: 14 additions & 7 deletions filebeat/module/elasticsearch/audit/test/test-audit.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
[
{
"@timestamp": "2018-10-31T09:34:25.109Z",
"@timestamp": "2018-10-31T11:34:25.109Z",
"elasticsearch.audit.layer": "rest",
"elasticsearch.audit.origin.type": "rest",
"elasticsearch.node.id": "DSiWcTyeThWtUXLB9J0BMw",
"event.action": "authentication_failed",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"input.type": "log",
"log.offset": 0,
Expand All @@ -19,13 +20,14 @@
"user.name": "elastic"
},
{
"@timestamp": "2018-10-31T09:34:25.207Z",
"@timestamp": "2018-10-31T11:34:25.207Z",
"elasticsearch.audit.layer": "rest",
"elasticsearch.audit.origin.type": "rest",
"elasticsearch.node.id": "DSiWcTyeThWtUXLB9J0BMw",
"event.action": "authentication_failed",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"input.type": "log",
"log.offset": 274,
Expand All @@ -38,7 +40,7 @@
"user.name": "elastic"
},
{
"@timestamp": "2018-10-31T09:35:11.428Z",
"@timestamp": "2018-10-31T11:35:11.428Z",
"elasticsearch.audit.action": "cluster:admin/xpack/security/realm/cache/clear",
"elasticsearch.audit.layer": "transport",
"elasticsearch.audit.origin.type": "local_node",
Expand All @@ -51,6 +53,7 @@
"event.action": "access_granted",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"input.type": "log",
"log.offset": 558,
Expand All @@ -62,7 +65,7 @@
"user.name": "_xpack_security"
},
{
"@timestamp": "2018-10-31T09:35:11.430Z",
"@timestamp": "2018-10-31T11:35:11.430Z",
"elasticsearch.audit.action": "cluster:admin/xpack/security/realm/cache/clear[n]",
"elasticsearch.audit.layer": "transport",
"elasticsearch.audit.origin.type": "local_node",
Expand All @@ -75,6 +78,7 @@
"event.action": "access_granted",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"input.type": "log",
"log.offset": 941,
Expand All @@ -86,7 +90,7 @@
"user.name": "_xpack_security"
},
{
"@timestamp": "2018-10-31T09:35:12.303Z",
"@timestamp": "2018-10-31T11:35:12.303Z",
"elasticsearch.audit.action": "cluster:admin/xpack/security/user/change_password",
"elasticsearch.audit.layer": "transport",
"elasticsearch.audit.origin.type": "rest",
Expand All @@ -99,6 +103,7 @@
"event.action": "access_granted",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"input.type": "log",
"log.offset": 1309,
Expand All @@ -110,7 +115,7 @@
"user.name": "elastic"
},
{
"@timestamp": "2018-10-31T09:35:12.314Z",
"@timestamp": "2018-10-31T11:35:12.314Z",
"elasticsearch.audit.action": "indices:admin/create",
"elasticsearch.audit.indices": [
".security-6"
Expand All @@ -126,6 +131,7 @@
"event.action": "access_granted",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"input.type": "log",
"log.offset": 1676,
Expand All @@ -137,7 +143,7 @@
"user.name": "_xpack_security"
},
{
"@timestamp": "2019-01-27T20:15:10.380Z",
"@timestamp": "2019-01-27T22:15:10.380Z",
"elasticsearch.audit.layer": "rest",
"elasticsearch.audit.origin.type": "rest",
"elasticsearch.audit.realm": "default_file",
Expand All @@ -147,6 +153,7 @@
"event.action": "authentication_success",
"event.dataset": "elasticsearch.audit",
"event.module": "elasticsearch",
"event.timezone": "-02:00",
"fileset.name": "audit",
"http.request.body.content": "\n{\n \"query\" : {\n \"term\" : { \"user\" : \"kimchy\" }\n }\n}\n",
"http.request.method": "GET",
Expand Down