Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Filebeat: Fix o365 module issues #18948

Merged
merged 11 commits into from
Jun 9, 2020
Merged

Filebeat: Fix o365 module issues #18948

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
0c1043b
Fix scary data-loss warning on startup
adriansr Jun 3, 2020
f8cd54c
Avoid API errors being processed by the JS pipeline
adriansr Jun 3, 2020
b459b69
Fix dissect error about overiding client.port
adriansr Jun 3, 2020
89edd25
Fix module passing API settings to the input
adriansr Jun 3, 2020
fc1bcd0
Add missing beta label to docs
adriansr Jun 3, 2020
d8e9218
Document max_period using the right units
adriansr Jun 3, 2020
41d87c1
Fix tests
adriansr Jun 3, 2020
bbe3bf5
Better templating of config
adriansr Jun 3, 2020
2d0e34f
Better has_fields condition
adriansr Jun 4, 2020
eb2df64
Changelog
adriansr Jun 8, 2020
cf48d92
Merge branch 'master' into fix_o365_issues
adriansr Jun 8, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Fix Kubernetes Watcher goroutine leaks when input config is invalid and `input.reload` is enabled. {issue}18629[18629] {pull}18630[18630]
- Okta module now sets the Elasticsearch `_id` field to the Okta UUID value contained in each system log to minimize the possibility of duplicating events. {pull}18953[18953]
- Fix improper nesting of session_issuer object in aws cloudtrail fileset. {issue}18894[18894] {pull}18915[18915]
- Fix `o365` module ignoring `var.api` settings. {pull}18948[18948]

*Heartbeat*

Expand Down
2 changes: 1 addition & 1 deletion x-pack/filebeat/filebeat.reference.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -852,7 +852,7 @@ filebeat.modules:
# authentication_endpoint: "https://login.microsoftonline.us/"
# resource: "https://manage.office365.us"
#
# max_retention: 7d
# max_retention: 168h
# max_requests_per_minute: 2000
# poll_interval: 3m

Expand Down
5 changes: 1 addition & 4 deletions x-pack/filebeat/input/o365audit/state.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,10 +114,7 @@ func (s *stateStorage) Load(key stream) (cursor, error) {
}
cur, err := s.persister.Load(key)
if err != nil {
if err != errStateNotFound {
return cur, err
}
cur = newCursor(key, time.Time{})
return newCursor(key, time.Time{}), err
}
return cur, s.saveUnsafe(cur)
}
Expand Down
8 changes: 2 additions & 6 deletions x-pack/filebeat/input/o365audit/state_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,18 +21,14 @@ func TestNoopState(t *testing.T) {
t.Run("new state", func(t *testing.T) {
st := newStateStorage(noopPersister{})
cur, err := st.Load(myStream)
if !assert.NoError(t, err) {
t.Fatal(err)
}
assert.Equal(t, errStateNotFound, err)
empty := newCursor(myStream, time.Time{})
assert.Equal(t, empty, cur)
})
t.Run("update state", func(t *testing.T) {
st := newStateStorage(noopPersister{})
cur, err := st.Load(myStream)
if !assert.NoError(t, err) {
t.Fatal(err)
}
assert.Equal(t, errStateNotFound, err)
advanced := cur.TryAdvance(content{
Type: tn,
ID: "1234",
Expand Down
2 changes: 1 addition & 1 deletion x-pack/filebeat/module/o365/_meta/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,6 @@
# authentication_endpoint: "https://login.microsoftonline.us/"
# resource: "https://manage.office365.us"
#
# max_retention: 7d
# max_retention: 168h
# max_requests_per_minute: 2000
# poll_interval: 3m
6 changes: 2 additions & 4 deletions x-pack/filebeat/module/o365/audit/config/input.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,7 @@ content_type:
{{ end }}
{{ end }}
{{ if .api }}
api:
{{ range $k, $v := .api }}
- {{ $k }}: {{ $v -}}
{{ end }}
api: {{.api | tojson }}
{{ end }}

{{ else if eq .input "file" }}
Expand Down Expand Up @@ -51,6 +48,7 @@ processors:
- 2006-01-02T15:04:05
{{ end }}
- script:
when.has_fields: ['o365audit']
lang: javascript
id: o365audit_script
file: ${path.home}/module/o365/audit/config/pipeline.js
Expand Down
5 changes: 4 additions & 1 deletion x-pack/filebeat/module/o365/audit/config/pipeline.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -729,7 +729,10 @@ function AuditProcessor(tenant_names, debug) {
tokenizer: '[%{_ip}]:%{port}',
field: 'client.address',
target_prefix: 'client',
'when.contains.client.address': ']:',
'when.and': [
{'not.has_fields': ['client._ip', 'client.port']},
{'contains.client.address': ']:'},
],
}));
builder.Add("extractClientIPv4Port", new processor.Dissect({
tokenizer: '%{_ip}:%{port}',
Expand Down
2 changes: 1 addition & 1 deletion x-pack/filebeat/modules.d/o365.yml.disabled
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,6 @@
# authentication_endpoint: "https://login.microsoftonline.us/"
# resource: "https://manage.office365.us"
#
# max_retention: 7d
# max_retention: 168h
# max_requests_per_minute: 2000
# poll_interval: 3m