Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat][New Module] Adding support for Microsoft Defender ATP #19197

Merged
merged 22 commits into from
Jul 14, 2020
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
4d3ebf8
atp MVP push
P1llus Jun 12, 2020
27a60ba
stashing changes for later
P1llus Jun 12, 2020
86e8f72
Initial MVP for Defender ATP
P1llus Jun 15, 2020
f344d42
Updating default config
P1llus Jun 15, 2020
443ac5d
some updates, still not complete before more testing with new httpjso…
P1llus Jun 18, 2020
0296140
added new things to the pipeline, sorted them and commented them for …
P1llus Jun 20, 2020
24c655f
updated ingest pipeline with more ECS fields and evidence fields
P1llus Jun 25, 2020
cb01b5e
updated pipelines to fix yml indentations and added date_cursor funct…
P1llus Jun 30, 2020
6be6205
mage fmt update
P1llus Jun 30, 2020
cbf15f4
applying some new changes for date cursor and rebasing
P1llus Jul 4, 2020
8d6b316
Cleaning up pipelines, adding testdata, updating config fields, makin…
P1llus Jul 5, 2020
b46e849
changes to generate golden files
leehinman Jul 6, 2020
31ace16
adding requested changes based on PR comments, fixing some field type…
P1llus Jul 7, 2020
e7132bc
adding null checks for evidence subfield
P1llus Jul 7, 2020
37eeb56
updating painless scripts
P1llus Jul 7, 2020
db32b3a
mage fmt update
P1llus Jul 9, 2020
fc70654
one last styling change, will need to revert the event.test.message o…
P1llus Jul 9, 2020
a23be78
update golden files
P1llus Jul 9, 2020
fcb1082
updating tests to ignore timestamp and event.ingested
P1llus Jul 9, 2020
00b9f63
temp removing the default text for dashboard, dashboard will be inclu…
P1llus Jul 10, 2020
a4e762a
Merge branch 'master' into filebeat_atp_module_mvp
P1llus Jul 14, 2020
937975b
updating docs due to merge with microsoft dhcp
P1llus Jul 14, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
one last styling change, will need to revert the event.test.message o…
…nce done. Updating field docs and running mage fmt update
  • Loading branch information
P1llus committed Jul 13, 2020
commit fc70654c8a0c22b55e10778b5c6fc6c8bfcc1130
6 changes: 3 additions & 3 deletions filebeat/docs/fields.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -25904,7 +25904,7 @@ Module for ingesting Microsoft Defender ATP.
The date and time (in UTC) the alert was last updated.


type: keyword
type: date

--

Expand All @@ -25914,7 +25914,7 @@ type: keyword
The date and time in which the status of the alert was changed to 'Resolved'.


type: keyword
type: date

--

Expand Down Expand Up @@ -26024,7 +26024,7 @@ type: keyword
IP address involved in the alert


type: keyword
type: ip

--

Expand Down
77 changes: 10 additions & 67 deletions x-pack/filebeat/module/microsoft/defender_atp/ingest/pipeline.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ processors:
- json.comments
- host
ignore_missing: true

#########################
## ECS General Mapping ##
#########################
Expand Down Expand Up @@ -43,8 +44,8 @@ processors:
value: azure
- set:
field: '@timestamp'
value: '{{json.alertCreationTime}}'
if: ctx.json?.alertCreationTime != null
value: '{{json.alertUpdateTime}}'
if: ctx.json?.alertUpdateTime != null
- rename:
field: json.aadTenantId
target_field: cloud.account.id
Expand Down Expand Up @@ -183,6 +184,7 @@ processors:
field: json.evidence.filePath
target_field: file.path
ignore_missing: true

######################
## ECS Process Mapping ##
######################
Expand All @@ -206,6 +208,7 @@ processors:
field: json.evidence.parentProcessCreationTime
target_field: process.parent.start
ignore_missing: true

##########################
## ECS Observer Mapping ##
##########################
Expand All @@ -219,6 +222,7 @@ processors:
field: json.detectionSource
target_field: observer.name
ignore_missing: true

#####################
## ECS URL Mapping ##
#####################
Expand All @@ -227,6 +231,7 @@ processors:
target_field: url.full
ignore_missing: true
if: ctx?.json?.evidence?.url != null

######################
## ECS Host Mapping ##
######################
Expand All @@ -238,6 +243,7 @@ processors:
field: host.name
value: '{{host.hostname}}'
if: ctx?.host?.hostname != null

######################
## ECS User Mapping ##
######################
Expand All @@ -253,6 +259,7 @@ processors:
field: json.evidence.userSid
target_field: host.user.id
ignore_missing: true

#########################
## ECS Related Mapping ##
#########################
Expand All @@ -272,74 +279,10 @@ processors:
field: related.hash
value: '{{file.hash.sha256}}'
if: ctx.file?.hash?.sha256 != null

#############
## Cleanup ##
#############
- remove:
field:
- json.assignedTo
ignore_missing: true
if: ctx.json?.assignedTo == null
- remove:
field:
- json.classification
ignore_missing: true
if: ctx.json?.classification == null
- remove:
field:
- json.determination
ignore_missing: true
if: ctx.json?.determination == null
- remove:
field:
- json.evidence.aadUserId
ignore_missing: true
if: ctx.json?.evidence?.aadUserId == null
- remove:
field:
- json.evidence.domainName
ignore_missing: true
if: ctx.json?.evidence?.domainName == null
- remove:
field:
- json.evidence.userPrincipalName
ignore_missing: true
if: ctx.json?.evidence?.userPrincipalName == null
- remove:
field:
- json.investigationId
ignore_missing: true
if: ctx.json?.investigationId == null
- remove:
field:
- json.rbacGroupName
ignore_missing: true
if: ctx.json?.rbacGroupName == null
- remove:
field:
- json.resolvedTime
ignore_missing: true
if: ctx.json?.resolvedTime == null
- remove:
field:
- json.threatFamilyName
ignore_missing: true
if: ctx.json?.threatFamilyName == null
- remove:
field:
- json.evidence.ipAddress
ignore_missing: true
if: ctx.json?.evidence?.ipAddress == null
- remove:
field:
- json.evidence.accountName
ignore_missing: true
if: ctx.json?.evidence?.accountName == null
- remove:
field:
- json.evidence.url
ignore_missing: true
if: ctx.json?.evidence?.url == null
- remove:
field:
- json.alertCreationTime
Expand Down