[Security Solution][Detection Engine] Adds threat matching to the rule creator #78955

FrankHassanabad · 2020-09-30T13:54:02Z

Summary

This adds threat matching rule type to the rule creator.

Screen shot of creating a threat match

Screen shot of the description after creating one

Screen shot of first creating a threat match without values filled out

Additions and bug fixes:

Changes the threat index to be an array
Adds a threat_language to the REST schema so that we can use KQL, Lucene, (others in the future)
Adds plumbing for threat_list to work with the other REST endpoints such as PUT, PATCH, etc...
Adds the AND, OR dialog and user interface

Usage
If you are a team member using the team servers you can skip this usage section of creating threat index. Otherwise if you want to know how to create a mock threat index, instructions are below.

Go to the folder:

/kibana/x-pack/plugins/security_solution/server/lib/detection_engine/scripts

And post a small ECS threat mapping to the index called mock-threat-list:

./create_threat_mapping.sh

Then to post a small number of threats that represent simple port numbers you can run:

./create_threat_data.sh

However, feel free to also manually create them directly in your dev tools like so:

# Posts a threat list item called some-name with an IP but change these out for valid data in your system
PUT mock-threat-list-1/_doc/9999
{
  "@timestamp": "2020-09-09T20:30:45.725Z",
  "host": {
    "name": "some-name",
    "ip": "127.0.0.1"
  }
}

# Posts a destination port number to watch
PUT mock-threat-list-1/_doc/10000
{
  "@timestamp": "2020-09-08T20:30:45.725Z",
  "destination": {
    "port": "443"
  }
}

# Posts a source port number to watch
PUT mock-threat-list-1/_doc/10001
{
  "@timestamp": "2020-09-08T20:30:45.725Z",
  "source": {
    "port": "443"
  }
}

Checklist

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Unit or functional tests were updated or added to match the most common scenarios
This was checked for keyboard-only and screenreader accessibility
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser
This was checked for cross-browser compatibility

elasticmachine · 2020-09-30T21:43:14Z

Pinging @elastic/siem (Team:SIEM)

x-pack/plugins/security_solution/common/detection_engine/schemas/types/threat_mapping.ts

spong · 2020-09-30T23:45:45Z

I'm thinking we should hide the other rule type cards when on the Edit Rule page since we can't currently switch rule types and it's a lot of real estate. This is the view when first clicking Edit Rule 😅

spong · 2020-10-01T14:05:17Z

Seeing the status get stuck in going to run when disabled:

And as we chatted about, looks like we're missing the running state (stays in going to run for duration of rule execution):

x-pack/plugins/security_solution/public/common/components/threat_match/and_badge.tsx

x-pack/plugins/security_solution/public/common/components/threat_match/entry_item.tsx

yctercero · 2020-10-01T16:45:07Z

x-pack/plugins/security_solution/common/detection_engine/schemas/types/threat_mapping.ts

-  )
+export const threatMap = t.exact(
+  t.type({
+    entries: threatMappingEntries,


Could this be made to be a non empty array? I just thought of that when I saw the containsEmptyItem helper. I remember having to do lots of checks like that with exceptions and finally just changed it so that it's required to not be empty since at the very least (in the UI) there's one item with empty values.

Let me add that to a potential follow up. I totally agree, just don't know about the time I have left with the other tasks but I will add it to my list as I will have to make one of those icky specific types and all the tests for it.

...ck/plugins/security_solution/public/common/components/threat_match/logic_buttons.stories.tsx

...ugins/security_solution/public/detections/components/rules/step_define_rule/translations.tsx

x-pack/plugins/security_solution/public/common/components/threat_match/helpers.tsx

spong · 2020-10-01T18:38:11Z

x-pack/plugins/security_solution/public/detections/components/rules/description_step/index.tsx

@@ -156,6 +159,7 @@ export const addFilterStateIfNotThere = (filters: Filter[]): Filter[] => {
  });
 };

+/* eslint complexity: ["error", 21]*/


2️⃣ 1️⃣ ! 📈

I was hoping you wouldn't see that one. :-) Yeah, someone needs to go in and refactor that. I kind of didn't want to add too much more additional things to this PR but we are accumulating more complexity debt and I left it as is for the next person. Probably me you know. hehe.

spong

Checked out, tested locally, and did a high level review of the code. So much goodness in this PR @FrankHassanabad!! Fantastic job exposing the backend threat matching features via this UI, and ++ for all the tests and re-use from the exceptions components.

Did quite a bit of testing and all error scenarios I could find are looking good. Good call on your last couple PR's focusing on bubbling up internal errors as they were extremely helpful in testing and watching where things fail when running out of resources. The one outstanding bug I was seeing was in reference to the last response getting stuck in going to run, but other than that most everything else looked 👍 and not worth holding up this PR.

May the Threat Matching begin! 🔥L🔥G🔥T🔥M🔥!🔥

yctercero

🚀 Focused on the code as I saw Garrett did a deep dive testing it. Had a few comments, but nothing to hold this up. This is really awesome and the code was so nice to read through!

kibanamachine · 2020-10-01T22:29:30Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 32783a3

Metrics [docs]

@kbn/optimizer bundle module count

id	before	after	diff
`securitySolution`	1978	1989	+11

async chunks size

id	before	after	diff
`securitySolution`	10.3MB	10.3MB	+48.3KB

page load bundle size

id	before	after	diff
`securitySolution`	583.6KB	585.0KB	+1.3KB

History

💚 Build #78873 succeeded 65d8bc0
💚 Build #78577 succeeded 84f94db
💚 Build #78539 succeeded ea66ac8
💚 Build #78528 succeeded a4fc2ad
💚 Build #78525 succeeded 6f27124

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@timestamp

…e creator (elastic#78955) ## Summary This adds threat matching rule type to the rule creator. Screen shot of creating a threat match <img width="1023" alt="Screen Shot 2020-09-30 at 3 31 09 PM" src="https://user-images.githubusercontent.com/1151048/94742158-791b1c00-0332-11eb-9d79-78ab431322f0.png"> --- Screen shot of the description after creating one <img width="1128" alt="Screen Shot 2020-09-30 at 3 29 32 PM" src="https://user-images.githubusercontent.com/1151048/94742203-8b955580-0332-11eb-837f-5b4383044a13.png"> --- Screen shot of first creating a threat match without values filled out <img width="1017" alt="Screen Shot 2020-09-30 at 3 27 29 PM" src="https://user-images.githubusercontent.com/1151048/94742222-95b75400-0332-11eb-9872-e7670e917941.png"> Additions and bug fixes: * Changes the threat index to be an array * Adds a threat_language to the REST schema so that we can use KQL, Lucene, (others in the future) * Adds plumbing for threat_list to work with the other REST endpoints such as PUT, PATCH, etc... * Adds the AND, OR dialog and user interface **Usage** If you are a team member using the team servers you can skip this usage section of creating threat index. Otherwise if you want to know how to create a mock threat index, instructions are below. Go to the folder: ```ts /kibana/x-pack/plugins/security_solution/server/lib/detection_engine/scripts ``` And post a small ECS threat mapping to the index called `mock-threat-list`: ```ts ./create_threat_mapping.sh ``` Then to post a small number of threats that represent simple port numbers you can run: ```ts ./create_threat_data.sh ``` However, feel free to also manually create them directly in your dev tools like so: ```ts # Posts a threat list item called some-name with an IP but change these out for valid data in your system PUT mock-threat-list-1/_doc/9999 { "@timestamp": "2020-09-09T20:30:45.725Z", "host": { "name": "some-name", "ip": "127.0.0.1" } } ``` ```ts # Posts a destination port number to watch PUT mock-threat-list-1/_doc/10000 { "@timestamp": "2020-09-08T20:30:45.725Z", "destination": { "port": "443" } } ``` ```ts # Posts a source port number to watch PUT mock-threat-list-1/_doc/10001 { "@timestamp": "2020-09-08T20:30:45.725Z", "source": { "port": "443" } } ``` ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist) - [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server) - [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers)

@timestamp

…e creator (#78955) (#79230) ## Summary This adds threat matching rule type to the rule creator. Screen shot of creating a threat match <img width="1023" alt="Screen Shot 2020-09-30 at 3 31 09 PM" src="https://user-images.githubusercontent.com/1151048/94742158-791b1c00-0332-11eb-9d79-78ab431322f0.png"> --- Screen shot of the description after creating one <img width="1128" alt="Screen Shot 2020-09-30 at 3 29 32 PM" src="https://user-images.githubusercontent.com/1151048/94742203-8b955580-0332-11eb-837f-5b4383044a13.png"> --- Screen shot of first creating a threat match without values filled out <img width="1017" alt="Screen Shot 2020-09-30 at 3 27 29 PM" src="https://user-images.githubusercontent.com/1151048/94742222-95b75400-0332-11eb-9872-e7670e917941.png"> Additions and bug fixes: * Changes the threat index to be an array * Adds a threat_language to the REST schema so that we can use KQL, Lucene, (others in the future) * Adds plumbing for threat_list to work with the other REST endpoints such as PUT, PATCH, etc... * Adds the AND, OR dialog and user interface **Usage** If you are a team member using the team servers you can skip this usage section of creating threat index. Otherwise if you want to know how to create a mock threat index, instructions are below. Go to the folder: ```ts /kibana/x-pack/plugins/security_solution/server/lib/detection_engine/scripts ``` And post a small ECS threat mapping to the index called `mock-threat-list`: ```ts ./create_threat_mapping.sh ``` Then to post a small number of threats that represent simple port numbers you can run: ```ts ./create_threat_data.sh ``` However, feel free to also manually create them directly in your dev tools like so: ```ts # Posts a threat list item called some-name with an IP but change these out for valid data in your system PUT mock-threat-list-1/_doc/9999 { "@timestamp": "2020-09-09T20:30:45.725Z", "host": { "name": "some-name", "ip": "127.0.0.1" } } ``` ```ts # Posts a destination port number to watch PUT mock-threat-list-1/_doc/10000 { "@timestamp": "2020-09-08T20:30:45.725Z", "destination": { "port": "443" } } ``` ```ts # Posts a source port number to watch PUT mock-threat-list-1/_doc/10001 { "@timestamp": "2020-09-08T20:30:45.725Z", "source": { "port": "443" } } ``` ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist) - [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server) - [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers)

elasticmachine · 2021-09-23T14:24:20Z

Pinging @elastic/security-solution (Team: SecuritySolution)

FrankHassanabad added 8 commits September 29, 2020 19:42

wip: initial threat_match ui and backend bug fixes

c957a2c

Merge branch 'master' into adds-threat-matching-ui

2a2d769

Fixed deprecation warning when running tests

26789ee

Fixes i18n strings that were duplicates

993d8c1

Added plumbing for the language field for threats and other misc fixes

97b884e

Removes TODO that is not needed

a8fbea5

Removed second TODO block that is not needed

6f27124

Added threat field label to the UI

a4fc2ad

FrankHassanabad self-assigned this Sep 30, 2020

FrankHassanabad changed the title ~~Adds threat matching ui~~ [Security Solution][Detection Engine] Adds threat matching to the rule creator Sep 30, 2020

FrankHassanabad marked this pull request as ready for review September 30, 2020 21:42

FrankHassanabad requested review from a team as code owners September 30, 2020 21:42

FrankHassanabad added release_note:enhancement v7.10.0 v7.11.0 Feature:Detection Rules Security Solution rules and Detection Engine Team:SIEM labels Sep 30, 2020

FrankHassanabad added 3 commits September 30, 2020 15:56

Updates the icon on the threat match rule

7abaef7

Merge branch 'master' into adds-threat-matching-ui

e6ead56

Set default for querying against the threat list to be *:*

ea66ac8

yctercero reviewed Sep 30, 2020

View reviewed changes

x-pack/plugins/security_solution/common/detection_engine/schemas/types/threat_mapping.ts Outdated Show resolved Hide resolved

Fixes validation issue with the threat matches

84f94db

Merge branch 'master' into adds-threat-matching-ui

409d2e7