[8.7] Add new data stream for API event types #328

calladoum-elastic · 2022-12-09T00:59:16Z

Related issues:

https://github.com/elastic/security-team/issues/5094

Change Summary

This PR adds a new data stream for collecting API events.
It also adds the fields for credential access events using the API data stream.

Sample values

Sample document:

{
    "@timestamp": "2023-01-09T19:38:51.5141503Z",
    "Target": {
        "process": {
            "name": "lsass.exe",
            "pid": 956
        }
    },
    "event": {
        "category": [
            "api"
        ],
        "created": "2023-01-09T19:38:51.5141503Z",
        "id": "MvlgiHxIZtj1+Abi++++++ul",
        "kind": "event",
        "type": [
            "credential_access"
        ]
    },
    "host": {
        "architecture": "x86_64",
        "hostname": "DESKTOP-OCG8CR6",
        "id": "dabadaba-0000-0000-0000-000000000000",
        "ip": [
            "169.254.104.226",
            "fe80::6037:d589:2ee9:772f",
            "172.22.6.230",
            "fe80::80ea:8e8d:e1f2:6622",
            "169.254.74.35",
            "fe80::900c:a9d3:fd8:9345",
            "127.0.0.1",
            "::1"
        ],
        "mac": [
            "00:15:5d:00:09:18",
            "00:15:5d:00:09:19",
            "00:15:5d:00:09:17"
        ],
        "name": "DESKTOP-OCG8CR6",
        "os": {
            "Ext": {
                "variant": "Windows 11 Enterprise N"
            },
            "family": "windows",
            "full": "Windows 11 Enterprise N 22H2 (10.0.22621.963)",
            "kernel": "22H2 (10.0.22621.963)",
            "name": "Windows",
            "platform": "windows",
            "type": "windows",
            "version": "22H2 (10.0.22621.963)"
        }
    },
    "message": "Endpoint credential access event",
    "process": {
        "Ext": {
            "ancestry": [
                "YWFhYWFhYWEtYWFhYS1hYWFhLWFhYWEtYWFhYWFhYWFhYWFhLTgxMi0xNjczMjkxNTMzLjcxNjczMTgwMA==",
                "YWFhYWFhYWEtYWFhYS1hYWFhLWFhYWEtYWFhYWFhYWFhYWFhLTQ5NjAtMTY3Mjk2NTgzMC4yODc0OTYxMDA=",
                "YWFhYWFhYWEtYWFhYS1hYWFhLWFhYWEtYWFhYWFhYWFhYWFhLTY2MjgtMTY3Mjk2NTc3Mi41MTg4MDA1MDA=",
                "YWFhYWFhYWEtYWFhYS1hYWFhLWFhYWEtYWFhYWFhYWFhYWFhLTQ2MDAtMTY3Mjk2NTcxMy40NzY1NDUyMDA="
            ],
            "api": {
                "name": "OpenProcess",
                "parameters": {
                    "desired_access": [
                        "PROCESS_QUERY_LIMITED_INFORMATION",
                        "PROCESS_VM_READ"
                    ],
                    "desired_access_numeric": 4112,
                    "handle_type": "process"
                }
            }
        },
        "entity_id": "YWFhYWFhYWEtYWFhYS1hYWFhLWFhYWEtYWFhYWFhYWFhYWFhLTkxNi0xNjczMjkzMTMwLjk0NjAwMjAw",
        "executable": "c\\git\\endpoint-dev\\Tools\\Leia\\modules\\exe_malware\\mimikatz.exe",
        "name": "mimikatz.exe",
        "pid": 916,
        "thread": {
            "Ext": {
                "call_stack": [
                    {
                        "instruction_pointer": 140717353267908,
                        "module_path": "C:\\Windows\\System32\\ntdll.dll"
                    },
                    {
                        "instruction_pointer": 140717309764254,
                        "module_path": "C:\\Windows\\System32\\KernelBase.dll"
                    },
                    {
                        "instruction_pointer": 140697175487234,
                        "module_path": "c\\users\\user\\desktop\\mimikatz.exe"
                    },
                    {
                        "instruction_pointer": 140697175488197,
                        "module_path": "c\\users\\user\\desktop\\mimikatz.exe"
                    },
                    {
                        "instruction_pointer": 140697175487041,
                        "module_path": "c\\users\\user\\desktop\\mimikatz.exe"
                    },
                    {
                        "instruction_pointer": 140697175277748,
                        "module_path": "c\\users\\user\\desktop\\mimikatz.exe"
                    },
                    {
                        "instruction_pointer": 140697175277292,
                        "module_path": "c\\users\\user\\desktop\\mimikatz.exe"
                    },
                    {
                        "instruction_pointer": 140697175276599,
                        "module_path": "c\\users\\user\\desktop\\mimikatz.exe"
                    },
                    {
                        "instruction_pointer": 140697175514281,
                        "module_path": "c\\users\\user\\desktop\\mimikatz.exe"
                    }
                ],
                "call_stack_contains_unbacked": false,
                "call_stack_final_user_module": {
                    "path": "mimikatz.exe"
                }
            },
            "id": 11628
        }
    },
    "user": {
        "domain": "DESKTOP-OCG8CR6",
        "id": "S-1-5-21-3820246941-898183108-3095036578-1001",
        "name": "User"
    }
}

Release Target

8.7

Q/A

For mapping changes:

I ran make after making the schema changes, and committed all changes
If these field(s) are "exception"-able, I made a companion PR to Kibana adding it (see Readme)
If this is a metadata change, I also updated both transform destination schemas to match

For Transform changes:

The new transform successfully starts in Kibana
The corresponding transform destination schema was updated if necessary

elasticmachine · 2022-12-09T01:04:52Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-01-18T18:44:39.694+0000
Duration: 8 min 37 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

paul-tavares · 2023-01-11T14:03:24Z

Hi @pzl - could you take a look when you get a chance? 🙏

kevinlog · 2023-01-11T14:04:48Z

@paul-tavares @pzl - I've been working with @calladoum-elastic on this - I can do the review. I added myself to the reviewers

kevinlog · 2023-01-11T19:58:16Z

custom_schemas/custom_api_credential_access.yml

+  fields:
+    - name: handle_type
+      level: custom
+      type: keywork


a typo here, should be keyword

fixed by f355012

kevinlog · 2023-01-11T19:58:23Z

custom_schemas/custom_api_credential_access.yml

+
+    - name: desired_access
+      level: custom
+      type: keywork


same here, should be keyword

fixed by f355012

kevinlog · 2023-01-11T19:59:09Z

package/endpoint/manifest.yml

 version: 8.7.0-next
 categories: ["security", "cloud"]
 # The package type. The options for now are [integration, input], more type might be added in the future.
+


can you remove the whitespace from here so that we don't have a diff on the manifest?

reverted by 029ffaf

kevinlog · 2023-01-11T20:00:08Z

.bumpversion.cfg

 commit = True
 parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)(\-(?P<release>[a-z]+)\.(?P<build>\d+))?
-serialize = 
+serialize =


can you revert the whitespace here so that we don't have a diff in this file?

reverted by 6b58626

kevinlog · 2023-01-12T12:43:32Z

@calladoum-elastic thank you for the changes. This LGTM!

I also checked out the package and tried it in a stack.

The template is created:

Also, when I stream your example document, the datastream is also created:

calladoum-elastic · 2023-01-18T02:34:22Z

Hi!

Is it ok to merge this PR or should there be a 2nd review?

Thanks,

kevinlog · 2023-01-18T18:46:46Z

Is it ok to merge this PR or should there be a 2nd review?

@calladoum-elastic - it's OK to merge with one review, thanks!

* Define the schema for CredentialAccess events under `process.Ext.api` * Adds a generic schema for API * Describes the schema of parameters for `Credential_access` API events * Added missing files: manifest, ingest pipeline configuration and sample_Event * Regenerated yamls: fields and api

elasticmachine · 2023-02-08T22:31:53Z

Package endpoint - 8.7.0 containing this change is available at https://epr.elastic.co/search?package=endpoint

calladoum-elastic added 3 commits December 8, 2022 15:39

Define the schema for CredentialAccess events under process.Ext.api

7993663

Adds a generic schema for API

c4726ec

Describes the schema of parameters for Credential_access API events

24d0033

calladoum-elastic added 6 commits December 12, 2022 14:55

completed fields/fields.yml

604f63c

added manifest

775e3cf

added ingest pipeline conf

c10bfa3

added sample_event for credaccess

6a64e9c

adding missing files

d05ec21

linting

950e012

calladoum-elastic changed the title ~~[8.7] Add new API event type and Credential Access event schema~~ [8.7] Add new data stream for API event types Jan 10, 2023

calladoum-elastic added the v8.7.0 label Jan 10, 2023

calladoum-elastic added 5 commits January 10, 2023 11:24

adding api custom subset

a5b579d

api custom_subset: added call_stack

c1f8405

generated fields.yml for data_stream/api

99ae580

adding missing files

e12570c

fixed type declaration

b637ad3

calladoum-elastic marked this pull request as ready for review January 11, 2023 00:32

calladoum-elastic requested a review from a team as a code owner January 11, 2023 00:32

calladoum-elastic requested review from ashokaditya and paul-tavares January 11, 2023 00:32

paul-tavares requested review from pzl and removed request for paul-tavares January 11, 2023 14:03

kevinlog self-requested a review January 11, 2023 14:04

kevinlog reviewed Jan 11, 2023

View reviewed changes

calladoum-elastic added 5 commits January 11, 2023 14:45

applying PR feedback

f355012

reverting e12570c on .bumpversion.cfg

6b58626

reverting comment align in manifest

029ffaf

reverting last change from manifest

cd87f71

regenerated yamls: fields and api

3fc5f12

calladoum-elastic requested a review from kevinlog January 12, 2023 00:27

kevinlog approved these changes Jan 12, 2023

View reviewed changes

updated event.type and event.category for sample

4396615

calladoum-elastic merged commit 72cb6e8 into main Jan 18, 2023

calladoum-elastic deleted the credaccess_event_type_api_schema branch January 18, 2023 18:57

[8.7] Add new data stream for API event types #328

[8.7] Add new data stream for API event types #328

Uh oh!

Conversation

calladoum-elastic commented Dec 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Summary

Sample values

Release Target

Q/A

For mapping changes:

For Transform changes:

Uh oh!

elasticmachine commented Dec 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

🤖 GitHub comments

Uh oh!

paul-tavares commented Jan 11, 2023

Uh oh!

kevinlog commented Jan 11, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevinlog commented Jan 12, 2023

Uh oh!

calladoum-elastic commented Jan 18, 2023

Uh oh!

kevinlog commented Jan 18, 2023

Uh oh!

elasticmachine commented Feb 8, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

calladoum-elastic commented Dec 9, 2022 •

edited

Loading

elasticmachine commented Dec 9, 2022 •

edited

Loading