Description
We've recently been reducing Endpoint's data volume and three approaches are worth documenting for users so they can understand how to turn them off (i.e. restore the old behavior) if they want. Can this please be documented on a new page, with the info below.
We need to link to this documentation from an 8.16 Kibana so the documentation needs to be finished before 8.16 ships. We also need to confirm all Endpoint work is completed in 8.16 and if not update the documentation.
1. Deduplicate network events.
Starting in 8.15, when repeated network connections are detected from the same process Endpoint will not produce network events for subsequent connections. There are two advanced options to disable this and restore the 8.14 and prior behavior.
[linux|mac|windows].advanced.events.deduplicate_network_events: This will completely disable deduplication.
[linux|mac|windows].advanced.events.deduplicate_network_events_below_bytes This will enable deduplication for connections below X bytes but disable it for connections above X bytes. (In other words, suppress repeated connections for small data transfers but always emit events for large transfers)
2. Minimize host.* fieldset in event documents
Starting in 8.16, Endpoint will only include a small subset of the data in the host.* fieldset in event documents. Full host.* information will still be included in documents written to the metrics-* index pattern and in Endpoint alerts.
Users should take note of how a lack of some host.* information may affect their event filters.
@brian-mckinney can you comment on this issue with the advanced option name to turn this off and restore the 8.15 and earlier behavior?
3. Merge process and network events
Starting in 8.16, Endpoint will merge process create/terminate (Windows) and fork/exec/end (macOS/Linux) events when possible. Effectively, for short lived processes only a single event will be emitted, containing the process details from when the process terminated.
Starting in 8.16, Endpoint will merge network connection/termination (Windows/macOS/Linux) for when possible for short lived connections.
Users should take note of how this merging might affect their event filters. Notably, for merged events event.action will be an array containing all actions merged into the single event (e.g. event.action=[fork, exec, end]. For instance, if a user has an event filter to drop all fork events (event.action : fork) the filter will need to be modified or it'll also drop all merged events.
@nicholasberlin can you comment on this issue with the advanced options need to turn this off and restore the 8.15 and earlier behavior?
4. Not report MD5 and SHA1 hashes by default
As outlined in this Kibana PR elastic/kibana#193912 (it's been closed but only because it'll be merged via different PR) Endpoint will stop reporting MD5 and SHA1 hashes by default. These will still be reported if any Trusted Apps, Blocklist, Event Filters, or Alert Exceptions require them. In addition to lowering data volume this will reduce Endpoint's CPU.
The advanced options to restore the old behavior are described in the aforementioned PR.
cc @nfritts @caitlinbetz @dasansol92 @joe-desimone @gabriellandau @intxgo
### Tasks
- [x] Create stub page for in-product docs links (both ESS and serverless, but only ESS linked for now) — https://github.com/elastic/security-docs/pull/5800
- [ ] Add content: https://github.com/elastic/security-docs/pull/5881
- [ ] Add content to serverless too (maybe just bundle into one PR)