[Auditbeat] Backport #9546 to 6.x: Add system module #9581

cwurm · 2018-12-16T11:12:04Z

This is an early attempt at backporting the system module to 6.x. #9546 is not merged yet, but I did want to get and give an impression already of the scope of the backport, as well as get some early CI feedback.

This branch was created using:

git checkout feature-auditbeat-host
git format-patch -k --stdout master..feature-auditbeat-host > ~/backup/system_module.patch.2018-12-15

git checkout 6.x
git checkout -b backport_9546_6.x
git am -3 -k ~/backup/system_module.patch.2018-12-15

When running git am I had to resolve conflicts in maybe about 20-30 files, mostly under dev-tools/ when applying the changes from #9362 (Add CI testing to x-pack/auditbeat).

Since a number of the changes in the build system rely on other things already introduced in master, I had to backport (well, copy paste) those as well (c3f9d13, 0a6f5d2, 881b4f9).

This adds an skeleton x-pack module to Auditbeat. The module is only included in the Elastic licensed Auditbeat binary. The config and fields.yml data are not yet included in the packaging. Additional updates are required.

Adds host, packages, and processes metricsets to Auditbeat. Host collects general host information, e.g. boottime, timezone, OS, network interfaces. Packages collects information about installed packages. For now, it supports debian and homebrew on darwin. Processes collects information about currently running, started, and stopped processes.

Changes necessary to make `mage fields` and `make system-tests` work again after merging master. (And deleting a misplaced parenthesis.)

Collects (via C functions) user information from /etc/passwd, /etc/shadow, and /etc/group on Linux. Detects new users, deleted users, changes to users (e.g. groups), and - as a special distinct category - password changes. Sends periodic state information about all users (frequency can be controlled). Otherwise, periodically checks the ctime of the above files, reads them if the ctime has changed, detects changes compared to its internal cache, and reports any changes. The cache is persisted to disk in a `beat.db` file (already used by the `file_integrity` module) after every `Fetch` and on `Close`. It contains a copy of all current user information incl. a SHA-512 hash of the password hash from /etc/shadow (to detect password changes between Auditbeat restarts - this hash is not sent to any output).

Collects information about open sockets (Linux only). Uses netlink to query for all currently open sockets. Sends information about all sockets on start, and periodically as determined by `state.period`. Otherwise, sends only newly opened or closed sockets. The sockets are enriched with process and user information.

Adds additional build flags to the new user metricset to prevent build failures on non-Linux systems. If it is configured, it will now throw an error and abort the launch.

Updates the `process` metricset to follow newest conventions: - Rename from `processes` to `process` - Change to single documents instead of arrays - Scheduled state reporting - Use top-level ECS fields

* Refactor Auditbeat build logic Update auditbeat and x-pack/auditbeat to share logic for generating config and packages. This makes auditbeat and x-pack/auditbeat have independent `package` targets where auditbeat generates only OSS packages and x-pack/auditbeat generates Elastic licensed packages. And x-pack/auditbeat will now be tested on Travis CI. * Skip failing Auditbeat system module test * Skip failing system/process test * Add temporary target alias for Windows CI * Fix file permission issues caused by Docker usage * Optimize chown by checking if UID/GID need changed

Updates the `host` metricset to be in line with the other metricsets in the `system` module: 1. Adds regular state reporting based on `state.period`/`host.state.period` 2. Persists state between restarts in `beat.db` 3. Detects changes in host information 4. Changes to using `system.host.ip`/`system.host.mac` instead of `system.host.network.interfaces`

Introduces a `user.detect_password_changes` config parameter that defaults to true in the config, but false in the code. Only if it is set to true will the code read the password field in /etc/passwd and /etc/shadow to detect password changes. The read password field values are put through 10 round of SHA-512 hashing before being locally stored.

To be compatible with ECS, changes the `event.type` field to `event.kind` throughout the system module.

The `packages` metricset is not yet ready. This disables the metricset's code, tests, fields, and docs until we have time to finish the work.

Namespaces all Auditbeat system module metricsets to `system.audit` to avoid any potential field clashes with Metricbeat.

…stic#9500) Setting `auditbeat.max_start_delay: 0` for system tests greatly reduces their execution time. (cherry picked from commit 7a2cf0f)

This adds a top-level `message` field to the `host`, `process`, `socket`, and `user` metricsets.

Adds config and asciidoc documentation for the four metricsets of the system module that are ready today: host, process, socket, user. Also adjusts the doc generation to include files from x-pack/auditbeat.

Allow the `process` metricset to run as any user by catching permission errors when trying to read other user's private process information.

elasticmachine · 2018-12-16T11:12:06Z

Pinging @elastic/secops

x-pack/auditbeat/module/system/process/process.go

x-pack/auditbeat/module/system/host/_meta/fields.yml

x-pack/auditbeat/module/system/packages/_meta/fields.yml.disabled

webmat · 2018-12-17T19:02:39Z

Test failures are almost surely unrelated to the backport

https://travis-ci.org/elastic/beats/jobs/469138029 seems related to Central management tests are failling under 6.x / master #9597
https://travis-ci.org/elastic/beats/jobs/469138037 has been fixed in master here: Update version of python elasticsearch client to 6.3.1 #9557

webmat

I'm not the right person to review the changes in the build system introduced as part of this backport. I don't have knowledge about this.

I'd like to discuss the introduction of source in 6.x, however. See my comment below.

If we decide to proceed with introducing source in 6.x for Auditbeat, the only thing missing is the field definitions for it (only for Auditbeat).

webmat · 2018-12-17T19:16:09Z

x-pack/auditbeat/module/system/socket/_meta/data.json

+    "service": {
+        "type": "system"
+    },
+    "source": {


In Filebeat we had to use source_ecs instead of source, because there was a hard conflict. It used to be a string field and in 7.x/ECS becomes an object field.

Auditbeat doesn't use source in 6.x, so we could simply introduce it right now and be done with this. It affects only the Auditbeat index, so it may be fine. If we take this route, querying on a field in source will mean the user needs to query both source for Auditbeat and source_ecs for Filebeat.

So even if there's no actual conflict, there is an advantage in prioritizing consistency instead, and naming this field source_ecs in 6.x for Auditbeat, then migrating it for 7.x.

WDYT @ruflin @tsg ?

Personally I think it may be fine to introduce source here and not have to do a transition of this new field right away for 7.0. I don't think using Auditbeat and Filebeat will be a big enough use case to warrant this in the short period we will have this inconsistency. SecOps will only work on 7.x anyway.

As far as I can tell, this PR does not introduce the fields definition for source.*, so if we decide to go this direction, @cwurm you should ensure to add them, but only in the context of Auditbeat.

I'm ok with introducing source.* in auditbeat only. If we do it definitively should follow ECS.

+1 to using source. One less thing to migrate.

Auditbeat is already using source in 6.x, so we should be good. But there are a few other fields that I'll add in a commit momentarily.

@cwurm Gotcha.

These field definitions are not quite in line with ECS' source field set, however. There's no incompatibility between the two, but we're also missing field definitions for source.

andrewkroh · 2018-12-17T20:13:42Z

I tested the mage package targets for auditbeat and x-pack/auditbeat. That all worked.

cwurm · 2018-12-18T14:44:53Z

I've added some missing field definitions. Test failures look unrelated. Anything else that needs to be addressed, or is this good to merge?

webmat

LGTM

Some field definitions for source seem to be missing vs ECS, but I think that's out of scope for this backport.

webmat · 2018-12-18T16:54:19Z

The libbeat failure in Travis has been fixed yesterday in master and 6.x, I think. This is unrelated indeed.

andrewkroh and others added 26 commits December 15, 2018 22:43

Add skeleton x-pack Auditbeat module (elastic#8252)

ef247ed

This adds an skeleton x-pack module to Auditbeat. The module is only included in the Elastic licensed Auditbeat binary. The config and fields.yml data are not yet included in the packaging. Additional updates are required.

Rename sysinfo module to system (elastic#8356)

d22fb54

Fixes after merging master into feature branch.

627d520

Changes necessary to make `mage fields` and `make system-tests` work again after merging master. (And deleting a misplaced parenthesis.)

[Auditbeat] Disable user metricset on non-Linux systems (elastic#9368)

1e3f4bf

Adds additional build flags to the new user metricset to prevent build failures on non-Linux systems. If it is configured, it will now throw an error and abort the launch.

[Auditbeat] Update process metricset (elastic#9139)

f51d36c

Updates the `process` metricset to follow newest conventions: - Rename from `processes` to `process` - Change to single documents instead of arrays - Scheduled state reporting - Use top-level ECS fields

[Auditbeat] Change event.type to event.kind (elastic#9489)

f41a560

To be compatible with ECS, changes the `event.type` field to `event.kind` throughout the system module.

[Auditbeat] Disable packages metricset (elastic#9495)

381e779

The `packages` metricset is not yet ready. This disables the metricset's code, tests, fields, and docs until we have time to finish the work.

[Auditbeat] Namespace system module to system.audit (elastic#9499)

ac6fbec

Namespaces all Auditbeat system module metricsets to `system.audit` to avoid any potential field clashes with Metricbeat.

[Auditbeat] Set auditbeat.max_start_delay to 0 for system tests. (ela…

2b0f240

…stic#9500) Setting `auditbeat.max_start_delay: 0` for system tests greatly reduces their execution time. (cherry picked from commit 7a2cf0f)

[Auditbeat] Add message field to system module (elastic#9483)

18af477

This adds a top-level `message` field to the `host`, `process`, `socket`, and `user` metricsets.

[Auditbeat] System module documentation (elastic#9512)

b0e756b

Adds config and asciidoc documentation for the four metricsets of the system module that are ready today: host, process, socket, user. Also adjusts the doc generation to include files from x-pack/auditbeat.

[Auditbeat] Fix process metricset when not root (elastic#9497)

6f5f20f

Allow the `process` metricset to run as any user by catching permission errors when trying to read other user's private process information.

Re-generate configs to include ILM.

0f84d05

Re-add newlines to configs.

71362b3

Temporarily disable docs references to x-pack/auditbeat

5f48d60

Re-generate x-pack configs

072e632

Fix make beats-dashboards

8f49938

Fix make update

c3f9d13

Fix make commands in x-pack/auditbeat

0a6f5d2

Fix mage integtest

881b4f9

cwurm added in progress Pull request is currently in progress. backport Auditbeat SecOps labels Dec 16, 2018

houndci-bot reviewed Dec 16, 2018

View reviewed changes

x-pack/auditbeat/module/system/process/process.go Show resolved Hide resolved

Christoph Wurm added 3 commits December 16, 2018 14:34

Fix x-pack/auditbeat make testsuite

fff5977

Fix process test

a051656

Fix auditbeat/ make testsuite

bab903b

cwurm added review and removed in progress Pull request is currently in progress. labels Dec 17, 2018

cwurm mentioned this pull request Dec 17, 2018

[Auditbeat] Socket metricset #8834

Merged

ruflin reviewed Dec 17, 2018

View reviewed changes

x-pack/auditbeat/module/system/host/_meta/fields.yml Show resolved Hide resolved

ruflin reviewed Dec 17, 2018

View reviewed changes

x-pack/auditbeat/module/system/packages/_meta/fields.yml.disabled Outdated Show resolved Hide resolved

Christoph Wurm added 2 commits December 17, 2018 14:15

Remove disabled packages metricset.

0a43226

Add missing dependency.

86fd446

webmat reviewed Dec 17, 2018

View reviewed changes

Add ECS fields

ce12565

andrewkroh approved these changes Dec 18, 2018

View reviewed changes

webmat approved these changes Dec 18, 2018

View reviewed changes

Add to CHANGELOG

0924fdf

cwurm merged commit 0218a8c into elastic:6.x Dec 18, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Auditbeat] Backport #9546 to 6.x: Add system module #9581

[Auditbeat] Backport #9546 to 6.x: Add system module #9581

cwurm commented Dec 16, 2018

elasticmachine commented Dec 16, 2018

webmat commented Dec 17, 2018

webmat left a comment

webmat Dec 17, 2018

ruflin Dec 18, 2018

tsg Dec 18, 2018

cwurm Dec 18, 2018

webmat Dec 18, 2018

andrewkroh commented Dec 17, 2018

cwurm commented Dec 18, 2018

webmat left a comment

webmat commented Dec 18, 2018

[Auditbeat] Backport #9546 to 6.x: Add system module #9581

[Auditbeat] Backport #9546 to 6.x: Add system module #9581

Conversation

cwurm commented Dec 16, 2018

elasticmachine commented Dec 16, 2018

webmat commented Dec 17, 2018

webmat left a comment

Choose a reason for hiding this comment

webmat Dec 17, 2018

Choose a reason for hiding this comment

ruflin Dec 18, 2018

Choose a reason for hiding this comment

tsg Dec 18, 2018

Choose a reason for hiding this comment

cwurm Dec 18, 2018

Choose a reason for hiding this comment

webmat Dec 18, 2018

Choose a reason for hiding this comment

andrewkroh commented Dec 17, 2018

cwurm commented Dec 18, 2018

webmat left a comment

Choose a reason for hiding this comment

webmat commented Dec 18, 2018