Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat] Add support for Cloudtrail digest files #21086

Merged
merged 3 commits into from
Sep 15, 2020

Conversation

leehinman
Copy link
Contributor

@leehinman leehinman commented Sep 14, 2020

What does this PR do?

Adds support for Cloudtrail Digest Logs and Cloudtrail Insight logs.
Also adds options to ignore cloudtrail, cloudtrail-digest or
cloudtrail-insight logs if necessary.

Why is it important?

  • Cloudtrail digest files are used to determine integrity of Cloudtrail
    logs.
  • Users need the ability to ignore the digest or insight logs if necessary.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

TESTING_FILEBEAT_MODULES=aws TESTING_FILEBEAT_FILESETS=cloudtrail mage -v pythonIntegTest

Related issues

- allow file matching with file_selectors in s3 input
- update cloudtrail pipeline
- update cloudtrail config to use file_selectors
- add cloudtrail digest fields

Closes elastic#20943
@leehinman leehinman added bug Filebeat Filebeat needs_backport PR is waiting to be backported to other branches. Team:SIEM labels Sep 14, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 14, 2020
type FileSelectorCfg struct {
RegexString string `config:"regex"`
Regex *regexp.Regexp `config:",ignore"`
ExpandEventListFromField string `config:"expand_event_list_from_field"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good! One question: what happens if file_selectors are specified and also the global expand_event_list_from_field is given, which one takes priority? Maybe we should clarify this in the documentation 😄

Copy link
Contributor

@kaiyan-sheng kaiyan-sheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this! It looks good overall besides we are missing a changelog entry here.

@elasticmachine
Copy link
Collaborator

elasticmachine commented Sep 14, 2020

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #21086 updated]

  • Start Time: 2020-09-14T23:37:56.940+0000

  • Duration: 53 min 49 sec

Test stats 🧪

Test Results
Failed 1
Passed 5593
Skipped 825
Total 6419

Test errors

Expand to view the tests failures

  • Name: Build and Test / Filebeat oss / test_default_settings – filebeat.tests.system.test_autodiscover.TestAutodiscover

    • Age: 4
    • Duration: 90.004
    • Error Details: Failed: Timeout >90.0s

Steps errors

Expand to view the steps failures

  • Name: Mage build test
    • Description: mage build test

    • Duration: 26 min 58 sec

    • Start Time: 2020-09-15T00:02:21.887+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-09-15T00:29:18.578Z]  Git commit:        48a66213fe
[2020-09-15T00:29:18.578Z]  Built:             Mon Jun 22 15:45:36 2020
[2020-09-15T00:29:18.578Z]  OS/Arch:           linux/amd64
[2020-09-15T00:29:18.578Z]  Experimental:      false
[2020-09-15T00:29:18.578Z] 
[2020-09-15T00:29:18.578Z] Server: Docker Engine - Community
[2020-09-15T00:29:18.578Z]  Engine:
[2020-09-15T00:29:18.578Z]   Version:          19.03.12
[2020-09-15T00:29:18.578Z]   API version:      1.40 (minimum version 1.12)
[2020-09-15T00:29:18.578Z]   Go version:       go1.13.10
[2020-09-15T00:29:18.578Z]   Git commit:       48a66213fe
[2020-09-15T00:29:18.578Z]   Built:            Mon Jun 22 15:44:07 2020
[2020-09-15T00:29:18.578Z]   OS/Arch:          linux/amd64
[2020-09-15T00:29:18.578Z]   Experimental:     false
[2020-09-15T00:29:18.578Z]  containerd:
[2020-09-15T00:29:18.578Z]   Version:          1.2.13
[2020-09-15T00:29:18.578Z]   GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
[2020-09-15T00:29:18.578Z]  runc:
[2020-09-15T00:29:18.578Z]   Version:          1.0.0-rc10
[2020-09-15T00:29:18.578Z]   GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
[2020-09-15T00:29:18.578Z]  docker-init:
[2020-09-15T00:29:18.578Z]   Version:          0.18.0
[2020-09-15T00:29:18.578Z]   GitCommit:        fec3683
[2020-09-15T00:29:18.578Z] Unable to find image 'alpine:3.4' locally
[2020-09-15T00:29:19.153Z] 3.4: Pulling from library/alpine
[2020-09-15T00:29:19.153Z] c1e54eec4b57: Pulling fs layer
[2020-09-15T00:29:19.417Z] c1e54eec4b57: Download complete
[2020-09-15T00:29:19.681Z] c1e54eec4b57: Pull complete
[2020-09-15T00:29:19.681Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2020-09-15T00:29:19.681Z] Status: Downloaded newer image for alpine:3.4
[2020-09-15T00:29:21.914Z] + python .ci/scripts/pre_archive_test.py
[2020-09-15T00:29:23.309Z] Copy ./x-pack/filebeat/build into build/x-pack/filebeat/build
[2020-09-15T00:29:23.322Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21086/src/github.com/elastic/beats/build
[2020-09-15T00:29:23.348Z] Recording test results
[2020-09-15T00:29:25.376Z] Stashed 4 file(s)
[2020-09-15T00:29:25.388Z] Archiving artifacts
[2020-09-15T00:29:26.173Z] + python .ci/scripts/search_system_tests.py
[2020-09-15T00:29:26.192Z] [INFO] system-tests='build/x-pack/filebeat/build/system-tests'. If no empty then let's create a tarball
[2020-09-15T00:29:26.532Z] + tar --version
[2020-09-15T00:29:26.862Z] + tar --exclude=x-pack-filebeat--system-tests-linux.tgz -czf x-pack-filebeat--system-tests-linux.tgz build/x-pack/filebeat/build/system-tests
[2020-09-15T00:29:41.841Z] Archiving artifacts
[2020-09-15T00:30:01.348Z] [INFO] unstashV2: JOB_GCS_BUCKET is set. bucket param got precedency instead.
[2020-09-15T00:30:01.371Z] [INFO] unstashV2: JOB_GCS_CREDENTIALS is set. credentialsId param got precedency instead.
[2020-09-15T00:30:01.443Z] [Google Cloud Storage Plugin] Found 1 files to download from pattern: gs://beats-ci-temp/Beats/beats/PR-21086-3/source/source.tgz
[2020-09-15T00:30:01.463Z] [Google Cloud Storage Plugin] Downloading: Beats/beats/PR-21086-3/source/source.tgz to local path: /var/lib/jenkins/workspace/Beats_beats_PR-21086/source.tgz
[2020-09-15T00:30:10.804Z] + tar --version
[2020-09-15T00:30:11.108Z] + tar -xpf source.tgz
[2020-09-15T00:30:23.679Z] + rm source.tgz
[2020-09-15T00:30:23.692Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21086/src/github.com/elastic/beats
[2020-09-15T00:30:23.714Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21086/src/github.com/elastic/beats/Lint
[2020-09-15T00:30:23.799Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21086/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-09-15T00:30:23.880Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21086/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-09-15T00:30:23.958Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21086/src/github.com/elastic/beats/Filebeat-Windows
[2020-09-15T00:30:24.040Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21086/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-09-15T00:30:24.120Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21086/src/github.com/elastic/beats/Filebeat-oss
[2020-09-15T00:30:24.199Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21086/src/github.com/elastic/beats/Filebeat-x-pack
[2020-09-15T00:30:24.611Z] + cat
[2020-09-15T00:30:24.611Z] + /usr/local/bin/runbld ./runbld-script --job-name elastic+beats+pull-request
[2020-09-15T00:30:24.611Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-09-15T00:30:31.210Z] runbld>>> runbld started
[2020-09-15T00:30:31.210Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-09-15T00:30:33.128Z] runbld>>> The following profiles matched the job 'elastic+beats+pull-request' in order of occurrence in the config (last value wins).
[2020-09-15T00:30:33.128Z] runbld>>> Matches in the system config:
[2020-09-15T00:30:33.128Z] runbld>>> - Matched ^elastic\+beats
[2020-09-15T00:30:33.128Z] runbld>>> - Matched ^elastic\+beats\+pull-request
[2020-09-15T00:30:34.070Z] runbld>>> Debug logging enabled.
[2020-09-15T00:30:34.070Z] runbld>>> Storing result
[2020-09-15T00:30:34.332Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-09-15T00:30:34.332Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20200915003033-6C2CD3C1
[2020-09-15T00:30:34.332Z] runbld>>> Adding system facts.
[2020-09-15T00:30:35.276Z] runbld>>> Adding vcs info for the latest commit:  daa9d04eae46d24208aa9464465b6b64c3b75c0b
[2020-09-15T00:30:35.276Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-09-15T00:30:35.276Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-09-15T00:30:35.276Z] + echo 'Processing JUnit reports with runbld...'
[2020-09-15T00:30:35.276Z] Processing JUnit reports with runbld...
[2020-09-15T00:30:35.848Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-09-15T00:30:35.848Z] runbld>>> DURATION: 20ms
[2020-09-15T00:30:35.848Z] runbld>>> STDOUT: 40 bytes
[2020-09-15T00:30:35.848Z] runbld>>> STDERR: 49 bytes
[2020-09-15T00:30:35.848Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-09-15T00:30:35.848Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-21086
[2020-09-15T00:30:36.792Z] runbld>>> Storing build metadata: 
[2020-09-15T00:30:36.792Z] runbld>>> Adding test report.
[2020-09-15T00:30:36.792Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-21086/src/github.com/elastic/beats
[2020-09-15T00:30:37.366Z] runbld>>> Found 16 test output files
[2020-09-15T00:30:39.285Z] runbld>>> Test output logs contained: Errors: 0 Failures: 1 Tests: 6419 Skipped: 799
[2020-09-15T00:30:39.285Z] runbld>>> Storing result
[2020-09-15T00:30:39.285Z] runbld>>> FAILURES: 1
[2020-09-15T00:30:39.546Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-09-15T00:30:39.546Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20200915003033-6C2CD3C1
[2020-09-15T00:30:39.807Z] runbld>>> Email notification disabled by environment variable.
[2020-09-15T00:30:39.808Z] runbld>>> Slack notification disabled by environment variable.
[2020-09-15T00:30:45.451Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-21086
[2020-09-15T00:30:45.560Z] [INFO] getVaultSecret: Getting secrets
[2020-09-15T00:30:45.636Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-09-15T00:30:46.387Z] + chmod 755 generate-build-data.sh
[2020-09-15T00:30:46.387Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21086/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21086/runs/3 FAILURE 3169186
[2020-09-15T00:30:46.387Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21086/runs/3/steps/?limit=10000 -o steps-info.json
[2020-09-15T00:30:46.938Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21086/runs/3/tests/?status=FAILED -o tests-errors.json
[2020-09-15T00:30:47.489Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21086/runs/3/log/ -o pipeline-log.txt

- Add changelog
- update docs
- add insight section
@leehinman
Copy link
Contributor Author

jenkins run tests

@leehinman leehinman merged commit c9f7a99 into elastic:master Sep 15, 2020
leehinman added a commit to leehinman/beats that referenced this pull request Sep 15, 2020
* Add support for Cloudtrail digest files

- allow file matching with file_selectors in s3 input
- update cloudtrail pipeline
- update cloudtrail config to use file_selectors
- add cloudtrail digest fields
- add cloudtrail insight fields

Closes elastic#20943

(cherry picked from commit c9f7a99)
@leehinman leehinman added v7.10.0 and removed needs_backport PR is waiting to be backported to other branches. labels Sep 15, 2020
leehinman added a commit to leehinman/beats that referenced this pull request Sep 15, 2020
* Add support for Cloudtrail digest files

- allow file matching with file_selectors in s3 input
- update cloudtrail pipeline
- update cloudtrail config to use file_selectors
- add cloudtrail digest fields
- add cloudtrail insight fields

Closes elastic#20943

(cherry picked from commit c9f7a99)
leehinman added a commit that referenced this pull request Sep 15, 2020
* Add support for Cloudtrail digest files

- allow file matching with file_selectors in s3 input
- update cloudtrail pipeline
- update cloudtrail config to use file_selectors
- add cloudtrail digest fields
- add cloudtrail insight fields

Closes #20943

(cherry picked from commit c9f7a99)
leehinman added a commit that referenced this pull request Sep 15, 2020
* Add support for Cloudtrail digest files

- allow file matching with file_selectors in s3 input
- update cloudtrail pipeline
- update cloudtrail config to use file_selectors
- add cloudtrail digest fields
- add cloudtrail insight fields

Closes #20943

(cherry picked from commit c9f7a99)
v1v added a commit to v1v/beats that referenced this pull request Sep 18, 2020
…ne-2.0

* upstream/master: (44 commits)
  Update users.asciidoc (elastic#20802) (elastic#21108)
  Fix docker provider builder. (elastic#21118)
  [Elastic Agent] Add docker composable dynamic provider. (elastic#20842)
  Add new modules/filesets from rsa2elk for 7.10 (elastic#20820)
  Fix broken links to external websites (elastic#21061)
  [docs] typo in the command line (elastic#20799)
  [Filebeat] add panos type and sub_type (elastic#20912)
  Move the `compute_vm_scalset` to  a light metricset and map the cloud metadata (elastic#21038)
  [Filebeat] Add support for Cloudtrail digest files (elastic#21086)
  Add metrics collection from cost explorer into aws/billing metricset (elastic#20527)
  Add vendoring to Google Cloud Functions again (elastic#21070)
  [Elastic Agent] Add fleet.host.id for sending to endpoint. (elastic#21042)
  Do not need Google credentials before using it (elastic#21072)
  [Filebeat][New Module] Zoom webhook module (elastic#20414)
  Add support for GMT timezone offset in decode_cef (elastic#20993)
  Filebeat: Fix random error on harvester close (elastic#21048)
  Add ingress controller dashboards (elastic#21052)
  Fix loggers in composable module. (elastic#21047)
  [Ingest Manager] Increase kibana client timeout to 5 minutes (elastic#21037)
  Add changelog. (elastic#21041)
  ...
@leehinman leehinman deleted the 20943_cloudtrail_digest branch October 5, 2020 19:14
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
…lastic#21089)

* Add support for Cloudtrail digest files

- allow file matching with file_selectors in s3 input
- update cloudtrail pipeline
- update cloudtrail config to use file_selectors
- add cloudtrail digest fields
- add cloudtrail insight fields

Closes elastic#20943

(cherry picked from commit 6e3cb57)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Filebeat] AWS cloudtrail module should handle Digest files
3 participants