Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Kerberos-aware Elasticsearch and integration test to ES output #18127

Merged
merged 6 commits into from
May 4, 2020

Conversation

kvch
Copy link
Contributor

@kvch kvch commented Apr 30, 2020

What does this PR do?

This PR adds an integration test to the Elasticsearch output to check Kerberos authentication.

Furthermore, it adds a new element to our testing environment, a Kerberos-aware Elasticsearch instance named elasticsearch_kerberos.elastic.

Why is it important?

To make sure Kerberos authentication is working.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
    - [ ] I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
    - [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@kvch kvch added needs_backport PR is waiting to be backported to other branches. [zube]: In Review Team:Services (Deprecated) Label for the former Integrations-Services team labels Apr 30, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-services (Team:Services)

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 30, 2020
Copy link
Contributor

@ycombinator ycombinator left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a couple of nits but otherwise LGTM. Thanks for adding this test! WFG.

@ycombinator ycombinator removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 30, 2020
@kvch kvch mentioned this pull request Apr 30, 2020
10 tasks
@elasticmachine
Copy link
Collaborator

elasticmachine commented Apr 30, 2020

💔 Build Failed

Pipeline View Test View Changes Artifacts preview stats

Expand to view the summary

Build stats

Test stats 🧪

Test Results
Failed 0
Passed 7583
Skipped 1205
Total 8788

Steps errors

Expand to view the steps failures

  • Name: Install Go 1.13.10

    • Description: .ci/scripts/install-go.sh

    • Result: FAILURE

    • Duration: 0 min 6 sec<

    • Start Time: 2020-05-04T10:58:59.585+0000

  • Name: Make -C libbeat testsuite

    • Description: make -C libbeat testsuite

    • Result: FAILURE

    • Duration: 6 min 38 sec<

    • Start Time: 2020-05-04T10:57:09.790+0000

  • Name: Make -C generator/_templates/metricbeat test

    • Description: make -C generator/_templates/metricbeat test

    • Result: FAILURE

    • Duration: 1 min 56 sec<

    • Start Time: 2020-05-04T10:57:14.395+0000

  • Name: Make -C generator/_templates/beat test

    • Description: make -C generator/_templates/beat test

    • Result: FAILURE

    • Duration: 1 min 53 sec<

    • Start Time: 2020-05-04T11:02:42.266+0000

  • Name: Make -C generator/_templates/metricbeat test

    • Description: make -C generator/_templates/metricbeat test

    • Result: FAILURE

    • Duration: 2 min 0 sec<

    • Start Time: 2020-05-04T11:05:28.583+0000

  • Name: Make -C generator/_templates/beat test

    • Description: make -C generator/_templates/beat test

    • Result: FAILURE

    • Duration: 3 min 43 sec<

    • Start Time: 2020-05-04T11:09:04.476+0000

Log output

Expand to view the last 100 lines of log output

[2020-05-04T11:45:41.421Z] + FILE=heartbeat/build/coverage/full.cov
[2020-05-04T11:45:41.421Z] + [ -f heartbeat/build/coverage/full.cov ]
[2020-05-04T11:45:41.421Z] + FILE=libbeat/build/coverage/full.cov
[2020-05-04T11:45:41.421Z] + [ -f libbeat/build/coverage/full.cov ]
[2020-05-04T11:45:41.421Z] + FILE=metricbeat/build/coverage/full.cov
[2020-05-04T11:45:41.421Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-05-04T11:45:41.421Z] + FILE=packetbeat/build/coverage/full.cov
[2020-05-04T11:45:41.421Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-05-04T11:45:41.421Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-05-04T11:45:41.421Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-05-04T11:45:41.421Z] + FILE=journalbeat/build/coverage/full.cov
[2020-05-04T11:45:41.421Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-05-04T11:45:42.902Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats
[2020-05-04T11:45:43.270Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-05-04T11:45:43.289Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Lint
[2020-05-04T11:45:43.464Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-05-04T11:45:43.625Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-05-04T11:45:43.808Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-05-04T11:45:43.966Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-05-04T11:45:44.113Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Winlogbeat-oss
[2020-05-04T11:45:44.323Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Libbeat-oss
[2020-05-04T11:45:44.546Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-05-04T11:45:44.803Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Dockerlogbeat
[2020-05-04T11:45:45.020Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-05-04T11:45:45.259Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Journalbeat-oss
[2020-05-04T11:45:45.536Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-05-04T11:45:45.820Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-05-04T11:45:46.015Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Generators-Metricbeat-Mac-OS-X
[2020-05-04T11:45:46.256Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Heartbeat-oss
[2020-05-04T11:45:46.491Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-05-04T11:45:46.637Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Libbeat-x-pack
[2020-05-04T11:45:46.864Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-05-04T11:45:47.064Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Filebeat-Windows
[2020-05-04T11:45:47.225Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Auditbeat-Linux
[2020-05-04T11:45:47.351Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-05-04T11:45:47.548Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Packetbeat-oss
[2020-05-04T11:45:47.706Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Generators-Beat-Mac-OS-X
[2020-05-04T11:45:47.869Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Functionbeat-Mac-OS-X-x-pack
[2020-05-04T11:45:48.003Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-05-04T11:45:48.100Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Heartbeat-Mac-OS-X
[2020-05-04T11:45:48.210Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Metricbeat-Windows
[2020-05-04T11:45:48.303Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-05-04T11:45:48.414Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-05-04T11:45:48.510Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Filebeat-x-pack
[2020-05-04T11:45:48.613Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Filebeat-oss
[2020-05-04T11:45:48.740Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Auditbeat-Mac-OS-X
[2020-05-04T11:45:48.877Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Metricbeat-Python-integration-tests
[2020-05-04T11:45:48.987Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests
[2020-05-04T11:45:49.099Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Heartbeat-Windows
[2020-05-04T11:45:49.215Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Functionbeat-Windows
[2020-05-04T11:45:49.329Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-05-04T11:45:49.446Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Auditbeat-Windows
[2020-05-04T11:45:49.851Z] + cat
[2020-05-04T11:45:49.851Z] + /usr/local/bin/runbld ./runbld-script
[2020-05-04T11:45:49.851Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-05-04T11:45:56.538Z] runbld>>> runbld started
[2020-05-04T11:45:56.538Z] runbld>>> 1.6.11/a66728ff8f4356963772e6e6d2069392fa06acbe
[2020-05-04T11:45:58.462Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-18127' in order of occurrence in the config (last value wins).
[2020-05-04T11:45:59.851Z] runbld>>> Debug logging enabled.
[2020-05-04T11:45:59.851Z] runbld>>> Storing result
[2020-05-04T11:45:59.851Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-05-04T11:45:59.851Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200504114559-01D855D9
[2020-05-04T11:45:59.851Z] runbld>>> Adding system facts.
[2020-05-04T11:46:00.803Z] runbld>>> Adding vcs info for the latest commit:  11f8cfba520888053dc890b587b4e207880d62c0
[2020-05-04T11:46:01.064Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-05-04T11:46:01.065Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-05-04T11:46:01.065Z] + echo 'Processing JUnit reports with runbld...'
[2020-05-04T11:46:01.065Z] Processing JUnit reports with runbld...
[2020-05-04T11:46:01.328Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-05-04T11:46:01.328Z] runbld>>> DURATION: 15ms
[2020-05-04T11:46:01.328Z] runbld>>> STDOUT: 40 bytes
[2020-05-04T11:46:01.328Z] runbld>>> STDERR: 49 bytes
[2020-05-04T11:46:01.328Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-05-04T11:46:01.328Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats
[2020-05-04T11:46:02.720Z] runbld>>> Storing build metadata: 
[2020-05-04T11:46:02.720Z] runbld>>> Adding test report.
[2020-05-04T11:46:02.720Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats
[2020-05-04T11:46:04.124Z] runbld>>> Found 101 test output files
[2020-05-04T11:46:04.696Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-05-04T11:46:04.956Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-05-04T11:46:04.956Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-05-04T11:46:04.956Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-05-04T11:46:04.956Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-05-04T11:46:04.956Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-cloudfoundry.xml
[2020-05-04T11:46:05.899Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-05-04T11:46:05.899Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-05-04T11:46:06.472Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 8638 Skipped: 1001
[2020-05-04T11:46:06.472Z] runbld>>> Storing result
[2020-05-04T11:46:06.472Z] runbld>>> FAILURES: 0
[2020-05-04T11:46:06.733Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-05-04T11:46:06.733Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200504114559-01D855D9
[2020-05-04T11:46:06.733Z] runbld>>> Email notification disabled by environment variable.
[2020-05-04T11:46:06.733Z] runbld>>> Slack notification disabled by environment variable.
[2020-05-04T11:46:12.410Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18127
[2020-05-04T11:46:12.552Z] [INFO] getVaultSecret: Getting secrets
[2020-05-04T11:46:12.623Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-05-04T11:46:13.481Z] + chmod 755 generate-build-data.sh
[2020-05-04T11:46:13.481Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18127/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18127/runs/8 FAILURE 6374488
[2020-05-04T11:46:14.032Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18127/runs/8/steps/?limit=10000 -o steps-info.json
[2020-05-04T11:46:14.583Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18127/runs/8/tests/?status=FAILED -o tests-errors.json

@kvch kvch force-pushed the testing-libbeat-kerberos branch from 78f336e to c4b90e3 Compare April 30, 2020 13:26
@kvch kvch merged commit 428ee72 into elastic:master May 4, 2020
kvch added a commit to kvch/beats that referenced this pull request May 4, 2020
…lastic#18127)

This PR adds an integration test to the Elasticsearch output to check Kerberos authentication.

Furthermore, it adds a new element to our testing environment, a Kerberos-aware Elasticsearch instance named `elasticsearch_kerberos.elastic`.
(cherry picked from commit 428ee72)
@kvch kvch added v7.8.0 and removed needs_backport PR is waiting to be backported to other branches. labels May 4, 2020
kvch added a commit that referenced this pull request May 4, 2020
…ration test to ES output (#18168)

* Add Kerberos-aware Elasticsearch and integration test to ES output (#18127)

This PR adds an integration test to the Elasticsearch output to check Kerberos authentication.

Furthermore, it adds a new element to our testing environment, a Kerberos-aware Elasticsearch instance named `elasticsearch_kerberos.elastic`.
(cherry picked from commit 428ee72)

* downgrade dockerfile version to 7.7
@andresrc
Copy link
Contributor

I think some AD environment would still be needed. I don't know if we would be able to use for automated testing but just for support scenarios I think it would be very useful. What options do we have here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Team:Services (Deprecated) Label for the former Integrations-Services team v7.8.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants