Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade k8s.io/client-go and k8s keystore tests #18817

Merged
merged 14 commits into from
Jun 1, 2020

Conversation

ChrsMark
Copy link
Member

@ChrsMark ChrsMark commented May 28, 2020

What does this PR do?

This PR is a follow up for #18096.
Aims to add tests for k8s keystore internal functionality using a fake client for retrieving k8s secrets. An upgrade of k8s.io/client-go is needed in order to make use of the fake client (https://github.com/kubernetes/client-go/releases/tag/v0.18.3, https://pkg.go.dev/k8s.io/client-go@v0.18.3/kubernetes?tab=doc).

Upgrade procedure:

go get k8s.io/client-go@v0.18.3
mage vendor
make notice

Changes:

  1. Upgrade k8s.io/client-go
  2. Add tests for k8s keystore
  3. Improve k8s testing by adding a wait phase until kube-state-metrics in order to avoid flakiness

Why is it important?

In order to cover k8s keystore Retrieve functionality.

Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark ChrsMark added in progress Pull request is currently in progress. [zube]: In Progress Team:Platforms Label for the Integrations - Platforms team labels May 28, 2020
@ChrsMark ChrsMark self-assigned this May 28, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-platforms (Team:Platforms)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels May 28, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented May 28, 2020

❕ Build Aborted

There is a new build on-going so the previous on-going builds have been aborted.

Pipeline View Test View Changes Artifacts

Expand to view the summary

Build stats

  • Build Cause: [Pull request #18817 updated, Started by user Chris Mark, Rebuilds build #17]

  • Reason: Aborted from #19

  • Start Time: 2020-06-01T07:17:49.402+0000

  • Duration: 9 min 21 sec

  • Commit: 67e0f67

Log output

Expand to view the last 100 lines of log output

[2020-06-01T07:26:48.499Z] Stage "Heartbeat" skipped due to earlier failure(s)
[2020-06-01T07:26:48.501Z] Stage "Libbeat" skipped due to earlier failure(s)
[2020-06-01T07:26:48.502Z] Stage "Metricbeat x-pack" skipped due to earlier failure(s)
[2020-06-01T07:26:48.503Z] Stage "Packetbeat" skipped due to earlier failure(s)
[2020-06-01T07:26:48.504Z] Stage "dockerlogbeat" skipped due to earlier failure(s)
[2020-06-01T07:26:48.505Z] Stage "Winlogbeat" skipped due to earlier failure(s)
[2020-06-01T07:26:48.506Z] Stage "Functionbeat" skipped due to earlier failure(s)
[2020-06-01T07:26:48.506Z] Stage "Journalbeat" skipped due to earlier failure(s)
[2020-06-01T07:26:48.507Z] Stage "Generators" skipped due to earlier failure(s)
[2020-06-01T07:26:48.960Z] Failed in branch Elastic Agent x-pack
[2020-06-01T07:26:48.961Z] Failed in branch Elastic Agent x-pack Windows
[2020-06-01T07:26:48.962Z] Failed in branch Elastic Agent Mac OS X
[2020-06-01T07:26:48.962Z] Failed in branch Filebeat oss
[2020-06-01T07:26:48.963Z] Failed in branch Filebeat x-pack
[2020-06-01T07:26:48.964Z] Failed in branch Filebeat Mac OS X
[2020-06-01T07:26:48.964Z] Failed in branch Filebeat x-pack Mac OS X
[2020-06-01T07:26:48.965Z] Failed in branch Filebeat Windows
[2020-06-01T07:26:48.965Z] Failed in branch Filebeat x-pack Windows
[2020-06-01T07:26:48.966Z] Failed in branch Auditbeat oss Linux
[2020-06-01T07:26:48.966Z] Failed in branch Auditbeat crosscompile
[2020-06-01T07:26:48.967Z] Failed in branch Auditbeat oss Mac OS X
[2020-06-01T07:26:48.968Z] Failed in branch Auditbeat oss Windows
[2020-06-01T07:26:48.968Z] Failed in branch Auditbeat x-pack
[2020-06-01T07:26:48.969Z] Failed in branch Auditbeat x-pack Mac OS X
[2020-06-01T07:26:48.969Z] Failed in branch Auditbeat x-pack Windows
[2020-06-01T07:26:48.970Z] Failed in branch Libbeat x-pack
[2020-06-01T07:26:48.970Z] Failed in branch Metricbeat OSS Unit tests
[2020-06-01T07:26:48.971Z] Failed in branch Metricbeat OSS Integration tests
[2020-06-01T07:26:48.972Z] Failed in branch Metricbeat Python integration tests
[2020-06-01T07:26:48.972Z] Failed in branch Metricbeat crosscompile
[2020-06-01T07:26:48.973Z] Failed in branch Metricbeat Mac OS X
[2020-06-01T07:26:48.973Z] Failed in branch Metricbeat x-pack Mac OS X
[2020-06-01T07:26:48.974Z] Failed in branch Metricbeat Windows
[2020-06-01T07:26:48.974Z] Failed in branch Metricbeat x-pack Windows
[2020-06-01T07:26:48.975Z] Failed in branch Winlogbeat Windows x-pack
[2020-06-01T07:26:48.976Z] Failed in branch Kubernetes
[2020-06-01T07:26:49.215Z] Stage "Heartbeat" skipped due to earlier failure(s)
[2020-06-01T07:26:49.216Z] Stage "Libbeat" skipped due to earlier failure(s)
[2020-06-01T07:26:49.217Z] Stage "Metricbeat x-pack" skipped due to earlier failure(s)
[2020-06-01T07:26:49.218Z] Stage "Winlogbeat" skipped due to earlier failure(s)
[2020-06-01T07:26:49.219Z] Stage "Functionbeat" skipped due to earlier failure(s)
[2020-06-01T07:26:49.221Z] Stage "Generators" skipped due to earlier failure(s)
[2020-06-01T07:26:49.273Z] Failed in branch Packetbeat
[2020-06-01T07:26:49.274Z] Failed in branch dockerlogbeat
[2020-06-01T07:26:49.274Z] Failed in branch Journalbeat
[2020-06-01T07:26:49.452Z] Stage "Heartbeat" skipped due to earlier failure(s)
[2020-06-01T07:26:49.453Z] Stage "Libbeat" skipped due to earlier failure(s)
[2020-06-01T07:26:49.454Z] Stage "Functionbeat" skipped due to earlier failure(s)
[2020-06-01T07:26:49.456Z] Stage "Generators" skipped due to earlier failure(s)
[2020-06-01T07:26:49.500Z] Failed in branch Metricbeat x-pack
[2020-06-01T07:26:49.501Z] Failed in branch Winlogbeat
[2020-06-01T07:26:49.669Z] Failed in branch Heartbeat
[2020-06-01T07:26:49.670Z] Failed in branch Libbeat
[2020-06-01T07:26:49.670Z] Failed in branch Functionbeat
[2020-06-01T07:26:49.671Z] Stage "Generators" skipped due to earlier failure(s)
[2020-06-01T07:26:49.738Z] Failed in branch Generators
[2020-06-01T07:26:49.850Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18817/src/github.com/elastic/beats
[2020-06-01T07:26:50.152Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-06-01T07:26:50.164Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18817/src/github.com/elastic/beats/Lint
[2020-06-01T07:26:50.527Z] + cat
[2020-06-01T07:26:50.527Z] + /usr/local/bin/runbld ./runbld-script
[2020-06-01T07:26:50.527Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-06-01T07:26:57.121Z] runbld>>> runbld started
[2020-06-01T07:26:57.121Z] runbld>>> 1.6.11/a66728ff8f4356963772e6e6d2069392fa06acbe
[2020-06-01T07:26:58.197Z] Click here to forcibly terminate running steps
[2020-06-01T07:26:59.039Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-18817' in order of occurrence in the config (last value wins).
[2020-06-01T07:26:59.982Z] runbld>>> Debug logging enabled.
[2020-06-01T07:26:59.982Z] runbld>>> Storing result
[2020-06-01T07:27:00.245Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-06-01T07:27:00.245Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200601072659-AF28ED7E
[2020-06-01T07:27:00.245Z] runbld>>> Adding system facts.
[2020-06-01T07:27:01.192Z] runbld>>> Adding vcs info for the latest commit:  67e0f672d86252a7bf378cb5daaa89c8c93da511
[2020-06-01T07:27:01.192Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-06-01T07:27:01.192Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-06-01T07:27:01.192Z] Processing JUnit reports with runbld...
[2020-06-01T07:27:01.192Z] + echo 'Processing JUnit reports with runbld...'
[2020-06-01T07:27:01.765Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-06-01T07:27:01.765Z] runbld>>> DURATION: 23ms
[2020-06-01T07:27:01.765Z] runbld>>> STDOUT: 40 bytes
[2020-06-01T07:27:01.765Z] runbld>>> STDERR: 49 bytes
[2020-06-01T07:27:01.765Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-06-01T07:27:01.765Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18817/src/github.com/elastic/beats
[2020-06-01T07:27:02.710Z] runbld>>> Storing build metadata: 
[2020-06-01T07:27:02.710Z] runbld>>> Adding test report.
[2020-06-01T07:27:02.710Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18817/src/github.com/elastic/beats
[2020-06-01T07:27:03.653Z] runbld>>> Found 0 test output files
[2020-06-01T07:27:03.653Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 0 Skipped: 0
[2020-06-01T07:27:03.654Z] runbld>>> Storing result
[2020-06-01T07:27:03.914Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-06-01T07:27:03.914Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200601072659-AF28ED7E
[2020-06-01T07:27:03.914Z] runbld>>> Email notification disabled by environment variable.
[2020-06-01T07:27:03.914Z] runbld>>> Slack notification disabled by environment variable.
[2020-06-01T07:27:09.438Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18817
[2020-06-01T07:27:09.540Z] [INFO] getVaultSecret: Getting secrets
[2020-06-01T07:27:09.585Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-06-01T07:27:10.363Z] + chmod 755 generate-build-data.sh
[2020-06-01T07:27:10.363Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18817/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18817/runs/18 ABORTED 560700
[2020-06-01T07:27:10.363Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18817/runs/18/steps/?limit=10000 -o steps-info.json
[2020-06-01T07:27:10.613Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18817/runs/18/tests/?status=FAILED -o tests-errors.json
[2020-06-01T07:27:10.613Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18817/runs/18/log/ -o pipeline-log.txt

Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark ChrsMark removed the in progress Pull request is currently in progress. label May 28, 2020
@ChrsMark ChrsMark requested review from a team May 28, 2020 15:01
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Copy link
Contributor

@exekias exekias left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Vendor update looks good but we need to solve the license detection, also make sure the project is compatible

@@ -434,6 +434,7 @@ def detect_license_summary(content):
"MPL-2.0",
"UPL-1.0",
"ISC",
"UNKNOWN",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we can accept these, did you dig in what license is this project using?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did check and json-patch has a BSD3 license (which is accepted), but detection is failing, we need to adjust the script to detect it

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 looking into it!

Copy link
Member Author

@ChrsMark ChrsMark May 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@exekias I'm afraid that their licence is not correct:
In https://github.com/evanphx/json-patch/blob/17efcbe3533fdf65efac5054bddbee809b4b4848/LICENSE#L9 there is a comma missing after notice.

Cross checked with https://opensource.org/licenses/BSD-3-Clause.

Tried with a sample repo of mine and Github seems to give a proper template for this (with the comma included) hence they might missed it somehow. Shall we mention it to them and/or a PR to fix it?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not, this looks like a minor typo, a PR sounds like a good idea.

In order to avoid blocking this one I wonder if we should update our script to accept that or wait a little bit to see it fixed

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR opened. I prefer to have it fixed there properly instead of changing our script :)

Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking good!

I have added some suggestions about the use of contexts in this PR, in general:

  • context.Background() should be used, quoting docs: "by the main function, initialization, and tests, and as the top-level Context for incoming requests."
  • context.TODO() should be used "when it's unclear which Context to use or it is not yet available (because the surrounding function has not yet been extended to accept a Context parameter)."
  • Functions having multiple calls that require a context should use the same context (or contexts based on the same one), so it is easier to refactor in the future when a context is provided.

dev-tools/mage/kubernetes/kuberemote.go Outdated Show resolved Hide resolved
dev-tools/mage/kubernetes/kuberemote.go Outdated Show resolved Hide resolved
Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark
Copy link
Member Author

ChrsMark commented May 29, 2020

Thanks for the suggestions @jsoriano I was thoughtful about this context thing too and you shed light :).

Regarding #18817 (comment), do you see a way of leveraging this in the scope of this PR?

Also tested in general(metricsets, autodiscover, metadata) k8s functionality manually and didn't notice anything suspicious 🤞 .

@jsoriano
Copy link
Member

Thanks for addressing all the comments!

Regarding #18817 (comment), do you see a way of leveraging this in the scope of this PR?

No, better to do this in future PRs.

Do you plan to wait for evanphx/json-patch#105 before merging this? I think it would be better to add another pattern to the script so this license is identified.
Take into account that if not, you will have to wait for the merge of the fix in upstream and for a version of client-go with the fix (json-patch is in client-go go.mod https://github.com/kubernetes/client-go/blob/v0.18.3/go.mod#L12).

Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark
Copy link
Member Author

Regarding #18817 (comment), do you see a way of leveraging this in the scope of this PR?

No, better to do this in future PRs.

👍

Do you plan to wait for evanphx/json-patch#105 before merging this? I think it would be better to add another pattern to the script so this license is identified.
Take into account that if not, you will have to wait for the merge of the fix in upstream and for a version of client-go with the fix (json-patch is in client-go go.mod https://github.com/kubernetes/client-go/blob/v0.18.3/go.mod#L12).

Well that's true, I hadn't predicted all this versioning overhead 😁 . Pushed a temp workaround so as to cover this case in our extractor script :).

@ChrsMark ChrsMark requested a review from exekias May 29, 2020 14:15
@ChrsMark ChrsMark added needs_backport PR is waiting to be backported to other branches. v7.9.0 labels May 29, 2020
@ChrsMark ChrsMark merged commit 5bfcc9c into elastic:master Jun 1, 2020
ChrsMark added a commit to ChrsMark/beats that referenced this pull request Jun 1, 2020
@ChrsMark ChrsMark removed the needs_backport PR is waiting to be backported to other branches. label Jun 1, 2020
ChrsMark added a commit that referenced this pull request Jun 1, 2020
v1v added a commit to v1v/beats that referenced this pull request Jun 2, 2020
…-stage-level

* upstream/master: (30 commits)
  Add a GRPC listener service for Agent (elastic#18827)
  Disable host.* fields by default for iptables module (elastic#18756)
  [WIP] Clarify capabilities of the Filebeat auditd module (elastic#17068)
  fix: rename file and remove extra separator (elastic#18881)
  ci: enable JJBB (elastic#18812)
  Disable host.* fields by default for Checkpoint module (elastic#18754)
  Disable host.* fields by default for Cisco module (elastic#18753)
  Update latest.yml testing env to 7.7.0 (elastic#18535)
  Upgrade k8s.io/client-go and k8s keystore tests (elastic#18817)
  Add missing Jenkins stages for Auditbeat (elastic#18835)
  [Elastic Log Driver] Create a config shim between libbeat and the user (elastic#18605)
  Use indexers and matchers in config when defaults are enabled (elastic#18818)
  Fix panic on `metricbeat test modules` (elastic#18797)
  [CI] Fix permissions in MacOSX agents (elastic#18847)
  [Ingest Manager] When not port are specified and the https is used fallback to 443 (elastic#18844)
  [Ingest Manager] Fix install service script for windows (elastic#18814)
  [Metricbeat] Fix getting compute instance metadata with partial zone/region config (elastic#18757)
  Improve error messages in s3 input (elastic#18824)
  Add memory metrics into compute googlecloud (elastic#18802)
  include bucket name when logging error (elastic#18679)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
review Team:Platforms Label for the Integrations - Platforms team v7.9.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants