Refactor kubelet metricsets to share response from endpoint #25782

ChrsMark · 2021-05-19T09:46:27Z

What does this PR do?

Follow up of #25640. This PR changes how kubernetes module handle metricsets which collect metrics from kubelet's API and which share same target endpoint.
Metricsets affected:

system
pod
node
volume
container

Why is it important?

To improve the performance of the module by avoid fetching same content multiple times.

How to test this PR locally

Deploy Metricbeat on k8s with https://github.com/elastic/beats/blob/master/deploy/kubernetes/metricbeat-kubernetes.yaml using the proper docker image (ie a BC, snapshot etc). Modify the configmaps accordingly using the configs below.
state_* config using leaderelection:

metricbeat.autodiscover:
  providers:
    - type: kubernetes
      scope: cluster
      node: ${NODE_NAME}
      unique: true
      templates:
        - config:
            - module: kubernetes
              hosts: ["kube-state-metrics:8080"]
              period: 10s
              add_metadata: true
              metricsets:
                - state_node
                - state_deployment
                - state_daemonset
                - state_replicaset
                - state_pod
            - module: kubernetes
              hosts: ["kube-state-metrics:8080"]
              period: 10s
              add_metadata: true
              metricsets:
                - state_container
                - state_cronjob
                - state_service
                - state_resourcequota
                - state_statefulset

Kubelet's metricsets' config:

- module: kubernetes
  metricsets:
    - container
    - volume
  period: 10s
  host: ${NODE_NAME}
  hosts: ["https://${NODE_NAME}:10250"]
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  ssl.verification_mode: "none" 
- module: kubernetes
  metricsets:
    - node
    - system
    - pod
  period: 10s
  host: ${NODE_NAME}
  hosts: ["https://${NODE_NAME}:10250"]
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  ssl.verification_mode: "none"

Verify that all metricsets are being populated (like in the screenshot below) and that k8s metadata are properly attached on the events.

Related issues

Closes #24869

Screenshots

Signed-off-by: chrismark <chrismarkou92@gmail.com>

elasticmachine · 2021-05-19T10:03:08Z

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Started by user Chris Mark
Start Time: 2021-05-27T20:16:45.596+0000
Duration: 2 min 6 sec
Commit: 8cfa7ed

Trends 🧪

Steps errors

Expand to view the steps failures

`Google Storage Download`

Took 0 min 0 sec . View more details on here
Description: Exception while performing download

Log output

Expand to view the last 100 lines of log output

[2021-05-27T20:16:45.596Z] Started by user Chris Mark
[2021-05-27T20:16:45.596Z] Restarted from build #8, stage Packaging
[2021-05-27T20:16:45.665Z] Connecting to https://api.github.com using 72677/****** (Jenkins - beats-ci)
[2021-05-27T20:16:46.350Z] Connecting to https://api.github.com to check permissions of obtain list of ChrsMark for elastic/beats
[2021-05-27T20:16:46.873Z] Obtained Jenkinsfile from 8cfa7ed2bf21eecba362ab61953a368f948b7666+2ee21d95aef89af7f7e7aef8d07f679a24d690b4 (6c4bb2c85c17c89fba8e542fc55a372d726e9a6c)
[2021-05-27T20:16:46.981Z] Copying 17 artifact(s) from #8
[2021-05-27T20:16:47.170Z] Resume disabled by user, switching to high-performance, low-durability mode.
[2021-05-27T20:17:03.562Z] Still waiting to schedule task
[2021-05-27T20:17:03.562Z] All nodes of label ‘ubuntu-18&&immutable’ are offline
[2021-05-27T20:18:01.801Z] Running on beats-ci-immutable-ubuntu-1804-1622146619733136934 in /var/lib/jenkins/workspace/Beats_beats_PR-25782
[2021-05-27T20:18:01.959Z] �[39;49m[INFO] Override default checkout�[0m
[2021-05-27T20:18:02.054Z] Sleeping for 10 sec
[2021-05-27T20:18:12.248Z] The recommended git tool is: git
[2021-05-27T20:18:17.267Z] using credential f6c7695a-671e-4f4f-a331-acdce44ff9ba
[2021-05-27T20:18:17.274Z] Wiping out workspace first.
[2021-05-27T20:18:17.350Z] Cloning the remote Git repository
[2021-05-27T20:18:17.350Z] Using shallow clone with depth 10
[2021-05-27T20:18:17.350Z] Avoid fetching tags
[2021-05-27T20:18:17.371Z] Cloning repository git@github.com:elastic/beats.git
[2021-05-27T20:18:17.458Z]  > git init /var/lib/jenkins/workspace/Beats_beats_PR-25782 # timeout=10
[2021-05-27T20:18:17.528Z] Fetching upstream changes from git@github.com:elastic/beats.git
[2021-05-27T20:18:17.528Z]  > git --version # timeout=10
[2021-05-27T20:18:17.532Z]  > git --version # 'git version 2.17.1'
[2021-05-27T20:18:17.533Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-05-27T20:18:17.601Z]  > git fetch --no-tags --progress -- git@github.com:elastic/beats.git +refs/heads/*:refs/remotes/origin/* # timeout=15
[2021-05-27T20:18:39.433Z] Cleaning workspace
[2021-05-27T20:18:39.451Z] Using shallow fetch with depth 10
[2021-05-27T20:18:39.451Z] Pruning obsolete local branches
[2021-05-27T20:18:40.428Z] Merging remotes/origin/master commit 2ee21d95aef89af7f7e7aef8d07f679a24d690b4 into PR head commit 8cfa7ed2bf21eecba362ab61953a368f948b7666
[2021-05-27T20:18:39.407Z]  > git config remote.origin.url git@github.com:elastic/beats.git # timeout=10
[2021-05-27T20:18:39.417Z]  > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
[2021-05-27T20:18:39.426Z]  > git config remote.origin.url git@github.com:elastic/beats.git # timeout=10
[2021-05-27T20:18:39.435Z]  > git rev-parse --verify HEAD # timeout=10
[2021-05-27T20:18:39.440Z] No valid HEAD. Skipping the resetting
[2021-05-27T20:18:39.441Z]  > git clean -fdx # timeout=10
[2021-05-27T20:18:39.457Z] Fetching upstream changes from git@github.com:elastic/beats.git
[2021-05-27T20:18:39.457Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-05-27T20:18:39.462Z]  > git fetch --no-tags --progress --prune -- git@github.com:elastic/beats.git +refs/pull/25782/head:refs/remotes/origin/PR-25782 +refs/heads/master:refs/remotes/origin/master # timeout=15
[2021-05-27T20:18:40.483Z]  > git config core.sparsecheckout # timeout=10
[2021-05-27T20:18:40.487Z]  > git checkout -f 8cfa7ed2bf21eecba362ab61953a368f948b7666 # timeout=15
[2021-05-27T20:18:42.393Z] Merge succeeded, producing 99aa2f03ee0859c46e5ec37c04db68fcf9c24c1c
[2021-05-27T20:18:42.394Z] Checking out Revision 99aa2f03ee0859c46e5ec37c04db68fcf9c24c1c (PR-25782)
[2021-05-27T20:18:42.123Z]  > git remote # timeout=10
[2021-05-27T20:18:42.128Z]  > git config --get remote.origin.url # timeout=10
[2021-05-27T20:18:42.134Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-05-27T20:18:42.139Z]  > git merge 2ee21d95aef89af7f7e7aef8d07f679a24d690b4 # timeout=10
[2021-05-27T20:18:42.386Z]  > git rev-parse HEAD^{commit} # timeout=10
[2021-05-27T20:18:42.397Z]  > git config core.sparsecheckout # timeout=10
[2021-05-27T20:18:42.402Z]  > git checkout -f 99aa2f03ee0859c46e5ec37c04db68fcf9c24c1c # timeout=15
[2021-05-27T20:18:47.176Z] Commit message: "Merge commit '2ee21d95aef89af7f7e7aef8d07f679a24d690b4' into HEAD"
[2021-05-27T20:18:47.185Z] First time build. Skipping changelog.
[2021-05-27T20:18:47.185Z] Cleaning workspace
[2021-05-27T20:18:48.233Z] Timeout set to expire in 3 hr 0 min
[2021-05-27T20:18:48.273Z] The timestamps step is unnecessary when timestamps are enabled for all Pipeline builds.
[2021-05-27T20:18:48.369Z] Stage "Checkout" skipped due to this build restarting at stage "Packaging"
[2021-05-27T20:18:48.446Z] Stage "Lint" skipped due to this build restarting at stage "Packaging"
[2021-05-27T20:18:47.179Z]  > git rev-list --no-walk 60f99ea72217ccaedc72ce5b36627044a359c500 # timeout=10
[2021-05-27T20:18:47.187Z]  > git rev-parse --verify HEAD # timeout=10
[2021-05-27T20:18:47.191Z] Resetting working tree
[2021-05-27T20:18:47.192Z]  > git reset --hard # timeout=10
[2021-05-27T20:18:47.335Z]  > git clean -fdx # timeout=10
[2021-05-27T20:18:48.528Z] Stage "Build&Test" skipped due to this build restarting at stage "Packaging"
[2021-05-27T20:18:48.608Z] Stage "Extended" skipped due to this build restarting at stage "Packaging"
[2021-05-27T20:18:49.437Z] [INFO] unstashV2: JOB_GCS_BUCKET is set. bucket param got precedency instead.
[2021-05-27T20:18:49.475Z] [INFO] unstashV2: JOB_GCS_CREDENTIALS is set. credentialsId param got precedency instead.
[2021-05-27T20:18:49.639Z] Stage "Packaging-Pipeline" skipped due to earlier failure(s)
[2021-05-27T20:18:49.751Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-25782/src/github.com/elastic/beats
[2021-05-27T20:18:50.352Z] Running on worker-395930 in /var/lib/jenkins/workspace/Beats_beats_PR-25782
[2021-05-27T20:18:50.509Z] [INFO] getVaultSecret: Getting secrets
[2021-05-27T20:18:50.563Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-05-27T20:18:52.998Z] + chmod 755 generate-build-data.sh
[2021-05-27T20:18:52.998Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25782/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25782/runs/9 FAILURE 126002
[2021-05-27T20:18:52.998Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25782/runs/9/steps/?limit=10000 -o steps-info.json
[2021-05-27T20:18:54.447Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25782/runs/9/tests/?status=FAILED -o tests-errors.json
[2021-05-27T20:18:55.266Z] Retry 1/3 exited 22, retrying in 1 seconds...
[2021-05-27T20:18:56.713Z] Retry 2/3 exited 22, retrying in 2 seconds...

❕ Flaky test report

No test was executed to be analysed.

Signed-off-by: chrismark <chrismarkou92@gmail.com>

elasticmachine · 2021-05-20T09:34:21Z

Pinging @elastic/integrations (Team:Integrations)

Signed-off-by: chrismark <chrismarkou92@gmail.com>

jsoriano

Looks good, only some nitpicking.

CHANGELOG.next.asciidoc

jsoriano · 2021-05-27T12:28:29Z

metricbeat/module/kubernetes/kubernetes.go

 	GetSharedFamilies(prometheus p.Prometheus) ([]*dto.MetricFamily, error)
+	GetSharedKubeletStats(http *helper.HTTP) ([]byte, error)


Nit. I am thinking now that this being shared is an implementation detail, interface and consumers don't care if the response is shared or cached.

Suggested change

GetSharedFamilies(prometheus p.Prometheus) ([]*dto.MetricFamily, error)

GetSharedKubeletStats(http *helper.HTTP) ([]byte, error)

GetFamilies(prometheus p.Prometheus) ([]*dto.MetricFamily, error)

GetKubeletStats(http *helper.HTTP) ([]byte, error)

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

Signed-off-by: chrismark <chrismarkou92@gmail.com>

ChrsMark · 2021-05-27T14:45:27Z

/package

ChrsMark · 2021-05-28T07:46:05Z

Failing tests seem to be unrelated (hitting issue reported at #25956). Merging this one.

…25782) (cherry picked from commit a39dd00)

…25982) (cherry picked from commit a39dd00)

Refactor kubelet metricsets to share response from endpoint

a87b0a0

Signed-off-by: chrismark <chrismarkou92@gmail.com>

ChrsMark added Team:Integrations Label for the Integrations team v7.14.0 kubernetes Enable builds in the CI for kubernetes labels May 19, 2021

ChrsMark requested a review from jsoriano May 19, 2021 09:46

ChrsMark self-assigned this May 19, 2021

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels May 19, 2021

ChrsMark added 2 commits May 19, 2021 15:28

fmt

7342b54

Signed-off-by: chrismark <chrismarkou92@gmail.com>

fix locks

2d3d4a6

Signed-off-by: chrismark <chrismarkou92@gmail.com>

ChrsMark marked this pull request as ready for review May 20, 2021 09:34

ChrsMark added 3 commits May 20, 2021 13:36

fixup

7f3632a

Signed-off-by: chrismark <chrismarkou92@gmail.com>

Add changelog

c26d2c4

Signed-off-by: chrismark <chrismarkou92@gmail.com>

Fix import order

2b7606a

Signed-off-by: chrismark <chrismarkou92@gmail.com>

ChrsMark mentioned this pull request May 26, 2021

Add granularity on Kubernetes integration elastic/integrations#1017

Closed

jsoriano approved these changes May 27, 2021

View reviewed changes

ChrsMark and others added 2 commits May 27, 2021 15:53

Update CHANGELOG.next.asciidoc

8fabd99

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>

Improve method naming

8cfa7ed

Signed-off-by: chrismark <chrismarkou92@gmail.com>

ChrsMark merged commit a39dd00 into elastic:master May 28, 2021

ChrsMark added a commit to ChrsMark/beats that referenced this pull request May 28, 2021

Refactor kubelet metricsets to share response from endpoint (elastic#…

fc42acd

…25782) (cherry picked from commit a39dd00)

ChrsMark mentioned this pull request May 28, 2021

Cherry-pick #25782 to 7.x: Refactor kubelet metricsets to share response from endpoint #25982

Merged

ChrsMark added a commit that referenced this pull request May 28, 2021

Refactor kubelet metricsets to share response from endpoint (#25782) (#…

0ec351b

…25982) (cherry picked from commit a39dd00)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor kubelet metricsets to share response from endpoint #25782

Refactor kubelet metricsets to share response from endpoint #25782

ChrsMark commented May 19, 2021 •

edited

Loading

elasticmachine commented May 19, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Trends 🧪

`Google Storage Download`

elasticmachine commented May 20, 2021

jsoriano left a comment

jsoriano May 27, 2021

ChrsMark commented May 27, 2021

ChrsMark commented May 28, 2021

		GetSharedFamilies(prometheus p.Prometheus) ([]*dto.MetricFamily, error)
		GetSharedKubeletStats(http *helper.HTTP) ([]byte, error)

Refactor kubelet metricsets to share response from endpoint #25782

Refactor kubelet metricsets to share response from endpoint #25782

Conversation

ChrsMark commented May 19, 2021 • edited Loading

What does this PR do?

Why is it important?

How to test this PR locally

Related issues

Screenshots

elasticmachine commented May 19, 2021 • edited by jenkins-beats-ci bot Loading

💔 Build Failed

Build stats

Trends 🧪

Steps errors

Google Storage Download

Log output

❕ Flaky test report

elasticmachine commented May 20, 2021

jsoriano left a comment

Choose a reason for hiding this comment

jsoriano May 27, 2021

Choose a reason for hiding this comment

ChrsMark commented May 27, 2021

ChrsMark commented May 28, 2021

ChrsMark commented May 19, 2021 •

edited

Loading

elasticmachine commented May 19, 2021 •

edited by jenkins-beats-ci bot

Loading

`Google Storage Download`