Add kubevirt_controller integration (#18186)

* initial scaffolding

* report kubevirt_controller.can_connect metric

* report kubevirt_controller metrics

* add kube_cluster_name, kube_namespace, kube_pod_name tags to all metrics

* update spec.yaml and ddev validate config and models

* disable TLS verify

* update README

* refactor / cleanup

* ddev validate

* update manifest.json

* fix title and description

* delete dashboard file

* fix changelog

* add e2e tests

* Update README.md

* Apply suggestions from code review

Co-authored-by: Jen Gilbert <jen.gilbert@datadoghq.com>

* apply suggestions

* fix spec.yaml and sync config files

* Apply suggestions from code review

Co-authored-by: Jen Gilbert <jen.gilbert@datadoghq.com>

* ddev validate ci

* rename Kubevirt Controller -> KubeVirt Controller

* rename Kubevirt to KubeVirt

* remove time.sleeps from dd_environment setup

* remove kube_cluster_name

* remove kube_cluster_name from tests, cleanup check.py with the suggestions from other PR

* remove unecessary line

* remove IMAGES_README.md

* ddev validate ci

---------

Co-authored-by: Jen Gilbert <jen.gilbert@datadoghq.com>

.codecov.yml

@@ -310,6 +310,10 @@ coverage:
         target: 75
         flags:
         - kong
+      KubeVirt_Controller:
+        target: 75
+        flags:
+        - kubevirt_controller
       KubeVirt_API:
         target: 75
         flags:
@@ -1162,6 +1166,11 @@ flags:
     paths:
     - kubernetes_state/datadog_checks/kubernetes_state
     - kubernetes_state/tests
+  kubevirt_controller:
+    carryforward: true
+    paths:
+    - kubevirt_controller/datadog_checks/kubevirt_controller
+    - kubevirt_controller/tests
   kubevirt_api:
     carryforward: true
     paths:

.github/workflows/config/labeler.yml

@@ -291,6 +291,8 @@ integration/kubernetes_state:
 - kubernetes_state/**/*
 integration/kubernetes_state_core:
 - kubernetes_state_core/**/*
+integration/kubevirt_controller:
+- kubevirt_controller/**/*
 integration/kubevirt_api:
 - kubevirt_api/**/*
 integration/kyototycoon:

.github/workflows/test-all.yml

@@ -2194,6 +2194,26 @@ jobs:
       minimum-base-package: ${{ inputs.minimum-base-package }}
       pytest-args: ${{ inputs.pytest-args }}
     secrets: inherit
+  jc4c1032:
+    uses: ./.github/workflows/test-target.yml
+    with:
+      job-name: KubeVirt Controller
+      target: kubevirt_controller
+      platform: linux
+      runner: '["ubuntu-22.04"]'
+      repo: "${{ inputs.repo }}"
+      python-version: "${{ inputs.python-version }}"
+      standard: ${{ inputs.standard }}
+      latest: ${{ inputs.latest }}
+      agent-image: "${{ inputs.agent-image }}"
+      agent-image-py2: "${{ inputs.agent-image-py2 }}"
+      agent-image-windows: "${{ inputs.agent-image-windows }}"
+      agent-image-windows-py2: "${{ inputs.agent-image-windows-py2 }}"
+      test-py2: ${{ inputs.test-py2 }}
+      test-py3: ${{ inputs.test-py3 }}
+      minimum-base-package: ${{ inputs.minimum-base-package }}
+      pytest-args: ${{ inputs.pytest-args }}
+    secrets: inherit
   j61e565f:
     uses: ./.github/workflows/test-target.yml
     with:

kubevirt_controller/CHANGELOG.md

@@ -0,0 +1,4 @@
+# CHANGELOG - KubeVirt Controller
+
+<!-- towncrier release notes start -->
+

kubevirt_controller/README.md

@@ -0,0 +1,88 @@
+# Agent Check: KubeVirt Controller
+
+## Overview
+
+This check monitors [KubeVirt Controller][1] through the Datadog Agent.
+
+## Setup
+
+Follow the instructions below to install and configure this check for an Agent running on a host. For containerized environments, see the [Autodiscovery Integration Templates][3] for guidance on applying these instructions.
+
+### Installation
+
+The KubeVirt Controller check is included in the [Datadog Agent][2] package.
+No additional installation is needed on your server.
+
+### Configuration
+
+The main use case to run the `kubevirt_controller` check is as a [cluster level check][4].
+
+In order to do that, you will need to update some RBAC permissions to give the `datadog-agent` service account read-only access to the`KubeVirt` resources by following the steps below:
+
+1. Bind the `kubevirt.io:view` ClusterRole to the `datadog-agent` service account:
+
+```yaml
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: datadog-agent-kubevirt
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: kubevirt.io:view
+subjects:
+  - kind: ServiceAccount
+  name: datadog-agent
+  namespace: default
+```
+
+2. Annotate the pods template of your `virt-controller` deployment by patching the `KubeVirt` resource as follows:
+
+```yaml
+apiVersion: kubevirt.io/v1
+kind: KubeVirt
+metadata:
+  name: kubevirt
+  namespace: kubevirt
+spec:
+  certificateRotateStrategy: {}
+  configuration: {}
+  customizeComponents:
+    patches:
+    - resourceType: Deployment
+        resourceName: virt-controller
+        patch: '{"spec": {"template":{"metadata":{"annotations":{ "ad.datadoghq.com/virt-controller.check_names": "[\"kubevirt_controller\"]", "ad.datadoghq.com/virt-controller.init_configs": "[{}]", "ad.datadoghq.com/virt-controller.instances": "[{ \"kubevirt_controller_metrics_endpoint\": \"https://%%host%%:%%port%%/metrics\",\"kubevirt_controller_healthz_endpoint\": \"https://%%host%%:%%port%%/healthz\", \"kube_namespace\":\"%%kube_namespace%%\", \"kube_pod_name\":\"%%kube_pod_name%%\", \"tls_verify\": \"false\"}]"}}}}}'
+        type: strategic
+```
+
+Replace `<DD_CLUSTER_NAME>` with the name you want for your cluster.
+
+### Validation
+
+[Run the Cluster Agent's `clusterchecks` subcommand][5] inside your Cluster Agent container and look for the `kubevirt_controller` check under the Checks section.
+
+## Data Collected
+
+### Metrics
+
+See [metadata.csv][6] for a list of metrics provided by this integration.
+
+### Events
+
+The KubeVirt Controller integration does not include any events.
+
+### Service Checks
+
+The KubeVirt Controller integration does not include any service checks.
+
+## Troubleshooting
+
+Need help? Contact [Datadog support][7].
+
+[1]: https://docs.datadoghq.com/integrations/kubevirt_controller
+[2]: https://app.datadoghq.com/account/settings/agent/latest
+[3]: https://docs.datadoghq.com/agent/kubernetes/integrations/
+[4]: https://docs.datadoghq.com/containers/cluster_agent/clusterchecks/?tab=datadogoperator
+[5]: https://docs.datadoghq.com/containers/troubleshooting/cluster-and-endpoint-checks/#dispatching-logic-in-the-cluster-agent
+[6]: https://github.com/DataDog/integrations-core/blob/master/kubevirt_controller/metadata.csv
+[7]: https://docs.datadoghq.com/help/

kubevirt_controller/assets/configuration/spec.yaml

@@ -0,0 +1,41 @@
+name: KubeVirt Controller
+files:
+  - name: kubevirt_controller.yaml
+    options:
+      - template: init_config
+        options:
+          - template: init_config/openmetrics
+      - template: instances
+        options:
+          - name: kubevirt_controller_metrics_endpoint
+            description: |
+              URL to the KubeVirt Controller Service /metrics endpoint.
+            value:
+              display_default: null
+              example: https://10.244.0.38:443/metrics
+              type: string
+          - name: kubevirt_controller_healthz_endpoint
+            description: |
+              URL to check the KubeVirt Controller /healthz endpoint.
+            value:
+              display_default: null
+              example: https://10.244.0.38:443/healthz
+              type: string
+          - name: kube_namespace
+            description: |
+              The namespace where the KubeVirt Controller is running. (Provided by autodiscovery template variables)
+            value:
+              display_default: null
+              example: kubevirt
+              type: string
+          - name: kube_pod_name
+            description: |
+              The name of the KubeVirt Controller pod. (Provided by autodiscovery template variables)
+            value:
+              display_default: null
+              example: virt-controller-id-1234
+              type: string
+          - template: instances/openmetrics
+            overrides:
+              openmetrics_endpoint.required: false
+              openmetrics_endpoint.hidden: true

kubevirt_controller/assets/service_checks.json

@@ -0,0 +1 @@
+[]

kubevirt_controller/changelog.d/18186.added

@@ -0,0 +1 @@
+Initial Release

kubevirt_controller/datadog_checks/__init__.py

@@ -0,0 +1,4 @@
+# (C) Datadog, Inc. 2024-present
+# All rights reserved
+# Licensed under a 3-clause BSD style license (see LICENSE)
+__path__ = __import__('pkgutil').extend_path(__path__, __name__)  # type: ignore

kubevirt_controller/datadog_checks/kubevirt_controller/__about__.py

@@ -0,0 +1,4 @@
+# (C) Datadog, Inc. 2024-present
+# All rights reserved
+# Licensed under a 3-clause BSD style license (see LICENSE)
+__version__ = '0.0.1'

kubevirt_controller/datadog_checks/kubevirt_controller/__init__.py

@@ -0,0 +1,7 @@
+# (C) Datadog, Inc. 2024-present
+# All rights reserved
+# Licensed under a 3-clause BSD style license (see LICENSE)
+from .__about__ import __version__
+from .check import KubeVirtControllerCheck
+
+__all__ = ['__version__', 'KubeVirtControllerCheck']

kubevirt_controller/datadog_checks/kubevirt_controller/check.py

@@ -0,0 +1,115 @@
+# (C) Datadog, Inc. 2024-present
+# All rights reserved
+# Licensed under a 3-clause BSD style license (see LICENSE)
+
+from datadog_checks.base import OpenMetricsBaseCheckV2, is_affirmative
+from datadog_checks.base.checks.openmetrics.v2.transform import get_native_dynamic_transformer
+
+from .metrics import METRICS_MAP
+
+
+class KubeVirtControllerCheck(OpenMetricsBaseCheckV2):
+    __NAMESPACE__ = "kubevirt_controller"
+    DEFAULT_METRIC_LIMIT = 0
+
+    def __init__(self, name, init_config, instances):
+        super(KubeVirtControllerCheck, self).__init__(name, init_config, instances)
+        self.check_initializations.appendleft(self._parse_config)
+        self.check_initializations.append(self._init_base_tags)
+        self.check_initializations.append(self._configure_additional_transformers)
+
+    def check(self, _):
+        if self.kubevirt_controller_healthz_endpoint:
+            self._report_health_check(self.kubevirt_controller_healthz_endpoint)
+        else:
+            self.log.warning(
+                "Skipping health check. Please provide a `kubevirt_controller_healthz_endpoint` to ensure the health of the KubeVirt Controller."  # noqa: E501
+            )
+
+        super().check(_)
+
+    def _init_base_tags(self):
+        self.base_tags = [
+            "pod_name:{}".format(self.pod_name),
+            "kube_namespace:{}".format(self.kube_namespace),
+        ]
+
+    def _report_health_check(self, health_endpoint):
+        try:
+            self.log.debug("Checking health status at %s", health_endpoint)
+            response = self.http.get(health_endpoint, verify=self.tls_verify)
+            response.raise_for_status()
+            self.gauge("can_connect", 1, tags=[f"endpoint:{health_endpoint}", *self.base_tags])
+        except Exception as e:
+            self.log.error(
+                "Cannot connect to KubeVirt Controller HTTP endpoint '%s': %s.\n",
+                health_endpoint,
+                str(e),
+            )
+            self.gauge("can_connect", 0, tags=[f"endpoint:{health_endpoint}", *self.base_tags])
+            raise
+
+    def _parse_config(self):
+        self.kubevirt_controller_metrics_endpoint = self.instance.get("kubevirt_controller_metrics_endpoint")
+        self.kubevirt_controller_healthz_endpoint = self.instance.get("kubevirt_controller_healthz_endpoint")
+        self.kube_namespace = self.instance.get("kube_namespace")
+        self.pod_name = self.instance.get("kube_pod_name")
+        self.tls_verify = is_affirmative(self.instance.get("tls_verify"))
+
+        self.scraper_configs = []
+
+        instance = {
+            "openmetrics_endpoint": self.kubevirt_controller_metrics_endpoint,
+            "namespace": self.__NAMESPACE__,
+            "enable_health_service_check": False,
+            "tls_verify": self.tls_verify,
+        }
+
+        self.scraper_configs.append(instance)
+
+    def _configure_additional_transformers(self):
+        metric_transformer = self.scrapers[self.kubevirt_controller_metrics_endpoint].metric_transformer
+        metric_transformer.add_custom_transformer(r".*", self.configure_transformer_kubevirt_metrics(), pattern=True)
+
+    def configure_transformer_kubevirt_metrics(self):
+        """
+        Return a metrics transformer that adds tags to all the collected metrics.
+        """
+
+        def transform(_metric, sample_data, _runtime_data):
+            for sample, tags, hostname in sample_data:
+                metric_name = _metric.name
+                metric_type = _metric.type
+
+                # ignore metrics we don't collect
+                if metric_name not in METRICS_MAP:
+                    continue
+
+                # attach tags to the metric
+                tags = tags + self.base_tags
+
+                # apply the METRICS_MAP mapping for the metric name
+                new_metric_name = METRICS_MAP[metric_name]
+                if isinstance(new_metric_name, dict) and "name" in new_metric_name:
+                    new_metric_name = new_metric_name["name"]
+
+                # call the correct metric submission method based on the metric type
+                if metric_type == "counter":
+                    self.count(new_metric_name + ".count", sample.value, tags=tags, hostname=hostname)
+                elif metric_type == "gauge":
+                    self.gauge(new_metric_name, sample.value, tags=tags, hostname=hostname)
+                else:
+                    metric_transformer = self.scrapers[self.kubevirt_controller_metrics_endpoint].metric_transformer
+
+                    native_transformer = get_native_dynamic_transformer(
+                        self, new_metric_name, None, metric_transformer.global_options
+                    )
+
+                    def add_tag_to_sample(sample, pod_tags):
+                        [sample, tags, hostname] = sample
+                        return [sample, tags + pod_tags, hostname]
+
+                    modified_sample_data = (add_tag_to_sample(x, self.base_tags) for x in sample_data)
+                    native_transformer(_metric, modified_sample_data, _runtime_data)
+
+        return transform

kubevirt_controller/datadog_checks/kubevirt_controller/config_models/__init__.py

@@ -0,0 +1,24 @@
+# (C) Datadog, Inc. 2024-present
+# All rights reserved
+# Licensed under a 3-clause BSD style license (see LICENSE)
+
+# This file is autogenerated.
+# To change this file you should edit assets/configuration/spec.yaml and then run the following commands:
+#     ddev -x validate config -s <INTEGRATION_NAME>
+#     ddev -x validate models -s <INTEGRATION_NAME>
+
+from .instance import InstanceConfig
+from .shared import SharedConfig
+
+
+class ConfigMixin:
+    _config_model_instance: InstanceConfig
+    _config_model_shared: SharedConfig
+
+    @property
+    def config(self) -> InstanceConfig:
+        return self._config_model_instance
+
+    @property
+    def shared_config(self) -> SharedConfig:
+        return self._config_model_shared