otel-add-on

A Helm chart for KEDA otel-add-on

:::^.     .::::^:     :::::::::::::::    .:::::::::.                   .^.
7???~   .^7????~.     7??????????????.   :?????????77!^.              .7?7.
7???~  ^7???7~.       ~!!!!!!!!!!!!!!.   :????!!!!7????7~.           .7???7.
7???~^7????~.                            :????:    :~7???7.         :7?????7.
7???7????!.           ::::::::::::.      :????:      .7???!        :7??77???7.
7????????7:           7???????????~      :????:       :????:      :???7?5????7.
7????!~????^          !77777777777^      :????:       :????:     ^???7?#P7????7.
7???~  ^????~                            :????:      :7???!     ^???7J#@J7?????7.
7???~   :7???!.                          :????:   .:~7???!.    ~???7Y&@#7777????7.
7???~    .7???7:      !!!!!!!!!!!!!!!    :????7!!77????7^     ~??775@@@GJJYJ?????7.
7???~     .!????^     7?????????????7.   :?????????7!~:      !????G@@@@@@@@5??????7:
::::.       :::::     :::::::::::::::    .::::::::..        .::::JGGGB@@@&7:::::::::
        _       _               _     _                               ?@@#~
   ___ | |_ ___| |     __ _  __| | __| |     ___  _ __                P@B^
  / _ \| __/ _ \ |    / _` |/ _` |/ _` |___ / _ \| '_ \             :&G:
 | (_) | ||  __/ |   | (_| | (_| | (_| |___| (_) | | | |            !5.
  \___/ \__\___|_|    \__,_|\__,_|\__,_|    \___/|_| |_|            ,
                                                                    .

Homepage: https://github.com/kedify/otel-add-on

Usage

Check available version in OCI repo:

crane ls ghcr.io/kedify/charts/otel-add-on | grep -E '^v?[0-9]'

Install specific version:

helm upgrade -i oci://ghcr.io/kedify/charts/otel-add-on --version=v0.1.3

Advanced stuff:

# check /examples dir in project root
find ./../../examples -name '*-values.yaml'

Source Code

• https://github.com/kedify/otel-add-on
• https://github.com/open-telemetry/opentelemetry-helm-charts

Requirements

Kubernetes: >= 1.19.0-0

Repository	Name	Version
https://open-telemetry.github.io/opentelemetry-helm-charts	otelCollector(opentelemetry-collector)	0.131.0
https://open-telemetry.github.io/opentelemetry-helm-charts	otelOperator(opentelemetry-operator)	0.93.0

OTel Collector Sub-Chart

This helm chart, when enabled by --set otelCollector.enabled=true, installs the OTel collector using its upstream helm chart.

To check all the possible values for this dependent helm chart, consult values.yaml or docs.

All these values goes under otelCollector section.

Example:

settings:
  metricStore:
    retentionSeconds: 60
otelCollector:
  enabled: true
  # <HERE>
  alternateConfig:
    receivers:
    ...

OTel Operator Sub-Chart

Using the otelCollector sub-chart (described in the previous section) we can install one instance of OTel collector. However, all the helm chart values needs to be passed in advance including the OTel collector configuration section. One limitation of Helm is the absence of templating the helm chart values itself. This would be very useful, because some things in the OTel configuration are dynamic (addresses etc.)

We can achieve that by using the upstream OTel Operator and render¹ its CRDs using this helm chart.

Configuration of OpenTelemetryCollector CR is driven by:

• .otelOperatorCrDefaultTemplate (defaults)
• .otelOperatorCrs (overrides)

Tip

If there is a default set on .otelOperatorCrDefaultTemplate level, say:

otelOperatorCrDefaultTemplate:
  alternateExporters:
    otlp/central:
      protocols:
        grpc:
          endpoint: external-backup:4317

and we want to make the field alternateExporters empty, we can do that by:

otelOperatorCrDefaultTemplate:
  alternateExporters:
    otlp/central:
      protocols:
        grpc:
          endpoint: external-backup:4317
otelOperatorCrs:
  - enabled: true
    name: "nonDefault"
    alternateExporters: null

Otherwise, the behavior of the config merge is as expected (code). Also if the alternateExporters field in the merged config is empty, we will create an implicit exporter that will feed the metrics into KEDA OTel scaler with preconfigured service name. If from any reason you would like to disable all the exporters for the OTel collector, add only a dummy debug exporter:

noglob helm template oci://ghcr.io/kedify/charts/otel-add-on --version=v0.1.3 \
   --set otelOperatorCrs[0].alternateExporters.debug.verbosity=basic \
   --set otelOperatorCrs[0].enabled=true

So one can deploy whole metric pipeline including multiple OTel collectors with different settings as one helm release using this chart. You can check the description for otelOperatorCrDefaultTemplate in Values section for such example.

For:

• receivers
• exporters
• processors
• extensions
• pipelines You can use the alternate{Receivers,Exporters,Processors,Extensions,Pipelines} config options on both CR level and default-template level to tweak the OTel collector config. This has the benefit that it will also enable it under the .service.pipelines option so there is no need to repeat yourself. However, if you want to provide the full OTel collector configuration, you can do that by putting it under alternateOtelConfig (again CR level or default template). When alternateOtelConfig is set, all the alternate{Receivers,Exporters,Processors,Extensions,Pipelines} are ignored.

Values

Key	Description	Default
image.repository (Type: string)	Image to use for the Deployment	"ghcr.io/kedify/otel-add-on"
image.pullPolicy (Type: image-pull-policy)	image pull policy for KEDA OTel Scaler pod	"Always"
image.tag (Type: string)	Image version to use for the Deployment, if not specified, it defaults to .Chart.AppVersion	""
settings .metricStore .retentionSeconds (Type: int)	how long the metrics should be kept in the short term (in memory) storage	120
settings .metricStore .lazySeries (Type: bool)	if enabled, no metrics will be stored until there is a request for such metric from KEDA operator.	false
settings .metricStore .lazyAggregates (Type: bool)	if enabled, the only aggregate that will be calculated on the fly is the one referenced in the metric query (by default, we calculate and store all of them - sum, rate, min, max, etc.)	false
settings .metricStore .errIfNotFound (Type: bool)	when enabled, the scaler will be returning error to KEDA's GetMetrics() call	false
settings .metricStore .valueIfNotFound (Type: float)	default value, that is reported in case of error or if the value is not in the mem store	0
settings .isActivePollingIntervalMilliseconds (Type: int)	how often (in milliseconds) should the IsActive method be tried	500
settings.internalMetricsPort (Type: int)	internal (mostly golang) metrics will be exposed on :8080/metrics	8080
settings.restApiPort (Type: int)	port where rest api should be listening	9090
settings.logs.logLvl (Type: string)	Can be one of 'debug', 'info', 'error', or any integer value > 0 which corresponds to custom debug levels of increasing verbosity	"info"
settings.logs.stackTracesLvl (Type: string)	one of: info, error, panic	"error"
settings.logs.noColor (Type: bool)	if anything else than 'false', the log will not contain colors	false
settings.logs.noBanner (Type: bool)	if anything else than 'false', the log will not print the ascii logo	false
settings.tls.caFile (Type: optional)	path to CA certificate. When provided, the client certificate will be verified using this CA where "client" ~ another OTLP exporter.	""
settings.tls.certFile (Type: optional)	path to TLS certificate that will be used for OTLP receiver	""
settings.tls.keyFile (Type: optional)	path to TLS key that will be used for OTLP receiver	""
settings.tls.reloadInterval (Type: optional)	specifies the duration after which the certificates will be reloaded. This is useful when using the CertManager for rotating the certs mounted as Secrets.	"5m"
settings.tls.keda.certFile (Type: optional)	path to TLS certificate that will be used for KEDA gRPC server. If empty, defaults to settings.tls.certFile	""
settings.tls.keda.keyFile (Type: optional)	path to TLS key that will be used for KEDA gRPC server. If empty, defaults to settings.tls.keyFile	""
settings.tls.secrets (Type: optional)	list of secrets that will be mounted to deployment's pod. One entry in this list, will create one volume and one volumeMount for pod. This is a convenient way for mounting the certs for TLS, but using .volumes & .volumeMounts for anything advanced will also work.	[]
deploymentStrategy (Type: strategy)	one of: RollingUpdate, Recreate	"RollingUpdate"
deployScaler (Type: bool)	when disabled, the deployment with KEDA Scaler will not be rendered	true
validatingAdmissionPolicy .enabled (Type: bool)	whether the ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding resources should be also rendered	false
asciiArt (Type: bool)	should the ascii logo be printed when this helm chart is installed	true
imagePullSecrets (Type: specifying-imagepullsecrets-on-a-pod)	imagePullSecrets for KEDA OTel Scaler pod	[]
serviceAccount.create (Type: bool)	should the service account be also created and linked in the deployment	true
serviceAccount.annotations (Type: object)	further custom annotation that will be added on the service account	{}
serviceAccount.name (Type: string)	name of the service account, defaults to otel-add-on.fullname ~ release name if not overriden	""
podAnnotations (Type: object)	additional custom pod annotations that will be used for pod	{}
podLabels (Type: object)	additional custom pod labels that will be used for pod	{}
podSecurityContext (Type: pod-security-standards)	securityContext for KEDA OTel Scaler pod	{}
securityContext (Type: pod-security-standards)	securityContext for KEDA OTel Scaler container	capabilities: drop: - ALL readOnlyRootFilesystem: true runAsNonRoot: true runAsUser: 1000
service.type (Type: publishing-services-service-types)	Under this service, the otel add on needs to be reachable by KEDA operator and OTel collector	"ClusterIP"
service.otlpReceiverPort (Type: int)	OTLP receiver will be opened on this port. OTel exporter configured in the OTel collector needs to have this value set.	4317
service.kedaExternalScalerPort (Type: int)	KEDA external scaler will be opened on this port. ScaledObject's .spec.triggers[].metadata.scalerAddress needs to be set to this svc and this port.	4318
service.name (Type: string)	name under which the scaler should be exposed, if left empty, it will try .Values.fullnameOverride and if this is empty, name of the release is used this should handle the case when one needs to install multiple instances of this chart into one cluster and at the same time provide a way to specify a stable address	""
resources (Type: manage-resources-containers)	resources for the OTel Scaler pod	limits: cpu: 500m memory: 256Mi requests: cpu: 500m memory: 128Mi
nodeSelector (Type: nodeselector)	node selector for KEDA OTel Scaler pod	{}
tolerations (Type: taint-and-toleration)	tolerations for KEDA OTel Scaler pod	[]
affinity (Type: affinity-and-anti-affinity)	affinity for KEDA OTel Scaler pod	{}
kubectlImage (Type: yaml)	helper container image that creates the OpenTelemetryCollector CR as post-install helm hook	tag: "v1.33.1" repository: ghcr.io/kedify/kubectl pullPolicy: Always pullSecrets: []
otelOperatorCrDefaultTemplate (Type: raw)	This field defines the default template for OpenTelemetryCollector CR Vast majority of the fields has its counterpart described in OpenTelemetryCollector CRD. In order to check their description, install the CRD and run: kubectl explain otelcol.spec These defaults are then used as a base layer of configuration for all the items in the .otelOperatorCrs list. So given we have this values: otelOperatorCrDefaultTemplate: mode: deployment otelOperatorCrs: - enabled: true name: "foo" mode: "daemonset" - enabled: true name: "bar" It will render1 two OpenTelemetryCollector CRs called foo and bar where foo will have the .spec.mode set to daemonset and foo will inherit the default mode from .otelOperatorCrDefaultTemplate.mode => deployment Note When specifying custom receivers, processors, exporters or extensions. Use alternate{Receivers,Processors,Exporters,Extensions}. And there is no need to enable these under pipeline section. This is done automagically here. Tip For overriding the whole OTel config, use the .alternateOtelConfig field. Advanced example: Expand values.yaml: otelOperator: enabled: true otelOperatorCrDefaultTemplate: mode: deployment alternateReceivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 alternateExporters: otlp: endpoint: col-fanout-collector:4317 tls: insecure: true otelOperatorCrs: - enabled: true name: "col-in-1" - enabled: true name: "col-in-2" - enabled: true name: "col-fanout" alternateExporters: otlp/external: endpoint: external-collector:4317 otlp/keda: endpoint: keda-otel-scaler.keda.svc:4317 tls: insecure: true alternateConnectors: routing: default_pipelines: [metrics/all] table: - context: metric condition: metric.name == "http_requests_total" pipelines: [metrics/keda] alternatePipelines: metrics/in: receivers: [otlp] exporters: [routing] metrics/keda: receivers: [routing] exporters: [otlp/keda] metrics/all: receivers: [routing] exporters: [otlp/external] resulting architecture: graph TD; A[col-in-1] -- metrics --> C{col-fanout} B[col-in-2] -- metrics, traces --> C{col-fanout} C -- one metric --> D(KEDA Scaler) C -- all --> E((external-col)) Loading
otelOperatorCrDefaultTemplate .debug (Type: bool)	container image for post-install helm hook that help with OpenTelemetryCollector CR installation	false
otelOperatorCrDefaultTemplate .mode (Type: string)	how the otel collector should be deployed: sidecar, statefulset, deployment, daemonset note: make sure the CertManager is installed and admission webhooks are enabled for the OTel operator when using mode=sidecar	"deployment"
otelOperatorCrDefaultTemplate .targetAllocatorEnabled (Type: bool)	whether TargetAllocator feature (Prometheus Custom Resources for service discovery) should be enabled (details) make sure the mode is not set to sidecar when this is enabled	false
otelOperatorCrDefaultTemplate .targetAllocatorClusterRoles (Type: list)	list of existing cluster roles that will be bound to the service account (in order to be able to work with {Pod,Service}Monitor CRD)	[ "kube-prometheus-stack-operator", "kube-prometheus-stack-prometheus" ]
otelOperatorCrDefaultTemplate .targetAllocator .prometheusCR .serviceMonitorSelector (Type: object)	further narrow the ServiceMonitor CRs (labels)	{}
otelOperatorCrDefaultTemplate .targetAllocator .prometheusCR .podMonitorSelector (Type: object)	further narrow the PodMonitor CRs	{}
otelOperatorCrDefaultTemplate .tls (Type: object)	TLS settings for OTel collector's exporter that feeds the metrics to KEDA OTel scaler it is not in scope of this helm chart to create the secrets with certificate, however this is a convenient way of configuring volumes and volumeMounts for each secret with cert. It has the same structure as tls settings for the scaler (check .Values.tls). One significant difference is that here we specify a client cert for OTLP exporter, while .Values.tls specify the server cert for OTLP receiver	{}
otelOperatorCrDefaultTemplate .resources (Type: manage-resources-containers)	resources for the OTel collector container	limits: cpu: 400m memory: 128Mi requests: cpu: 200m memory: 64Mi
otelOperatorCrDefaultTemplate .alternateOtelConfig (Type: object)	free form OTel configuration that will be used for the OpenTelemetryCollector CR (no checks) this is mutually exclusive w/ all the following options	{}
otelOperatorCrDefaultTemplate .prometheusScrapeConfigs (Type: list)	static targets for prometheus receiver, this needs to take into account the deployment mode of the collector (127.0.0.1 in case of a sidecar mode will mean something else than for statefulset mode)	[ { "job_name": "otel-collector", "scrape_interval": "3s", "static_configs": [ { "targets": [ "0.0.0.0:8080" ] } ] } ]
otelOperatorCrDefaultTemplate .alternateReceivers (Type: object)	mutually exclusive with prometheusScrapeConfigs option	{}
otelOperatorCrDefaultTemplate .includeMetrics (Type: list)	if not empty, only following metrics will be sent. This translates to filter/metrics processor. Empty array means include all.	[]
otelOperatorCrs (Type: yaml)	create also OpenTelemetryCollector CRs that will be reconciled by OTel Operator it takes all the default settings defined in otelOperatorCrDefaultTemplate and allows overriding them here	# -- if enabled, the OpenTelemetryCollector CR will be created using post-install hook job_name - enabled: false # -- name of the OpenTelemetryCollector CR. If left empty, the release name will be used. name: "" # -- in what k8s namespace the OpenTelemetryCollector CR should be created. If left empty, the release namespace will be used. namespace: "" - name: target-allocator enabled: false targetAllocatorEnabled: true mode: deployment
otelOperatorCrs[0] (Type: object)	if enabled, the OpenTelemetryCollector CR will be created using post-install hook job_name	{ "enabled": false, "name": "", "namespace": "" }
otelOperatorCrs[0].name (Type: string)	name of the OpenTelemetryCollector CR. If left empty, the release name will be used.	""
otelOperatorCrs[0].namespace (Type: string)	in what k8s namespace the OpenTelemetryCollector CR should be created. If left empty, the release namespace will be used.	""
otelOperator (Type: yaml)	values for OTel operator helm chart - these values overrides the defaults defined here by default the operator is disabled	enabled: false fullnameOverride: otel-operator manager: collectorImage: # repository: otel/opentelemetry-collector-k8s repository: otel/opentelemetry-collector-contrib env: ENABLE_WEBHOOKS: "false" serviceAccount: name: otel-operator admissionWebhooks: create: false
otelCollector (Type: yaml)	values for OTel collector helm chart - these values overrides the defaults defined here by default the collector is disabled	# -- If enabled, the OTel collector sub-chart will be rendered enabled: false # -- Valid values are "daemonset", "deployment", "sidecar" and "statefulset" mode: deployment image: # repository: otel/opentelemetry-collector-k8s repository: otel/opentelemetry-collector-contrib # -- Container image - OTel collector distribution fullnameOverride: otelcol # ports: # opencensus: # enabled: true # containerPort: 55678 # servicePort: 55678 # hostPort: 55678 # protocol: TCP # -- Configuration for OTel collector that will be installed # @notationType -- yaml alternateConfig: receivers: {} processors: resourcedetection/env: detectors: [env] timeout: 2s override: false transform: metric_statements: - context: datapoint statements: - set(attributes["namespace"], resource.attributes["k8s.namespace.name"]) - set(attributes["pod"], resource.attributes["k8s.pod.name"]) - set(attributes["deployment"], resource.attributes["k8s.deployment.name"]) exporters: otlp: # make sure this is the same hostname and port as .service (when using different namespace) endpoint: keda-otel-scaler.keda.svc:4317 compression: "none" tls: insecure: true debug: verbosity: detailed service: extensions: - health_check pipelines: metrics: receivers: [] processors: [resourcedetection/env, transform] exporters: [debug, otlp] extensions: health_check: endpoint: ${env:MY_POD_IP}:13133
otelCollector.enabled (Type: bool)	If enabled, the OTel collector sub-chart will be rendered	false
otelCollector.mode (Type: string)	Valid values are "daemonset", "deployment", "sidecar" and "statefulset"	"deployment"
otelCollector.alternateConfig (Type: yaml)	Configuration for OTel collector that will be installed	receivers: {} processors: resourcedetection/env: detectors: [env] timeout: 2s override: false transform: metric_statements: - context: datapoint statements: - set(attributes["namespace"], resource.attributes["k8s.namespace.name"]) - set(attributes["pod"], resource.attributes["k8s.pod.name"]) - set(attributes["deployment"], resource.attributes["k8s.deployment.name"]) exporters: otlp: # make sure this is the same hostname and port as .service (when using different namespace) endpoint: keda-otel-scaler.keda.svc:4317 compression: "none" tls: insecure: true debug: verbosity: detailed service: extensions: - health_check pipelines: metrics: receivers: [] processors: [resourcedetection/env, transform] exporters: [debug, otlp] extensions: health_check: endpoint: ${env:MY_POD_IP}:13133

Footnotes

1. Well in fact it doesn't render the OpenTelemetryCollector CRs directly, but nested as part of a ConfigMap. Then this CM is read durin post-install hook and CR is created. This is because we can't render CRD and its instances in one helm command. ↩ ↩²