[otel-demo] Use otel-operator to inject OTel configuration and manage OTel Collectors

[cyrille-leclerc]

Use the OpenTelemetry Kubernetes Operator (aka OTel Operator) to inject SDK configuration and manage OTel Collectors.

To ease usage of the OTel Operator, we deploy it through the OpenTelemetry Kube Stack Helm Chart.

Noteworthy:

• Instrumentation: we only inject the OTel configuration through he inject-sdk with env vars like OTEL_OTLP_EXPORTER_ENDPOINT, we don't inject the SDKs themselves like inject-java because OTel SDKs are already bundled in the container images. Note that bundling OTel SDKs in container images enables the OTel Demo to produce the same container images for th Docker Compose and Kubernetes deployments.
• The OTel Demo components lack consistency in their usage of OTLP grpc and http/protobuf requiring to inject the 2 grpc and http/protobuf endpoint configurations while the OTel Operator Instrumentation CRD doesn't offer such flexibility, requiring for the moment to sometimes override the OTEL_EXPORTER_OTLP_ENDPOINT env var at the component level.
• The OTel Demo Helm Chart MUST ensure that the OTel Operator is up and the Instrumentation CRD is deployed before starting the component (e.g. ad, fraud-detection...) so that these components get the OTel configuration injected through env vars.
  • ⚠️ the solution found in this PR to ensure the components are started after the otel-operator instrumentation is up is to restart all components after the helm chart installed the resources through the restart-services-after-otel-operator-is-ready job
• Important limitation on Docker Desktop Kubernetes: limitations of file system mounts requires to disable pod logs and host metrics scrapping.

Pending work:

• Review all ./examples an propose alternatives:
  • bring-your-own-observability: should just be a change in the otel-col config in Values.yaml to add exporters to "your own observability"
  • collector-as-damonset: I deleted it as daemon becomes the default deployment
  • custom-environment-variable: should just be a change in Values.yaml
  • kubernetes-infra-monitoring: I guess we could delete it, it will just be changing the presets in Values.yaml: logsCollection, hostMetrics, kubeletMetrics, kubernetesEvents, and clusterMetrics
  • public-hosted-ingres: TODO
• Fix the problem of OTel Col receivers (httpcheck/frontend-proxy, nginx, postgresql, redis...) that are currently deployed on daemon collectors even though they should run as singletons. Solutions like leader election, OTel Col receiver creator, or the creation of a Gateway collector should be looked at.

Successfully tested on

• Docker Desktop Mac Kubernetes: cpu_limit=8, memory_limit=8GB, swap=1GB
• Digital Ocean Kubernetes 2 nodes with 4vCPUs and 8GB each

FYI @rogercoll

[rogercoll]

Could you share more context on why we should use auto-instrumentation for the otel-demo services? The otel-demo services are already configured with its corresponding Otel SDK, what would be the use case of dynamically injecting another SDK?

An alterative would be using the kube-stack Helm Chart for deploying the collectors and create a new k8s-only uninstrumented (no SDK) service in the otel-demo for the auto instrumentation use case.

[cyrille-leclerc]

Could you share more context on why we should use auto-instrumentation for the otel-demo services? The otel-demo services are already configured with its corresponding Otel SDK, what would be the use case of dynamically injecting another SDK?

+1 that otel-demo services are instrumented with OTel SDKs today. However, I think it's valuable to demo the OTel Operator Instrumentation CRD with the inject-sdk pod annotation today:

• Show the best practices and soon hopefully stop bundling the OTel SDK in the container images of the otel-demo service
• Get the benefit of the OTel Operator injecting the SDK config through the OTEL_ env vars, particularly resource attributes with the following benefits:
  • For OTel practitioners, get their telemetry fully enriched and compliant with OTel specs like Specify resource attributes using Kubernetes annotations. Manually setting resource attributes on K8s is very error prone.
  • For the OTel project, to verify that the OTel Operator Instrumentation CRD doesn't forget anything. For example I discovered through this PR that most services of the demo set a wrong host.name value, using the k8s.pod.name value. I guess it's because the OTel SDK HostNameProvider that is used when it's not the desired value in containers

An alterative would be using the kube-stack Helm Chart for deploying the collectors and create a new k8s-only uninstrumented (no SDK) service in the otel-demo for the auto instrumentation use case.

+1 I see it as a subsequent milestone.