Webhook "address is not allowed" when creating an Instrumentation on EKS

[ilyamochalov]

Component(s)

instrumentation

Describe the issue you're reporting

Hi,
When creating an Instrumentation resource on a AWS EKS cluster I encounter the following error:

Error from server (InternalError): error when creating "m.yaml": Internal error occurred: failed calling webhook "mopentelemetrycollector.kb.io": Post: https://opentelemetry-operator-webhook.NAMESAPCE.svc/mutate-opentelemetry-io-v1alpha1-opentelemetrycollector?timeout=10s Address is not allowed

AWS EKS: v1.26.7-eks-2d98532
Open-telemetry operator: 0.39.2

This issue is similar to:

• Webhook timeout when creating an Instrumentation on GKE private cluster #1009
• How to resolve failing or missing response / Address is not allowed from custom kubernetes api service? kubernetes-sigs/prometheus-adapter#543

Steps to reproduce

Install the operator in an existing private EKS cluster where

Once the opentelemetry-operator deployment is ready, create an OpenTelemetry Instrumentation resource:

Solution

Use hostNetwork: true https://github.com/open-telemetry/opentelemetry-helm-charts/blob/opentelemetry-operator-0.41.0/charts/opentelemetry-operator/values.yaml#L251

Where is a good place to document this configuration consideration?

[TylerHelmuth]

I'd like to understand more about this issue and why using hostNetwork is a solution. I do not believe this is pervasive to all AWS EKS clusters.

[TylerHelmuth]

@ilyamochalov what are you using to manage the certificates?

[ilyamochalov]

@TylerHelmuth Hm... Address is not allowed is quite a generic error message. I tried to dig out if from Golang standard library but didn't find exact matches.

For certificates we are using https://github.com/cert-manager/cert-manager/tree/v1.10.1/deploy/charts/cert-manager

[AlecAttwood]

@ilyamochalov I know this issue has been around for a few months. But I encountered the same problem recently when installed OTEL on a cluster with the Calico CNI Installed. Are you using the base EKS VPC CNI or you using a different CNI like Calico?

[ilyamochalov]

@AlecAttwood I am using Calico too