smoke-simplest e2e test fails intermittently

[kevinearls]

I have seen this on CI and can reproduce it on my laptop by running the test repeatedly. The last time I did so it failed on the 11th iteration with the output below. The key part is: ".spec.ports.name: value mismatch, expected: jaeger-grpc != actual: otlp-grpc" as if it's confusing the jaeger-grpc and otlp-grpc ports.

=== RUN   kuttl/harness/smoke-simplest
=== PAUSE kuttl/harness/smoke-simplest
=== CONT  kuttl/harness/smoke-simplest
    logger.go:42: 10:29:06 | smoke-simplest | Creating namespace: kuttl-test-kind-kangaroo
    logger.go:42: 10:29:06 | smoke-simplest/0-install | starting test step 0-install
I0712 10:29:08.218699   29266 request.go:655] Throttling request took 1.045197889s, request: GET:https://192.168.64.2:8443/apis/discovery.k8s.io/v1beta1?timeout=32s
    logger.go:42: 10:29:08 | smoke-simplest/0-install | OpenTelemetryCollector:kuttl-test-kind-kangaroo/simplest created
    logger.go:42: 10:31:39 | smoke-simplest/0-install | test step failed 0-install
    case.go:361: failed in step 0-install
    case.go:363: --- Service:kuttl-test-kind-kangaroo/simplest-collector
        +++ Service:kuttl-test-kind-kangaroo/simplest-collector
        @@ -1,15 +1,85 @@
         apiVersion: v1
         kind: Service
         metadata:
        +  labels:
        +    app.kubernetes.io/component: opentelemetry-collector
        +    app.kubernetes.io/instance: kuttl-test-kind-kangaroo.simplest
        +    app.kubernetes.io/managed-by: opentelemetry-operator
        +    app.kubernetes.io/name: simplest-collector
        +    app.kubernetes.io/part-of: opentelemetry
        +    app.kubernetes.io/version: latest
        +  managedFields:
        +  - apiVersion: v1
        +    fieldsType: FieldsV1
        +    fieldsV1:
        +      f:metadata:
        +        f:labels:
        +          .: {}
        +          f:app.kubernetes.io/component: {}
        +          f:app.kubernetes.io/instance: {}
        +          f:app.kubernetes.io/managed-by: {}
        +          f:app.kubernetes.io/name: {}
        +          f:app.kubernetes.io/part-of: {}
        +          f:app.kubernetes.io/version: {}
        +        f:ownerReferences:
        +          .: {}
        +          k:{"uid":"dfc52c93-ed85-4cab-8953-0a02fe04c545"}: {}
        +      f:spec:
        +        f:internalTrafficPolicy: {}
        +        f:ports:
        +          .: {}
        +          k:{"port":4317,"protocol":"TCP"}:
        +            .: {}
        +            f:appProtocol: {}
        +            f:name: {}
        +            f:port: {}
        +            f:protocol: {}
        +            f:targetPort: {}
        +          k:{"port":4318,"protocol":"TCP"}:
        +            .: {}
        +            f:appProtocol: {}
        +            f:name: {}
        +            f:port: {}
        +            f:protocol: {}
        +            f:targetPort: {}
        +          k:{"port":14250,"protocol":"TCP"}:
        +            .: {}
        +            f:appProtocol: {}
        +            f:name: {}
        +            f:port: {}
        +            f:protocol: {}
        +            f:targetPort: {}
        +          k:{"port":55681,"protocol":"TCP"}:
        +            .: {}
        +            f:appProtocol: {}
        +            f:name: {}
        +            f:port: {}
        +            f:protocol: {}
        +            f:targetPort: {}
        +        f:selector: {}
        +        f:sessionAffinity: {}
        +        f:type: {}
        +    manager: manager
        +    operation: Update
        +    time: "2022-07-12T08:29:08Z"
           name: simplest-collector
           namespace: kuttl-test-kind-kangaroo
        +  ownerReferences:
        +  - apiVersion: opentelemetry.io/v1alpha1
        +    blockOwnerDeletion: true
        +    controller: true
        +    kind: OpenTelemetryCollector
        +    name: simplest
        +    uid: dfc52c93-ed85-4cab-8953-0a02fe04c545
         spec:
        +  clusterIP: 10.111.179.189
        +  clusterIPs:
        +  - 10.111.179.189
        +  internalTrafficPolicy: Cluster
        +  ipFamilies:
        +  - IPv4
        +  ipFamilyPolicy: SingleStack
           ports:
        -  - appProtocol: grpc
        -    name: jaeger-grpc
        -    port: 14250
        -    protocol: TCP
        -    targetPort: 14250
           - appProtocol: grpc
             name: otlp-grpc
             port: 4317
        @@ -25,4 +95,20 @@
             port: 55681
             protocol: TCP
             targetPort: 4318
        +  - appProtocol: grpc
        +    name: jaeger-grpc
        +    port: 14250
        +    protocol: TCP
        +    targetPort: 14250
        +  selector:
        +    app.kubernetes.io/component: opentelemetry-collector
        +    app.kubernetes.io/instance: kuttl-test-kind-kangaroo.simplest
        +    app.kubernetes.io/managed-by: opentelemetry-operator
        +    app.kubernetes.io/name: simplest-collector
        +    app.kubernetes.io/part-of: opentelemetry
        +    app.kubernetes.io/version: latest
        +  sessionAffinity: None
        +  type: ClusterIP
        +status:
        +  loadBalancer: {}


    case.go:363: resource Service:kuttl-test-kind-kangaroo/simplest-collector: .spec.ports.name: value mismatch, expected: jaeger-grpc != actual: otlp-grpc
Warning: events.k8s.io/v1beta1 Event is deprecated in v1.22+, unavailable in v1.25+
    logger.go:42: 10:31:39 | smoke-simplest | smoke-simplest events from ns kuttl-test-kind-kangaroo:
    logger.go:42: 10:31:39 | smoke-simplest | 2022-07-12 10:29:09 +0200 CEST	Normal	Pod simplest-collector-85b4bc9b74-g6b8f		Scheduled	Successfully assigned kuttl-test-kind-kangaroo/simplest-collector-85b4bc9b74-g6b8f to minikube
    logger.go:42: 10:31:39 | smoke-simplest | 2022-07-12 10:29:09 +0200 CEST	Normal	Pod simplest-collector-85b4bc9b74-g6b8f.spec.containers{otc-container}		Pulled	Container image "ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector:0.54.0" already present on machine
    logger.go:42: 10:31:39 | smoke-simplest | 2022-07-12 10:29:09 +0200 CEST	Normal	Pod simplest-collector-85b4bc9b74-g6b8f.spec.containers{otc-container}		Created	Created container otc-container
    logger.go:42: 10:31:39 | smoke-simplest | 2022-07-12 10:29:09 +0200 CEST	Normal	ReplicaSet.apps simplest-collector-85b4bc9b74		SuccessfulCreate	Created pod: simplest-collector-85b4bc9b74-g6b8f
    logger.go:42: 10:31:39 | smoke-simplest | 2022-07-12 10:29:09 +0200 CEST	Normal	Deployment.apps simplest-collector		ScalingReplicaSet	Scaled up replica set simplest-collector-85b4bc9b74 to 1
    logger.go:42: 10:31:39 | smoke-simplest | 2022-07-12 10:29:10 +0200 CEST	Normal	Pod simplest-collector-85b4bc9b74-g6b8f.spec.containers{otc-container}		Started	Started container otc-container
=== CONT  kuttl
    harness.go:402: run tests finished
    harness.go:511: cleaning up
    harness.go:568: removing temp folder: ""
--- FAIL: kuttl (165.93s)
    --- FAIL: kuttl/harness (0.00s)
        --- FAIL: kuttl/harness/smoke-simplest (152.86s)
FAIL

[pavolloffay]

Please provide as well links to the failed jobs. It helps with the "investigation".

[kevinearls]

See: https://github.com/open-telemetry/opentelemetry-operator/actions/runs/2648700881

[kevinearls]

It looks like this is an issue with how ports are listed. Normally "kubectl get service..." for the collector or collector-headless object created returns yaml that looks like:

ports:
  - appProtocol: grpc
    name: jaeger-grpc
    port: 14250
    protocol: TCP
    targetPort: 14250
  - appProtocol: grpc
    name: otlp-grpc
    port: 4317
    protocol: TCP
    targetPort: 4317
  - appProtocol: http
    name: otlp-http
    port: 4318
    protocol: TCP
    targetPort: 4318
  - appProtocol: http
    name: otlp-http-legacy
    port: 55681
    protocol: TCP
    targetPort: 4318

Occasionally, however it returns yaml like that below with the jaeger-grpc port listed last instead of first, which causes the test to fail:

 ports:
  - appProtocol: grpc
    name: otlp-grpc
    port: 4317
    protocol: TCP
    targetPort: 4317
  - appProtocol: http
    name: otlp-http
    port: 4318
    protocol: TCP
    targetPort: 4318
  - appProtocol: http
    name: otlp-http-legacy
    port: 55681
    protocol: TCP
    targetPort: 4318
  - appProtocol: grpc
    name: jaeger-grpc
    port: 14250
    protocol: TCP
    targetPort: 14250

[pavolloffay]

The issue is most likely in the way how the operator derives ports. It might store ports in a map which is used to create port array - this approach can produce different results.

Some code pointers to places where the ports are handled:

opentelemetry-operator/pkg/collector/adapters/config_to_ports.go

Line 63 in 90e4d31

ports := []corev1.ServicePort{}

opentelemetry-operator/pkg/collector/adapters/config_to_ports.go

Line 92 in 90e4d31

return ports, nil