Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add autoinstrumentation of NodeJS #507

Merged
merged 20 commits into from
Nov 10, 2021
42 changes: 25 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,27 +155,16 @@ EOF

### OpenTelemetry auto-instrumentation injection

The operator can inject and configure OpenTelemetry auto-instrumentation libraries. At this moment, the operator can inject only OpenTelemetry [Java auto-instrumentation](https://github.com/open-telemetry/opentelemetry-java-instrumentation).
The operator can inject and configure OpenTelemetry auto-instrumentation libraries. Currently Java and NodeJS are supported.

The injection of the Java agent can be enabled by adding an annotation to the namespace, so that all pods within that namespace will get the instrumentation, or by adding the annotation to individual PodSpec objects, available as part of Deployment, Statefulset, and other resources.

```bash
instrumentation.opentelemetry.io/inject-java: "true"
```

The value can be
* `"false"` - do not inject
* `"true"` - inject and `Instrumentation` resource from the namespace.
* `"java-instrumentation"` - name of `Instrumentation` CR instance.

In addition to the annotation, the following `CR` has to be created. The `Instrumentation` resource provides configuration for OpenTelemetry SDK and auto-instrumentation.
To use auto-instrumentation, configure an `Instrumentation` resource with the configuration for the SDK and instrumentation.

```yaml
kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: java-instrumentation
name: my-instrumentation
spec:
exporter:
endpoint: http://otel-collector:4317
Expand All @@ -187,14 +176,33 @@ spec:
type: parentbased_traceidratio
argument: "0.25"
java:
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest # <1>
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
nodejs:
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest
EOF
```

1. Container image with [OpenTelemetry Java auto-instrumentation](https://github.com/open-telemetry/opentelemetry-java-instrumentation). The image must contain the Java agent JAR `/javaagent.jar`, and the operator will copy it to a shared volume mounted to the application container.

The above CR can be queried by `kubectl get otelinst`.

Then add an annotation to a pod to enable injection. The annotation can be added to a namespace, so that all pods within
that namespace wil get instrumentation, or by adding the annotation to individual PodSpec objects, available as part of
Deployment, Statefulset, and other resources.

Java:
```bash
instrumentation.opentelemetry.io/inject-java: "true"
```

NodeJS:
```bash
instrumentation.opentelemetry.io/inject-nodejs: "true"
```

The possible values for the annotation can be
* `"true"` - inject and `Instrumentation` resource from the namespace.
* `"my-instrumentation"` - name of `Instrumentation` CR instance.
* `"false"` - do not inject

## Compatibility matrix

### OpenTelemetry Operator vs. OpenTelemetry Collector
Expand Down
13 changes: 13 additions & 0 deletions apis/instrumentation/v1alpha1/instrumentation_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,11 @@ type InstrumentationSpec struct {
// +optional
// +operator-sdk:gen-csv:customresourcedefinitions.specDescriptors=true
Java JavaSpec `json:"java,omitempty"`

// NodeJS defines configuration for nodejs auto-instrumentation.
// +optional
// +operator-sdk:gen-csv:customresourcedefinitions.specDescriptors=true
NodeJS NodeJSSpec `json:"nodejs,omitempty"`
}

// JavaSpec defines Java SDK and instrumentation configuration.
Expand All @@ -52,6 +57,14 @@ type JavaSpec struct {
Image string `json:"image,omitempty"`
}

// NodeJSSpec defines NodeJS SDK and instrumentation configuration.
type NodeJSSpec struct {
// Image is a container image with NodeJS SDK and autoinstrumentation.
// +optional
// +operator-sdk:gen-csv:customresourcedefinitions.specDescriptors=true
Image string `json:"image,omitempty"`
}

// Exporter defines OTLP exporter configuration.
type Exporter struct {
// Endpoint is address of the collector with OTLP endpoint.
Expand Down
16 changes: 16 additions & 0 deletions apis/instrumentation/v1alpha1/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 7 additions & 0 deletions bundle/manifests/opentelemetry.io_instrumentations.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,13 @@ spec:
description: Image is a container image with javaagent JAR.
type: string
type: object
nodejs:
description: NodeJS defines configuration for nodejs auto-instrumentation.
properties:
image:
description: Image is a container image with NodeJS SDK and autoinstrumentation.
type: string
type: object
propagators:
description: Propagators defines inter-process context propagation
configuration.
Expand Down
7 changes: 7 additions & 0 deletions config/crd/bases/opentelemetry.io_instrumentations.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,13 @@ spec:
description: Image is a container image with javaagent JAR.
type: string
type: object
nodejs:
description: NodeJS defines configuration for nodejs auto-instrumentation.
properties:
image:
description: Image is a container image with NodeJS SDK and autoinstrumentation.
type: string
type: object
propagators:
description: Propagators defines inter-process context propagation
configuration.
Expand Down
9 changes: 5 additions & 4 deletions pkg/instrumentation/annotation.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,15 +23,16 @@ import (
const (
// annotationInjectJava indicates whether java auto-instrumentation should be injected or not.
// Possible values are "true", "false" or "<Instrumentation>" name.
annotationInjectJava = "instrumentation.opentelemetry.io/inject-java"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you explain the intentions for changing this annotation?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I'm happy to split out the annotation change if it makes sense.

I found our current, SDK-agnostic (?) sdk.go uses this Java annotation. I need it to be able to support multiple languages and came up with this change. Any idea that could work better?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please explain why we need two annotations?

We can have "instrumentation.opentelemetry.io/inject-java", "instrumentation.opentelemetry.io/inject-node" and then `"instrumentation.opentelemetry.io/inject" - to just inject the basic configuration (users could use the operator as a control plane to manage instrumentations).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we want to allow multiple languages on the same pod as that shouldn't ever be a use case I think. While we could check the annotations and fail if there are multiple this split seemed to model more directly that only one language can be injected.

I made language required without thinking too much but could definitely remove that to allow the control plane only injection if this otherwise makes sense.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see much value of limiting it to only a single language, quite the opposite.

What I don't like is to have two annotations from the UX perspective.

	annotationInject   = "instrumentation.opentelemetry.io/inject"
	annotationLanguage = "instrumentation.opentelemetry.io/language"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe these sort of annotations are always copy-pasted or frameworked, so I don't think there is a usability difference between the two patterns. If we think there is a use case for multiple annotations, then it makes sense but for example having both otel.inject=true and otel.inject-java=true on the same pod would be redundant and I think that leads to some cognitive dissonance.

Great to hear everyone's ideas on the matter :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed with @pavolloffay offline and for now sticking to the current annotation format, but we'll need to address it soon. We can see the awkwardness here

https://github.com/open-telemetry/opentelemetry-operator/pull/507/files#diff-92f790ce871022e5e7de195a2f96235dac6ac3fb04337974f8584c7c020a67cbR48

where common config can be injected from different instrumentations in a surprising way. Having separate annotations for the instrumentation config and what languages to use in the future will hopefully solve that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 let's discuss it in a separate issue. maybe there are other ways we can mitigate the issue

annotationInjectJava = "instrumentation.opentelemetry.io/inject-java"
annotationInjectNodeJS = "instrumentation.opentelemetry.io/inject-nodejs"
)

// annotationValue returns the effective annotationInjectJava value, based on the annotations from the pod and namespace.
func annotationValue(ns metav1.ObjectMeta, pod metav1.ObjectMeta) string {
func annotationValue(ns metav1.ObjectMeta, pod metav1.ObjectMeta, annotation string) string {
// is the pod annotated with instructions to inject sidecars? is the namespace annotated?
// if any of those is true, a sidecar might be desired.
podAnnValue := pod.Annotations[annotationInjectJava]
nsAnnValue := ns.Annotations[annotationInjectJava]
podAnnValue := pod.Annotations[annotation]
nsAnnValue := ns.Annotations[annotation]

// if the namespace value is empty, the pod annotation should be used, whatever it is
if len(nsAnnValue) == 0 {
Expand Down
2 changes: 1 addition & 1 deletion pkg/instrumentation/annotation_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ func TestEffectiveAnnotationValue(t *testing.T) {
} {
t.Run(tt.desc, func(t *testing.T) {
// test
annValue := annotationValue(tt.ns.ObjectMeta, tt.pod.ObjectMeta)
annValue := annotationValue(tt.ns.ObjectMeta, tt.pod.ObjectMeta, annotationInjectJava)

// verify
assert.Equal(t, tt.expected, annValue)
Expand Down
68 changes: 68 additions & 0 deletions pkg/instrumentation/nodejs.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
// Copyright The OpenTelemetry Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package instrumentation

import (
"github.com/go-logr/logr"
corev1 "k8s.io/api/core/v1"

"github.com/open-telemetry/opentelemetry-operator/apis/instrumentation/v1alpha1"
)

const (
envNodeOptions = "NODE_OPTIONS"
nodeRequireArgument = " --require /otel-auto-instrumentation/autoinstrumentation.js"
)

func injectNodeJSSDK(logger logr.Logger, nodeJSSpec v1alpha1.NodeJSSpec, pod corev1.Pod) corev1.Pod {
// caller checks if there is at least one container
container := &pod.Spec.Containers[0]
idx := getIndexOfEnv(container.Env, envNodeOptions)
if idx == -1 {
container.Env = append(container.Env, corev1.EnvVar{
Name: envNodeOptions,
Value: nodeRequireArgument,
})
} else if idx > -1 {
if container.Env[idx].ValueFrom != nil {
// TODO add to status object or submit it as an event
logger.Info("Skipping NodeJS SDK injection, the container defines NODE_OPTIONS env var value via ValueFrom", "container", container.Name)
return pod
}
container.Env[idx].Value = container.Env[idx].Value + nodeRequireArgument
}
container.VolumeMounts = append(container.VolumeMounts, corev1.VolumeMount{
Name: volumeName,
MountPath: "/otel-auto-instrumentation",
})

pod.Spec.Volumes = append(pod.Spec.Volumes, corev1.Volume{
Name: volumeName,
VolumeSource: corev1.VolumeSource{
EmptyDir: &corev1.EmptyDirVolumeSource{},
}})

pod.Spec.InitContainers = append(pod.Spec.InitContainers, corev1.Container{
Name: initContainerName,
Image: nodeJSSpec.Image,
Command: []string{"cp", "-a", "/autoinstrumentation/.", "/otel-auto-instrumentation/"},
VolumeMounts: []corev1.VolumeMount{{
Name: volumeName,
MountPath: "/otel-auto-instrumentation",
}},
})

return pod
}
Loading