Auto-instrumentation cp operation not permitted

[smallc2009]

Component(s)

instrumentation

What happened?

Description

when I add annotation to our .net application, it is not able to inject the library and produces the error logs from init container.

I set up USER 64189 in Dockerfile. I'm running pod with non-root user, the following securityContext options are set in our helm deployment template.

allowPrivilegeEscalation: false
privileged: false
readOnlyRootFilesystem: false
runAsNonRoot: true

Steps to Reproduce

Enable RunAsNonRoot option

Expected Result

inti container copy library to the application pod without errors

Actual Result

Kubernetes Version

1.26.9

Operator version

0.93.0

Collector version

0.93.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

Log output

2023-04-04T00:57:56-04:00 cp: can't preserve ownership of '/otel-auto-instrumentation/./store/x64/net7.0/microsoft.extensions.configuration.binder/7.0.0/lib/net7.0': Operation not permitted
2023-04-04T00:57:56-04:00 cp: can't preserve ownership of '/otel-auto-instrumentation/./store/x64/net7.0/microsoft.extensions.configuration.binder/7.0.0/lib': Operation not permitted
2023-04-04T00:57:56-04:00 cp: can't preserve ownership of '/otel-auto-instrumentation/./store/x64/net7.0/microsoft.extensions.configuration.binder/7.0.0': Operation not permitted
2023-04-04T00:57:56-04:00 cp: can't preserve ownership of '/otel-auto-instrumentation/./store/x64/net7.0/microsoft.extensions.configuration.binder': Operation not permitted
2023-04-04T00:57:56-04:00 cp: can't preserve ownership of '/otel-auto-instrumentation/./store/x64/net7.0/mongodb.driver.core/2.13.3/runtimes/win/native/libzstd.dll': Operation not permitted
2023-04-04T00:57:56-04:00 cp: can't preserve ownership of '/otel-auto-instrumentation/./store/x64/net7.0/mongodb.driver.core/2.13.3/runtimes/win/native/snappy32.dll': Operation not permitted
2023-04-04T00:57:56-04:00 cp: can't preserve ownership of '/otel-auto-instrumentation/./store/x64/net7.0/mongodb.driver.core/2.13.3/runtimes/win/native/snappy64.dll': Operation not permitted
2023-04-04T00:57:56-04:00 cp: can't preserve ownership of '/otel-auto-instrumentation/./store/x64/net7.0/mongodb.driver.core/2.13.3/runtimes/win/native': Operation not permitted
2023-04-04T00:57:56-04:00 cp: can't preserve ownership of '/otel-auto-instrumentation/./store/x64/net7.0/mongodb.driver.core/2.13.3/runtimes/win': Operation not permitted
2023-04-04T00:57:56-04:00 cp: can't preserve ownership of '/otel-auto-instrumentation/./store/x64/net7.0/mongodb.driver.core/2.13.3/runtimes': Operation not permitted
2023-04-04T00:57:56-04:00 cp: can't preserve ownership of '/otel-auto-instrumentation/./store/x64/net7.0/mongodb.driver.core/2.13.3/lib/netstandard2.1/MongoDB.Driver.Core.dll': Operation not permitted
2023-04-04T00:57:56-04:00 cp: can't preserve ownership of '/otel-auto-instrumentation/./store/x64/net7.0/mongodb.driver.core/2.13.3/lib/netstandard2.1': Operation not permitted
2023-04-04T00:57:56-04:00 cp: can't preserve ownership of '/otel-auto-instrumentation/./store/x64/net7.0/mongodb.driver.core/2.13.3/lib': Operation not permitted
2023-04-04T00:57:56-04:00 cp: can't preserve ownership of '/otel-auto-instrumentation/./store/x64/net7.0/mongodb.driver.core/2.13.3': Operation not permitted
2023-04-04T00:57:56-04:00 cp: can't preserve ownership of '/otel-auto-instrumentation/./store/x64/net7.0/mongodb.driver.core': Operation not permitted
2023-04-04T00:57:56-04:00 cp: can't preserve ownership of '/otel-auto-instrumentation/./store/x64/net7.0/microsoft.extensions.configuration/7.0.0/lib/net7.0/Microsoft.Extensions.Configuration.dll': Operation not permitted
2023-04-04T00:57:56-04:00 cp: can't preserve ownership of '/otel-auto-instrumentation/./store/x64/net7.0/microsoft.extensions.configuration/7.0.0/lib/net7.0': Operation not permitted
2023-04-04T00:57:56-04:00 cp: can't preserve ownership of '/otel-auto-instrumentation/./store/x64/net7.0/microsoft.extensions.configuration/7.0.0/lib': Operation not permitted
2023-04-04T00:57:56-04:00 cp: can't preserve ownership of '/otel-auto-instrumentation/./store/x64/net7.0/microsoft.extensions.configuration/7.0.0': Operation not permitted
2023-04-04T00:57:56-04:00 cp: can't preserve ownership of '/otel-auto-instrumentation/./store/x64/net7.0/microsoft.extensions.configuration': Operation not permitted
2023-04-04T00:57:56-04:00 cp: can't preserve ownership of '/otel-auto-instrumentation/./store/x64/net7.0': Operation not permitted
2023-04-04T00:57:56-04:00 cp: can't preserve ownership of '/otel-auto-instrumentation/./store/x64': Operation not permitted
2023-04-04T00:57:56-04:00 cp: can't preserve ownership of '/otel-auto-instrumentation/./store': Operation not permitted
2023-04-04T00:57:56-04:00 cp: can't preserve times of '/otel-auto-instrumentation/.': Operation not permitted
2023-04-04T00:57:56-04:00 cp: can't preserve ownership of '/otel-auto-instrumentation/.': Operation not permitted
2023-04-04T00:57:56-04:00 cp: can't preserve permissions of '/otel-auto-instrumentation/.': Operation not permitted

Additional context

No response

[pavolloffay]

cc) @iblancasa

[iblancasa]

@pavolloffay thanks for the heads-up!

Can you check if #2695 fixes your issue?

[berry2012]

Looks like it is not resolved yet. Reproduced the issue with the ffg:

    spec:
      # securityContext:
      #   seccompProfile:
      #     type: RuntimeDefault
      #   runAsNonRoot: true 
      #   runAsUser: 5678    
      containers:
      - image: XYZ/app
        name: flask-app
        # securityContext:
        #   allowPrivilegeEscalation: false
        #   readOnlyRootFilesystem: true # or false
        #   capabilities:
        #     drop:
        #       - ALL       

Container user 5678

Error

     cp: preserving times for '/otel-auto-instrumentation-python/.': Operation not permitted

[iblancasa]

Looks like it is not resolved yet. Reproduced the issue with the ffg:

1. What does it mean ffg?
2. What version of the operator are you using?
3. Are you using the default container images for the autoinstrumentation?
4. The configuration you provided seems it have a lot of commented fields. Can you try the examples we have as part of the E2E tests to see if you reproduce the issues?

[jurgenroels]

Hi @iblancasa ,

We are encountering the same issue when trying to instrument Nginx:

Some additional information:

• Operator version: v0.104.0
• We are using a default container image for Nginx: "NGINX_VERSION=1.23.3" unprivileged (We also tried other images)

Are there any suggestions on how to fix this?

[iblancasa]

Would you be able to provide a reproducer? @jurgenroels

[jurgenroels]

@iblancasa What can we use to make the reproducer? This way we can see what it is collecting and if we can share it.

[iblancasa]

@jurgenroels if you can reproduce it using kind and provide the image, Instrumentation CR and Deployment used it would help.

[jurgenroels]

@iblancasa in attachment you can find the files to reproduce. We also tested other Nginx images
We are using TKGI.
The files: yaml.zip

[iblancasa]

Please, provide one of the images or Dockerfiles of the container image to instrument.

[jurgenroels]

This is the link: https://hub.docker.com/r/nginxinc/nginx-unprivileged

[excitedbumpkin]

My guess would be "cp -ar ..." needs to be changed to "cp -r ..."

opentelemetry-operator/pkg/instrumentation/nginx.go

Line 189 in 58a3a4f

cp -ar /opt/opentelemetry/* ${NGINX_AGENT_DIR_FULL} \n

This has been done for other instrumentations but apparently not for nginx.

[iblancasa]

I was not able to reproduce but... yes. There were other instrumentation where something similar happened.

[jurgenroels]

Thanks @excitedbumpkin and @iblancasa
We'll try it when the release is available

[tobgen]

I would like to reopen this one.
Because it applies not only for nginx, the nodejs image have the same issue as this one.

"cp: can't preserve permissions of '/otel-auto-instrumentation-nodejs/.': Operation not permitted"

This will fail on all containers running as a non-root user.

[iblancasa]

I would like to reopen this one. Because it applies not only for nginx, the nodejs image have the same issue as this one.

"cp: can't preserve permissions of '/otel-auto-instrumentation-nodejs/.': Operation not permitted"

This will fail on all containers running as a non-root user.

Please, create a new issue for that. It should be a different issue because the NodeJS libraries are copied like:

Command:   []string{"cp", "-r", "/autoinstrumentation/.", nodejsInstrMountPath},