Unable to run Perfcollector .NET 8 core

[samyaghosh2]

Type of issue

Missing information

Description

To enable the perf collector on K8 Linux container, on Linux VM (in AKS Linux Gen05 nodes) for .NET Core 8, we followed the following steps (following this guide https://learn.microsoft.com/en-us/dotnet/core/diagnostics/trace-perfcollect-lttng) still we are getting following error.

Running "/usr/bin/lttng add-context --kernel -t pid -t procname"
Error: pid: Kernel tracer not available
Error: procname: Kernel tracer not available

Here are the steps for reference:

1 curl -OL https://aka.ms/perfcollect
2 apt updat
3 apt update
4 apt install curl
5 curl -OL https://aka.ms/perfcollect
6 chmod +x perfcollect
7 sudo ./perfcollect install
8 dotnet tool install --global dotnet-symbol
9 mkdir mySymbolsNet8
10 ~/.dotnet/tools/dotnet-symbol --output ./mySymbolsNet8/ /usr/share/dotnet/shared/Microsoft.NETCore.App/8.0.0/lib*.so
11 cp ./mySymbolsNet8/* /usr/share/dotnet/shared/Microsoft.NETCore.App/8.0.0/
12 apt-get install linux-tools-5.15.0-1041-azure -y
13 cp /usr/lib/linux-tools/5.15.0-1041-azure/perf /usr/bin/perf
14 ps -ef | grep -i dotnet
15 export DOTNET_EnableWriteXorExecute=0
16 sudo ./perfcollect collect a -pid 166
perfcollector.log

Please provide us the missing/updated steps

Page URL

https://learn.microsoft.com/en-us/dotnet/core/diagnostics/trace-perfcollect-lttng

Content source URL

https://github.com/dotnet/docs/blob/main/docs/core/diagnostics/trace-perfcollect-lttng.md

Document Version Independent Id

ac6cc555-a033-766d-0597-bd9bda98820f

Article author

@tommcdon

Metadata

• ID: 7dd230f8-907f-5f07-e3ab-7ad37fee3aa4
• Product: dotnet-fundamentals

[tommcdon]

Hi @samyaghosh2, please see https://learn.microsoft.com/en-us/dotnet/core/diagnostics/trace-perfcollect-lttng#collect-in-a-docker-container for information on collecting diagnostics in a container. Please let me know if this resolves the issue.
cc @brianrob

[samyaghosh2]

Hi @tommcdon cc @brianrob ,

We tried exact steps in the document (https://learn.microsoft.com/en-us/dotnet/core/diagnostics/trace-perfcollect-lttng#collect-in-a-docker-container), still it is giving the same error as mentioned earlier .

One more information we are using "containerd" as runtime our AKS cluster.

[tommcdon]

@samyaghosh2 can you verify:

1. The container has the SYS_ADMIN capability
2. DOTNET_PerfMapEnabled is set to 1 for the target process
3. If the above is all correct, please try adding the -nolttng argument to the perfcollect script.

[ankishagarwal]

Tried with the flag -nolttng

Yes the container has the SYS_ADMIN capabilities,
Also all the four following env variables are set:
export DOTNET_PerfMapEnabled=1
export DOTNET_EnableEventLog=1
export DOTNET_EnableWriteXorExecute=0
export DOTNET_ZapDisable=1

We got the following error:

Running "/usr/lib/linux-tools/5.15.0-1053-azure/perf record -k 1 -g -a -F 1000 -e cpu-clock"
Error:
Access to performance monitoring and observability operations is limited.
Consider adjusting /proc/sys/kernel/perf_event_paranoid setting to open
access to performance monitoring and observability operations for processes
without CAP_PERFMON, CAP_SYS_PTRACE or CAP_SYS_ADMIN Linux capability.
More information can be found at 'Perf events and tool security' document:
https://www.kernel.org/doc/html/latest/admin-guide/perf-security.html
perf_event_paranoid setting is 4:
-1: Allow use of (almost) all events by all users
Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK

= 0: Disallow raw and ftrace function tracepoint access
sudo ./perfcollect collect sampleTrace>= 1: Disallow CPU event access
= 2: Disallow kernel profiling
To make the adjusted perf_event_paranoid setting permanent preserve it
in /etc/sysctl.conf (e.g. kernel.perf_event_paranoid = )

[brianrob]

If I recall correctly, the behavior you're seeing is expected because the kernel module cannot be built and installed inside the container. If you need kernel-level data, then you will need to capture on the node instead of inside the container.

[tommcdon]

@brianrob should we update the docs to clearly state that perfcollect is not supported in containers or are there other caveats to be aware of?