trace cmd

Capturing traces

There are several scripts in the sample directory to help with tracing.

• trace-cmd-setup.sh - Set up tracing directory and trace-cmd permissions. May require root.
• trace-cmd-start-tracing.sh - Start tracing into kernel ring buffer (2000kb per CPU)
• trace-cmd-status.sh - Check tracing status
• trace-cmd-capture.sh - Capture trace snapshot for last few seconds, continue tracing
• trace-cmd-stop-tracing.sh - Stop tracing

To use above shell scripts

1. Run trace-cmd-setup.sh
2. Run trace-cmd-start-tracing.sh
3. Launch app
4. When interesting thing happens, run trace-cmd-capture.sh
5. Goto step 4 (until you run out of interesting events)
6. Stop tracing with trace-cmd-stop-tracing.sh
7. Launch gpuvis on capture .dat trace files

On systems running Linux and having an Intel GPU, additional GPU generated data can also be captured.

Example directly using trace-cmd record

Trace-cmd command to capture a trace.dat file with amdgpu/i915 and kernel schedule information.
NOTE: the i915_gem_request_submit, i915_gem_request_in, i915_gem_request_out tracepoints require the CONFIG_DRM_I915_LOW_LEVEL_TRACEPOINTS Kconfig option to be enabled.

trace-cmd record -i \
    -e "sched:sched_switch" \
    -e "sched:sched_process_exec" \
    -e "sched:sched_process_fork" \
    -e "sched:sched_process_exit" \
    -e "amdgpu:amdgpu_vm_flush" \
    -e "amdgpu:amdgpu_cs_ioctl" \
    -e "amdgpu:amdgpu_sched_run_job" \
    -e "*fence:*fence_signaled" \
    -e "drm:drm_vblank_event" \
    -e "drm:drm_vblank_event_queued" \
    -e "i915:i915_flip_request" \
    -e "i915:i915_flip_complete" \
    -e "i915:intel_gpu_freq_change" \
    -e "i915:i915_gem_request_add" \
    -e "i915:i915_gem_request_submit" \
    -e "i915:i915_gem_request_in" \
    -e "i915:i915_gem_request_out" \
    -e "i915:intel_engine_notify" \
    -e "i915:i915_gem_request_wait_begin" \
    -e "i915:i915_gem_request_wait_end" \
    sleep 4

View trace.dat events with trace-cmd (and latency + tgid information):

trace-cmd report -l -g

Note: amdgpu events require Linux kernel v4.12+

trace-cmd links

• trace-cmd: A front-end for Ftrace
• trace-cmd tips
• trace-cmd tracing
• Brendan Gregg's Linux Performance overview (Super useful)
• upstream source: git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git
• trace-cmd capture is integrated into steamvr. Enable GPU profiling then press F6 to capture and launch gpuvis.

To capture tgid (thread group information), you will need this modified trace-cmd:

https://github.com/mikesart/trace-cmd

Several of the trace-cmd changes have been submitted upstream (waiting for merge):
[[https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1467935.html]]

Patches for tgid information will be submitted after above patches are accepted. Specifically, these commits add tgid information to the trace file:

* Add tgid support to trace record and report
* Read tgid info from procfs if saved_tgids file not found
* When saved_tgids file is empty, read tgid info from procfs

Gpuvis tgid screenshots

Thread group information in main gpuvis window

Thread group information in Info for 'trace.dat' (Under Options Menu)

Context menu item for hiding tgid process rows