Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Runtime event trace visualization #36870

Closed
wants to merge 6 commits into from
Closed

Runtime event trace visualization #36870

wants to merge 6 commits into from

Conversation

kpamnany
Copy link
Contributor

@kpamnany kpamnany commented Jul 31, 2020

This adds a bpftrace script containing handlers for many of the probes in the Julia runtime, and also adds some probes. The various handlers produce JSON output that can be read by Perfetto.

Here's what this looks like for parallel Fibonacci with 4 threads:
image

BUILD INSTRUCTIONS:

  • Create Make.user containing WITH_DTRACE = 1.

TODO:

  • Determine why there are misplaced end events and fix.
  • Cannot use task pointer as task ID as tasks can be reused.
  • Add more probes (start+finish inference, start+finish codegen, others?).

Cc: @vtjnash, @JeffBezanson

src/trace.c Outdated Show resolved Hide resolved
src/trace.c Outdated Show resolved Hide resolved
@vilterp
Copy link
Contributor

vilterp commented Oct 6, 2021

May be worth trying to feed this info into Catapult, the tracing UI built into Chrome, so we don't have to write our own. The format is documented here: https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/preview

The Go runtime uses this to visualize scheduling and GC events. Here's an example debugging session: https://about.sourcegraph.com/go/an-introduction-to-go-tool-trace-rhys-hiltner/

@kpamnany
Copy link
Contributor Author

kpamnany commented Nov 3, 2021

With the current set of jl_trace() calls, the trace file generated from julia --threads=4 --trace-file=julia_startup.xml looks like:

image

@kpamnany
Copy link
Contributor Author

kpamnany commented Jan 4, 2022

After some discussions, we decided that this will be reworked on top of #43453 as it will be possible to completely avoid overhead when not interested in tracing (without recompiling).

@kpamnany kpamnany changed the title Add event tracing mechanism Runtime event trace visualization Mar 4, 2022
@kpamnany
Copy link
Contributor Author

kpamnany commented Apr 12, 2022

For the added probes:

  • Type inference begin and end seem to properly demarcate inference but no contextual information is available at these probe points (i.e. what is being inferred).
  • Compilation begin and end seem to include codegen (begin and end) and LLVM optimization (begin and end) but from looking at the visualization, there is more than these two things happening during compilation.

I've also switched to using jl_object_id(task) for a unique task identifier (since the task pointer itself is not unique).

Hope to get suggestions for additional useful probe points, new/more useful probe parameters or any other changes. Otherwise, this is good to go.

@kpamnany
Copy link
Contributor Author

Here's a zoomed-in section of the Perfetto visualization for parallel fib with 4 threads, showing the newly added sections:

image

kpamnany and others added 6 commits October 5, 2022 14:08
Produces a JSON file that can be read by Perfetto, allowing for
visualization of Julia runtime activity across threads.
And remove taskq get/put probes (that code has moved to Julia).
Also, timestamps should be in us and not ns.
@kpamnany kpamnany closed this by deleting the head repository Nov 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants