Compile Time Tracking for Key Workloads

[whitneywhtsang]

Start tracking compile time to identify regressions and bottlenecks.
Decided to track compile time at the component level for simplicity, we can consider tracking per stages (ttir, ttgir, etc) or per Triton passes in the future.
Components to track: PyTorch (if not starting from Triton kernels, e.g., Flex Attention), Triton, LLVM opt, SPIRV-LLVM translator, and IGC.

Idea: instead of running benchmarks in a new workflow, one can reuse triton-benchmark.yml, and collect extra info like compile time and IR sizes on top of kernel performance.

Please assign to Egor when compile time tracking is implemented, and Egor can work on visualizing the data in Grafana.

[whitneywhtsang]

FYI @vlad-penkin @Egor-Krivov

[anmyachev]

May be useful: triton-lang/triton#5957

[Egor-Krivov]

@whitneywhtsang

Currently based on #4777

results look something like:
{ "time": 0.6139442920684814, "asm": { "source": 170, "ttir": 89, "ttgir": 90, "llir": 670, "spv": 19856 } So we can track total compilation time and number of lines in the IR. Is that enough, or we need deeper granularity?

I think ideally we could track our triton-benchmarks.yaml and maybe some simple pytorch script with inference or LLM finetuning. Then will have both some reproducible small cases and end to end example.

Also, currently we'll be able to provide one total compile time across all configurations for benchmark script. Is that enough?

[whitneywhtsang]

@Egor-Krivov Let's start with that. Ideally, we can have a way to break it down to per component, and per shape/autoconfig.
Agree to start with tracking triton-benchmarks.yaml.