LLVM Opt Benchmark

LLVM Opt Benchmark is an LLVM IR dataset for data-driven compiler optimization research. This repository is also used by LLVM developers to evaluate the impact of their patches on real-world applications.

Don't submit PR to add new benchmarks. You can request new open-source C/C++/Rust repos here.

Please cite this work with the following BibTex entry:

@misc{opt-benchmark,
  title = {LLVM Opt Benchmark},
  url = {https://github.com/dtcxzyw/llvm-opt-benchmark},
  author = {Yingwei Zheng},
  year = {2023},
}

FAQs

LLVM developers use this corpus to assess the impact of your patches on real-world applications. If you see a link to this repository in your PR, it means that the target PR demonstrates some performance regressions or improvements caused by your changes. Here are some common questions you may have:

How can I reproduce the regression locally?

You should be able to reproduce the regression locally in the following steps:

# Apply your patch and rebuild opt.
...
# Download the source IR. Note that you should replace `optimized` with `original`.
wget https://raw.githubusercontent.com/dtcxzyw/llvm-opt-benchmark/refs/heads/main/bench/<program_name>/original/<file_name>.ll
# Run opt to generate the optimized IR.
bin/opt -O3 -disable-loop-unrolling -vectorize-loops=false -vectorize-slp=false -S <file_name>.ll -o opt.ll

Note that you don't need to clone the whole repository.

How can I evaluate my patch on this benchmark locally?

It is not recommended, as you can use the online service to evaluate your patch on GitHub if you have commit access to the LLVM repository.

You can use python3 ./scripts/gen_optimized.py bench <path-to-opt>. It will update the optimized IR files. Then you can review the diff with git.

The compile-time evaluation shows a huge impact on some files. What should I do?

Don't worry about it. If it doesn't affect the compile-time of the parent projects, it is generally acceptable. Otherwise, you may need to adjust the threshold or just handle simple cases.

What should I do when I see a regression?

Don't panic. Perfect is the enemy of good. We never ask the contributors to fix all the regressions before landing their patches.

Please follow the InstCombineContributorGuide to generalize your patch to cover the regression. If it doesn't work, try to find the pattern and file a separate issue. If it is hard to be caught by a separate transformation, try to bail out on the regression case. If we cannot make it better, the patch can still be accepted if the net effect is positive. Ask your reviewer to help you with the decision.

My method is expensive in compile time. But it shows some optimization opportunities. Should I abandon it?

Though we cannot accept the patch, we still encourage you to explore alternative approaches to handle the exposed optimization opportunities. As the distribution of the real-world code is not uniform, in general, a simple heuristic is good enough to cover most of the cases.

The evaluation result shows my patch has no effect on the benchmark. What does it mean?

We ask the issue reporter and the contributor to provide a motivating example from real-world scenarios. This benchmark only provides additional evidences to support the claim. It is highly recommended to run this benchmark if the real-world use case is missing, or it is found by fuzzers and super-optimizers. See also InstCombineContributorGuide.

The following patches may not be suitable for this benchmark:

• SLPVectorizer/LoopVectorize/LoopUnroll patches. Vectorization and loop unrolling are disabled since the diff is huge and hard to review. The performance is highly dependent on the target machine so the running time may be more representative.
• Sanitizer/Instrumentation/GPU patches. The related patterns are not included in this corpus.
• Patches which handle scalable vectors. This corpus only contains fixed-width vectors (generated from X86 intrinsics).

Do the regressions in IR diff imply the run-time performance regressions?

Not necessarily. The IR diff is only a proxy for the run-time performance. Generally fewer instructions at IR level implies better analysis result and less instructions at run-time. However, it depends on the target micro-architecture and the LLVM CodeGen components. For example, a canonicalization in InstCombine may cause the SelectionDAG to not recognize certain patterns, leading to bad codegen. Please refer to llvm-codegen-benchmark for frequent isel patterns. Anyway, the run-time performance should be the golden metric. The IR diff only helps us to find the root cause of regressions.

In addition, most of IR snippets are not the hot paths in the real-world applications. I choose to keep all the source IR files instead of only keeping the hot spots, as it is useful for monitoring the code size changes, which is also critical for the frontend performance on modern devices. Another reason is that we cannot find the hot paths in large applications like LLVM and verilator-generated simulators. BTW the training data for PGO in some programs is unavailable or highly biased, you know :).

The IR diff looks weird. It contains some invalid instructions. Is it a bug?

Many IR diffs only change the name of instructions and basic blocks. Previously, I used llvm-diff to reduce meaningless changes. However, it is slow and ineffective. Now I use a heuristic name-remapping algorithm to reduce the noise. The algorithm can reduce up to 70% of line changes. However, as it works on textual diff and does not understand the semantics of LLVM IR, it may produce some invalid instructions. Please check the raw diff in the previous commit pre-commit: Update.

The IR diff contains hundreds of file changes. How can I review it efficiently?

To fit the GitHub's limit of diff rendering, only part of the files are picked to be committed. It is chosen by a heuristic algorithm to improve the diversity of the dataset.

In the diff mode, a summary of the diff is also provided. It contains some key information to allow you to quickly review the changes:

• The number of files changed, lines added and removed (provided by git diff --shortstat). It is different from the numbers on the GitHub page, as it counts the statistics before diff reduction.
• A summary of the top-10 LLVM statistics changes.
• The number of line changes in each file (provided by git show <base>..HEAD --numstat --oneline). You can use this to quickly find the file with the most line additions or deletions (e.g., cat log | awk '{print $1 - $2, $3}' | sort -n).
• A summary from LLM (powered by Qwen). It provides a high-level overview of the changes. However, it always gives a positive response, so it may not be very useful. You can use it to find the files that are worth reviewing in detail.

From my own experience, the patterns are likely to be similar in the same project. So you can skip the whole project after you review the first few files in the same project. If your patch optimizes the C++/Rust standard library and other widely-used libraries, you can also skip the files with similar bb names in the hunk header (e.g., _ZNSt6vector...).

The IR diff is totally unrelated to my patch. Why?

Your changes may break existing optimizations. Please reproduce it locally and try to provide a minimal phase-ordering regression test. Then follow the instructions for dealing with regressions above.

Online services (previously hosted by PLCT Lab, ISCAS/currently hosted by SUSTech ARiSE Lab)

Special Acknowledgement: Thank @goldsteinn for providing additional computational resources to meet the growing demand for testing!

• Fuzzy DAG matching

  Please file an issue to provide LLVM IR with a single function. I will add the grep label to trigger CI.

  Example: #1072

• Middle-end optimization pre-commit testing

  Ping me if you want to see what is affected by your PR. It is useful for reviewers to find potential performance regressions and new optimization opportunities.

  For convenience, all llvm members are authorized to request pre-commit tests in #1312. Some basic PR editing commands are also supported by leaving a comment starts with /:

  • /close : Close the PR
  • /reopen: Reopen the PR
  • /add-label labels: Add labels (separated by comma). Available labels: reviewed, regression, crash, hang and miscompilation.
  • /remove-label labels: Remove labels.

• Codegen pre-commit testing

  See also llvm-codegen-benchmark.

• Weekly coverage report:

  https://dtcxzyw.github.io/llvm-opt-benchmark/

Benchmark List

Currently, this repository contains the following libraries/applications:

Name	Language	Stars	Last Updated	Active Files
abc	C		2025-01-02	940
bdwgc	C		2025-02-01	4
box2d	C		2025-01-27	81
brotli	C		2025-01-31	20
c3c	C		2024-04-16	62
chibicc	C		2020-12-07	8
cjson	C		2024-09-23	2
clamav	C		2025-02-03	226
cmake	C		2025-02-04	624
coremark	C		2023-01-24	3
cpython	C		2025-02-03	244
curl	C		2025-02-03	108
darktable	C		2025-02-03	390
ffmpeg	C		2025-05-19	1890
flac	C		2025-02-03	46
freetype	C		2025-01-28	30
git	C		2025-02-03	325
graphviz	C		2025-02-10	230
hdf5	C		2025-02-14	337
hwloc	C		2025-02-13	45
jemalloc	C		2025-02-13	46
jq	C		2025-02-16	41
kcp	C		2024-12-01	1
lean4	C		2025-05-20	1046
libdeflate	C		2025-01-20	12
libevent	C		2025-02-03	26
libjpeg-turbo	C		2024-12-18	74
libpng	C		2025-02-12	14
libquic	C		2016-09-22	369
libsodium	C		2025-01-26	78
libuv	C		2025-02-17	23
libwebp	C		2025-01-30	98
linux	C		2024-02-29	1185
lua	C		2025-01-29	29
luajit	C		2025-01-13	65
lvgl	C		2025-02-17	126
lz4	C		2025-02-03	10
memcached	C		2025-02-04	25
mimalloc	C		2025-02-17	15
miniaudio	C		2023-11-15	1
nanosvg	C		2024-12-19	1
nuklear	C		2025-02-07	1
nuttx	C		2024-03-04	74
ompi	C		2025-02-14	225
oniguruma	C		2025-02-11	18
openblas	C		2025-02-17	330
openssl	C		2025-02-18	1027
osqp	C		2025-02-13	17
php-src	C		2025-02-17	342
portaudio	C		2025-02-08	11
postgres	C		2025-02-18	791
qemu	C		2025-02-16	49
qoi	C		2025-02-12	1
quickjs	C		2024-07-27	8
raylib	C		2025-02-17	7
redis	C		2025-02-16	145
riscv-isa-sim	C		2025-02-12	928
ruby	C		2025-02-18	175
sdl	C		2025-05-19	275
slurm	C		2025-02-17	294
sqlite	C		2025-02-18	3
stb	C		2024-11-08	17
sundials	C		2024-12-20	185
wireshark	C		2025-02-18	1527
wolfssl	C		2025-02-17	39
yyjson	C		2025-02-12	1
zlib	C		2025-02-13	13
zstd	C		2025-02-13	30
abseil-cpp	C++		2025-02-15	329
annoy	C++		2024-07-28	1
arrow	C++		2025-02-17	166
assimp	C++		2025-02-17	206
boost	C++		2024-10-25	354
bullet3	C++		2025-01-29	196
casadi	C++		2025-02-18	198
ceres-solver	C++		2025-02-17	120
cpp-httplib	C++		2025-02-17	1
crow	C++		2025-02-10	13
csmith	C++		2023-11-02	59
cvc5	C++		2025-02-17	678
cxxopts	C++		2025-01-14	1
double-conversion	C++		2025-02-14	7
draco	C++		2025-01-28	81
duckdb	C++		2025-02-18	217
eastl	C++		2023-08-16	85
entt	C++		2025-02-14	72
evmone	C++		2025-09-29	28
faiss	C++		2025-02-14	160
flatbuffers	C++		2025-02-10	35
fmt	C++		2025-02-14	26
folly	C++		2025-02-17	229
g2o	C++		2025-02-09	122
glog	C++		2025-02-16	20
glslang	C++		2024-06-25	37
gromacs	C++		2025-02-24	769
grpc	C++		2025-02-24	314
gsl	C++		2025-02-14	12
harfbuzz	C++		2025-02-23	13
hermes	C++		2023-12-15	223
hyperscan	C++		2023-04-19	197
icu	C++		2025-02-21	426
imgui	C++		2025-02-22	5
ipopt	C++		2025-02-23	104
json	C++		2025-02-21	77
jsonnet	C++		2025-02-23	17
libcxx	C++		2025-05-20	86
libigl	C++		2025-05-14	519
libphonenumber	C++		2025-02-13	33
libzmq	C++		2024-12-30	74
lief	C++		2025-02-23	321
lightgbm	C++		2025-02-24	31
llama.cpp	C++		2025-02-23	37
llvm-project	C++		2025-02-03	2152
lodepng	C++		2024-12-28	3
luau	C++		2025-02-21	151
meshlab	C++		2024-02-13	199
meshoptimizer	C++		2025-02-21	15
minetest	C++		2024-03-26	308
mitsuba3	C++		2024-03-22	152
mixbox	C++		2022-12-16	1
mold	C++		2025-02-21	85
msdfgen	C++		2024-01-06	15
msgpack-c	C++		2025-02-21	19
nanobind	C++		2025-02-21	29
ncnn	C++		2025-02-20	359
nghttp2	C++		2025-02-18	16
ninja	C++		2025-02-19	58
nix	C++		2024-03-06	211
node	C++		2023-12-17	148
nori	C++		2023-11-15	45
open3d	C++		2025-04-03	384
open_spiel	C++		2024-08-27	252
opencc	C++		2025-02-12	24
opencolorio	C++		2025-02-10	178
opencv	C++		2025-02-25	1506
openexr	C++		2025-02-18	145
openimageio	C++		2025-02-25	103
openjdk	C++		2024-07-16	1080
openusd	C++		2024-07-24	862
openvdb	C++		2023-12-06	33
ozz-animation	C++		2025-01-19	38
pbrt-v4	C++		2025-01-30	60
pcg-cpp	C++		2022-04-08	6
pocketpy	C++		2024-06-20	26
proj	C++		2025-02-22	143
protobuf	C++		2023-12-15	117
proxy	C++		2024-05-22	5
proxygen	C++		2023-12-16	80
pugixml	C++		2025-02-19	1
quantlib	C++		2024-09-10	873
quest	C++		2025-02-08	6
re2	C++		2023-12-14	16
readerwriterqueue	C++		2024-07-09	2
recastnavigation	C++		2024-01-28	45
rocksdb	C++		2025-02-26	319
sentencepiece	C++		2025-02-27	51
simdjson	C++		2025-02-21	1
snappy	C++		2024-08-17	1
soc-simulator	C++		2024-06-25	5
spdlog	C++		2025-02-11	7
stockfish	C++		2024-03-03	14
taskflow	C++		2025-02-21	39
tev	C++		2024-01-12	22
tinygltf	C++		2025-01-22	1
tinympc	C++		2025-02-11	8
tinyobjloader	C++		2025-01-29	1
tinyrenderer	C++		2025-02-21	4
tomlplusplus	C++		2025-02-27	1
vcpkg-tool	C++		2025-02-27	139
velox	C++		2023-12-15	167
verilator	C++		2025-03-02	141
wasmedge	C++		2024-07-15	66
xgboost	C++		2025-03-01	103
yalantinglibs	C++		2023-12-17	52
yaml-cpp	C++		2025-01-24	30
yoga	C++		2025-02-27	13
yosys	C++		2025-03-01	310
z3	C++		2025-02-28	799
zfp	C++		2025-02-12	35
zxing-cpp	C++		2025-02-19	94
actix-web	Rust		2024-04-15	111
anki	Rust		2024-06-24	8
clap	Rust		2024-03-01	19
coreutils	Rust		2024-04-23	706
deku	Rust		2025-05-16	3
delta-rs	Rust		2024-04-23	118
diesel	Rust		2024-03-01	223
egg	Rust		2024-08-30	15
elfshaker	Rust		2025-05-09	16
fish-shell	Rust		2025-05-19	25
foundations	Rust		2025-05-19	15
html5ever	Rust		2023-09-06	42
hyper	Rust		2024-03-02	4
image	Rust		2024-02-22	16
influxdb	Rust		2024-03-01	46
jiff	Rust		2025-05-18	16
json	Rust		2024-01-11	15
just	Rust		2024-04-01	16
log	Rust		2024-02-29	1
logos	Rust		2024-06-10	31
meilisearch	Rust		2024-06-25	43
mini-lsm	Rust		2024-02-26	45
nom	Rust		2024-04-21	6
ockam	Rust		2024-04-22	278
pingora	Rust		2025-05-09	124
polars	Rust		2025-05-19	292
pyo3	Rust		2024-06-24	30
qdrant	Rust		2024-03-19	43
quiche	Rust		2025-05-19	62
quinn	Rust		2025-05-20	48
raft-rs	Rust		2025-02-28	26
rand	Rust		2024-02-18	8
rayon	Rust		2024-02-27	20
regex	Rust		2024-01-10	44
ring	Rust		2024-03-03	16
ripgrep	Rust		2024-03-27	89
ropey	Rust		2024-04-08	15
ruff	Rust		2025-05-19	420
rust-analyzer	Rust		2024-04-22	468
rust-base64	Rust		2024-03-01	7
rustfmt	Rust		2024-03-04	16
rustls	Rust		2024-03-07	15
salsa	Rust		2025-05-19	16
serde	Rust		2024-01-08	1
smol	Rust		2024-03-04	16
softposit-rs	Rust		2022-12-14	11
statrs	Rust		2024-06-24	15
syn	Rust		2024-01-13	16
tikv	Rust		2025-05-20	10
tokenizers	Rust		2024-05-06	16
tokio	Rust		2024-03-04	41
tree-sitter	Rust		2024-03-08	85
turborepo	Rust		2024-10-03	59
typst	Rust		2024-03-25	81
unicode-normalization	Rust		2024-03-03	2
uv	Rust		2025-05-19	538
wasmi	Rust		2025-05-17	85
wasmtime	Rust		2024-04-22	298
yara-x	Rust		2025-08-08	143
zed	Rust		2024-10-04	1214