Cranelift: major performance regression (-14.93%) for spidermonkey-json

[bongjunj]

Tested with sightglass for spidermonkey-json: https://github.com/bytecodealliance/sightglass/blob/main/benchmarks/spidermonkey/spidermonkey-json.wasm

Phase	Base	Upstream	Relative Performance
Compilation	30125453533	29900200416	-0.75%
Instantiation	811322	803595.3	+0.96%
Execution	544499791.7	640038909.6	-14.93%

• The unit of measure is CPU cycle (sightglass default)
• Base is a wasmtime/cranelift variant with no mid-end ISLE optimization rules
• Upstream is the latest version of wasmtime/cranelift.

Expected Results

Spidermonkey represents a major workload for wasmtime.
Mid-end optimization should not regress performance.

Actual Results

The performance is severely regressed.

Versions and Environment

Cranelift version or commit: 8315a90ced0d01bdddfea92af514a6cd30da4abf

Operating system: Linux, x64

Architecture

lscpu

Architecture:                x86_64
  CPU op-mode(s):            32-bit, 64-bit
  Address sizes:             46 bits physical, 48 bits virtual
  Byte Order:                Little Endian
CPU(s):                      64
  On-line CPU(s) list:       0-63
Vendor ID:                   GenuineIntel
  Model name:                Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz
    CPU family:              6
    Model:                   85
    Thread(s) per core:      2
    Core(s) per socket:      16
    Socket(s):               2
    Stepping:                7
    CPU(s) scaling MHz:      33%
    CPU max MHz:             3900.0000
    CPU min MHz:             1200.0000
    BogoMIPS:                5800.00
    Flags:                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch
                             _perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 s
                             se4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 intel_ppin ssbd mba ibrs ibpb stibp ibrs_en
                             hanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_p
                             t avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts vnmi pku ospke avx512_vnni md_clear
                             flush_l1d arch_capabilities ibpb_exit_to_user
Virtualization features:
  Virtualization:            VT-x
Caches (sum of all):
  L1d:                       1 MiB (32 instances)
  L1i:                       1 MiB (32 instances)
  L2:                        32 MiB (32 instances)
  L3:                        44 MiB (2 instances)
NUMA:
  NUMA node(s):              2
  NUMA node0 CPU(s):         0-15,32-47
  NUMA node1 CPU(s):         16-31,48-63
Vulnerabilities:
  Gather data sampling:      Mitigation; Microcode
  Indirect target selection: Mitigation; Aligned branch/return thunks
  Itlb multihit:             KVM: Mitigation: VMX disabled
  L1tf:                      Not affected
  Mds:                       Not affected
  Meltdown:                  Not affected
  Mmio stale data:           Mitigation; Clear CPU buffers; SMT vulnerable
  Reg file data sampling:    Not affected
  Retbleed:                  Mitigation; Enhanced IBRS
  Spec rstack overflow:      Not affected
  Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop
  Srbds:                     Not affected
  Tsa:                       Not affected
  Tsx async abort:           Mitigation; TSX disabled
  Vmscape:                   Mitigation; IBPB before exit to userspace

Extra Info

[tschneidereit]

cc @cfallin @fitzgen

[cfallin]

Sorry, we had a long weekend (Fri/Mon off) during which this came in and I'm just catching up!

@bongjunj did you happen to bisect where the no-opt-vs-opt slowdown started? I can examine this if not, but thought I would ask first. (And, by the way, since this report looks auto-generated: if you're building tooling to do this, a bisection is the most useful bit of information for such issues -- thanks!)

[bongjunj]

@cfallin Thanks for the comment!

I was running DDMin on the mid-end ruleset (and it took a lot of time), and it figured out that, when the following rules are removed, the performance of Opt vs. Base is not only restored but also becomes superior.

Iteration: 10 times each
Base: 549317765.0 cycles
Opt: 547874348.3 cycles

(rule (simplify (iadd (fits_in_64 ty) (iconst ty (u64_from_imm64 k1)) (iconst ty (u64_from_imm64 k2)))) (subsume (iconst ty (imm64_masked ty (u64_wrapping_add k1 k2)))))
(rule (simplify (iadd ty (isub ty x (iconst ty (u64_from_imm64 k1))) (iconst ty (u64_from_imm64 k2)))) (iadd ty x (iconst ty (imm64_masked ty (u64_wrapping_sub k2 k1)))))
(rule (simplify (iadd ty k @ (iconst ty _) x)) (iadd ty x k))
(rule (simplify (sshr ty x (uextend _ y))) (sshr ty x y))
(rule (simplify (uextend (fits_in_64 wide) (iconst_u narrow k))) (subsume (iconst_u wide k)))

You can see our base version here: prosyslab@3149daa,
and the compared version at prosyslab@9b3e094

I guess constant foldings can sometimes break the assumptions that backends make (for example, what an IR should look like to emit an ideal x64 assembly...). Furthermore, as the suspected rules contains add/sub, it could be related to address or loop induction variable computations, which could affect branch/jump machine instructions.

[cfallin]

@bongjunj I wasn't able to reproduce this on main. It looks like your tested commits branch off of upstream around March 23. Would you be able to check on latest main and see if you still observe the issue?

[bongjunj]

@cfallin I'm still observing this issue.
I tested upstream version(3df8ce1) and no-opt version(bongjunj@43c5ff3) with the sightglass (789ac095):

> cargo run --release benchmark benchmarks/spidermonkey/spidermonkey-json.wasm -e v-base-260517/wasmtime/target/release/libwasmtime_bench_api.so main-260517/wasmtime/target/release/libwasmtime_bench_api.so --iterations-per-process=10
    Finished `release` profile [optimized] target(s) in 0.18s
     Running `target/release/sightglass-cli benchmark benchmarks/spidermonkey/spidermonkey-json.wasm -e base-260517/wasmtime/target/release/libwasmtime_bench_api.so ntal/v-main-260517/wasmtime/target/release/libwasmtime_bench_api.so --iterations-per-process=10`

Running 200 total iterations (2 engines * 1 benchmarks * 10 processes * 10 iterations per process)

[Done] [Elapsed    ] [Est. Rem.  ]
[  5%] [00h:00m:08s] [00h:02m:32s] ..........
[ 55%] [00h:01m:32s] [00h:01m:15s] ..........

Finished benchmarking in 00h:02m:48s

execution :: cycles :: benchmarks/spidermonkey/spidermonkey-json.wasm

  Δ = 47061945.96 ± 7156475.56 (confidence = 99%)

  v-base-260517/wasmtime/target/release/libwasmtime_bench_api.so is 1.07x to 1.09x faster than v-main-260517/wasmtime/target/release/libwasmtime_bench_api.so!

  [581745126 593119831.98 690791320] v-base-260517/wasmtime/target/release/libwasmtime_bench_api.so
  [631312126 640181777.94 739371462] v-main-260517/wasmtime/target/release/libwasmtime_bench_api.so

compilation :: cycles :: benchmarks/spidermonkey/spidermonkey-json.wasm

  Δ = 33776664.00 ± 26492776.81 (confidence = 99%)

  v-main-260517/wasmtime/target/release/libwasmtime_bench_api.so is 1.00x to 1.04x faster than v-base-260517/wasmtime/target/release/libwasmtime_bench_api.so!

  [1311399220 1534236565.90 1730050064] v-base-260517/wasmtime/target/release/libwasmtime_bench_api.so
  [1344967908 1500459901.90 1699803632] v-main-260517/wasmtime/target/release/libwasmtime_bench_api.so

instantiation :: cycles :: benchmarks/spidermonkey/spidermonkey-json.wasm

  No difference in performance.

  [735834 808589.04 1171376] v-base-260517/wasmtime/target/release/libwasmtime_bench_api.so
  [722858 795706.08 2101592] v-main-260517/wasmtime/target/release/libwasmtime_bench_api.so

> cargo run --release benchmark benchmarks/spidermonkey/spidermonkey-json.wasm -e v-base-260517/wasmtime/target/release/libwasmtime_bench_api.so main-260517/wasmtime/target/release/libwasmtime_bench_api.so --iterations-per-process=100 --processes=1
    Finished `release` profile [optimized] target(s) in 0.18s
     Running `target/release/sightglass-cli benchmark benchmarks/spidermonkey/spidermonkey-json.wasm -e base-260517/wasmtime/target/release/libwasmtime_bench_api.so ntal/v-main-260517/wasmtime/target/release/libwasmtime_bench_api.so --iterations-per-process=100 --processes=1`

execution :: cycles :: benchmarks/spidermonkey/spidermonkey-json.wasm

  Δ = 46004063.16 ± 2685004.67 (confidence = 99%)

  v-base-260517/wasmtime/target/release/libwasmtime_bench_api.so is 1.07x to 1.08x faster than v-main-260517/wasmtime/target/release/libwasmtime_bench_api.so!

  [581724878 585888810.80 645912744] v-base-260517/wasmtime/target/release/libwasmtime_bench_api.so
  [627163556 631892873.96 652945300] v-main-260517/wasmtime/target/release/libwasmtime_bench_api.so

compilation :: cycles :: benchmarks/spidermonkey/spidermonkey-json.wasm

  Δ = 45072400.40 ± 25749426.66 (confidence = 99%)

  v-main-260517/wasmtime/target/release/libwasmtime_bench_api.so is 1.01x to 1.05x faster than v-base-260517/wasmtime/target/release/libwasmtime_bench_api.so!

  [1333486888 1503053131.78 1810715182] v-base-260517/wasmtime/target/release/libwasmtime_bench_api.so
  [1299260868 1457980731.38 1840198928] v-main-260517/wasmtime/target/release/libwasmtime_bench_api.so

instantiation :: cycles :: benchmarks/spidermonkey/spidermonkey-json.wasm

  No difference in performance.

  [732578 768323.40 1182792] v-base-260517/wasmtime/target/release/libwasmtime_bench_api.so
  [751076 777308.76 969436] v-main-260517/wasmtime/target/release/libwasmtime_bench_api.so

Compared to the past versions, the regression is reduced, but this experiment shows that it still persists.