Speedup tests

[FrancoGiachetta]

Speedup tests

This PR adds cargo-nextest as the tool used to run tests. It also parallelizes how tests are run in cairo-native-test and scarb-native-test which were run rather sequentially.

Improvements:

• Parallelizing cairo-native-test gives a ~100% improvement over make test-cairo command.

Introduces Breaking Changes?

No.

• Created PR in sequencer
• Created PR in starknet-replay
• Updated the starknet-blocks.yml workflow to use these PRs.

These PRs should be merged after this one right away, in that order.

Checklist

• Linked to Github Issue.
• Unit tests added.
• Integration tests added.
• This change requires new documentation.
  • Documentation has been added/updated.

[github-actions]

Benchmark results Main vs HEAD.

Base

Command	Mean [s]	Min [s]	Max [s]	Relative
base dict_insert.cairo (JIT)	2.562 ± 0.022	2.531	2.602	1.03 ± 0.01
base dict_insert.cairo (AOT)	2.491 ± 0.024	2.468	2.537	1.00

Head

Command	Mean [s]	Min [s]	Max [s]	Relative
head dict_insert.cairo (JIT)	2.573 ± 0.027	2.537	2.624	1.04 ± 0.01
head dict_insert.cairo (AOT)	2.480 ± 0.020	2.454	2.521	1.00

Base

Command	Mean [s]	Min [s]	Max [s]	Relative
base dict_snapshot.cairo (JIT)	2.272 ± 0.032	2.239	2.333	1.05 ± 0.02
base dict_snapshot.cairo (AOT)	2.157 ± 0.027	2.128	2.217	1.00

Head

Command	Mean [s]	Min [s]	Max [s]	Relative
head dict_snapshot.cairo (JIT)	2.280 ± 0.014	2.266	2.305	1.04 ± 0.01
head dict_snapshot.cairo (AOT)	2.185 ± 0.017	2.167	2.228	1.00

Base

Command	Mean [s]	Min [s]	Max [s]	Relative
base factorial_2M.cairo (JIT)	2.630 ± 0.019	2.598	2.667	1.01 ± 0.01
base factorial_2M.cairo (AOT)	2.593 ± 0.021	2.564	2.636	1.00

Head

Command	Mean [s]	Min [s]	Max [s]	Relative
head factorial_2M.cairo (JIT)	2.650 ± 0.015	2.633	2.684	1.02 ± 0.01
head factorial_2M.cairo (AOT)	2.605 ± 0.022	2.571	2.634	1.00

Base

Command	Mean [s]	Min [s]	Max [s]	Relative
base fib_2M.cairo (JIT)	2.174 ± 0.026	2.138	2.217	1.01 ± 0.02
base fib_2M.cairo (AOT)	2.143 ± 0.032	2.101	2.193	1.00

Head

Command	Mean [s]	Min [s]	Max [s]	Relative
head fib_2M.cairo (JIT)	2.188 ± 0.024	2.145	2.224	1.00 ± 0.03
head fib_2M.cairo (AOT)	2.182 ± 0.059	2.115	2.263	1.00

Base

Command	Mean [s]	Min [s]	Max [s]	Relative
base linear_search.cairo (JIT)	2.344 ± 0.021	2.320	2.387	1.06 ± 0.01
base linear_search.cairo (AOT)	2.210 ± 0.018	2.187	2.243	1.00

Head

Command	Mean [s]	Min [s]	Max [s]	Relative
head linear_search.cairo (JIT)	2.368 ± 0.027	2.342	2.417	1.07 ± 0.02
head linear_search.cairo (AOT)	2.214 ± 0.030	2.179	2.285	1.00

Base

Command	Mean [s]	Min [s]	Max [s]	Relative
base logistic_map.cairo (JIT)	2.456 ± 0.018	2.424	2.478	1.07 ± 0.01
base logistic_map.cairo (AOT)	2.286 ± 0.016	2.261	2.317	1.00

Head

Command	Mean [s]	Min [s]	Max [s]	Relative
head logistic_map.cairo (JIT)	2.484 ± 0.043	2.434	2.595	1.07 ± 0.02
head logistic_map.cairo (AOT)	2.326 ± 0.025	2.286	2.370	1.00

[github-actions]

Benchmarking results

Benchmark for program dict_insert

Open benchmarks

Command	Mean [s]	Min [s]	Max [s]	Relative
Cairo-vm (Rust, Cairo 1)	10.910 ± 0.045	10.807	10.964	4.36 ± 0.04
cairo-native (embedded AOT)	2.501 ± 0.019	2.484	2.542	1.00
cairo-native (embedded JIT using LLVM's ORC Engine)	2.552 ± 0.016	2.528	2.587	1.02 ± 0.01

Benchmark for program dict_snapshot

Open benchmarks

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
Cairo-vm (Rust, Cairo 1)	532.3 ± 8.5	524.5	547.5	1.00
cairo-native (embedded AOT)	2187.7 ± 27.1	2150.0	2234.7	4.11 ± 0.08
cairo-native (embedded JIT using LLVM's ORC Engine)	2279.7 ± 12.8	2262.8	2299.0	4.28 ± 0.07

Benchmark for program factorial_2M

Open benchmarks

Command	Mean [s]	Min [s]	Max [s]	Relative
Cairo-vm (Rust, Cairo 1)	4.813 ± 0.039	4.772	4.890	1.85 ± 0.02
cairo-native (embedded AOT)	2.602 ± 0.010	2.584	2.618	1.00
cairo-native (embedded JIT using LLVM's ORC Engine)	2.648 ± 0.017	2.625	2.685	1.02 ± 0.01

Benchmark for program fib_2M

Open benchmarks

Command	Mean [s]	Min [s]	Max [s]	Relative
Cairo-vm (Rust, Cairo 1)	4.740 ± 0.022	4.713	4.787	2.23 ± 0.01
cairo-native (embedded AOT)	2.128 ± 0.010	2.111	2.142	1.00
cairo-native (embedded JIT using LLVM's ORC Engine)	2.199 ± 0.034	2.163	2.252	1.03 ± 0.02

Benchmark for program linear_search

Open benchmarks

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
Cairo-vm (Rust, Cairo 1)	573.2 ± 14.8	562.8	611.9	1.00
cairo-native (embedded AOT)	2219.1 ± 19.1	2193.7	2243.5	3.87 ± 0.11
cairo-native (embedded JIT using LLVM's ORC Engine)	2375.6 ± 31.8	2330.2	2434.6	4.14 ± 0.12

Benchmark for program logistic_map

Open benchmarks

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
Cairo-vm (Rust, Cairo 1)	381.6 ± 5.7	374.8	390.2	1.00
cairo-native (embedded AOT)	2291.0 ± 19.7	2265.7	2326.6	6.00 ± 0.10
cairo-native (embedded JIT using LLVM's ORC Engine)	2469.5 ± 17.6	2448.8	2499.1	6.47 ± 0.11

[JulianGCalderon]

Running make test prints the following warning. Is the config file required?

warning: ignoring unknown configuration keys in config file /Users/julian/projects/cairo_native/.config/nextest.toml:
  - profile.ci.debug
  - profile.ci.inherits
  - profile.ci.opt-level

[FrancoGiachetta]

Oh, I may have missed that message when I ran it. I'll check it.

[FrancoGiachetta]

I ended up removing the config file.

[JulianGCalderon]

LGTM!