Parsing integers has regressed on arm32

[adamsitnik]

Various integer parsing benchmarks have regressed on arm (not on arm64).

Since arm is not covered by the perf lab, we don't have historical data in the Reporting System and all we know is that it happened between preview 2 and 3.

Repro:

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_monthly.py net7.0-preview2 net7.0-preview3 --filter System.Tests.Perf_Int32.Parse --architecture arm

System.Tests.Perf_Int32.Parse(value: "4")

Result	Base	Diff	Ratio	Operating System	Bit	Processor Name
Same	51.09	49.52	1.03	alpine 3.12	Arm64	Unknown processor
Same	44.74	42.88	1.04	ubuntu 18.04	Arm64	Unknown processor
Same	10.27	10.62	0.97	Windows 10	Arm64	Microsoft SQ1 3.0 GHz
Same	10.40	10.74	0.97	Windows 11	Arm64	Microsoft SQ1 3.0 GHz
Slower	8.29	11.18	0.74	Windows 11	Arm64	Unknown processor
Same	6.03	5.85	1.03	macOS Monterey 12.3	Arm64	Apple M1 Max
Same	11.31	10.11	1.12	Windows 10	X64	Intel Xeon Platinum 8272CL CPU 2.60GHz
Slower	9.58	13.24	0.72	Windows 10	X64	Intel Xeon CPU E5-1650 v4 3.60GHz
Same	8.40	8.12	1.03	Windows 11	X64	AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same	6.30	6.66	0.95	Windows 11	X64	AMD Ryzen 9 5900X
Faster	17.81	13.32	1.34	Windows 11	X64	Intel Core i5-4300U CPU 1.90GHz (Haswell)
Faster	13.77	11.00	1.25	Windows 11	X64	Unknown processor
Same	8.62	8.38	1.03	Windows 11	X64	Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Same	9.95	9.48	1.05	Windows 11	X64	Intel Core i9-9900T CPU 2.10GHz
Slower	9.92	15.37	0.65	alpine 3.13	X64	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Faster	26.16	23.34	1.12	centos 7	X64	Intel Xeon CPU E5530 2.40GHz
Same	9.61	9.78	0.98	debian 11	X64	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same	10.43	9.73	1.07	opensuse-leap 15.3	X64	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same	11.03	11.26	0.98	ubuntu 18.04	X64	Intel Xeon CPU E5-1650 v4 3.60GHz
Faster	18.41	15.63	1.18	ubuntu 18.04	X64	Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
Slower	17.64	23.96	0.74	Windows 10	Arm	Microsoft SQ1 3.0 GHz
Slower	47.96	58.39	0.82	ubuntu 18.04	Arm	ARMv7 Processor rev 3 (v7l)
Same	9.09	8.50	1.07	Windows 10	X86	Intel Xeon CPU E5-1650 v4 3.60GHz
Same	7.44	7.89	0.94	Windows 11	X86	AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same	17.27	17.77	0.97	Windows 11	X86	Intel Core i7-10510U CPU 1.80GHz
Same	7.38	7.61	0.97	Windows 7 SP1	X86	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same	11.96	12.18	0.98	macOS Monterey 12.2.1	X64	Intel Core i7-5557U CPU 3.10GHz (Broadwell)
Same	11.25	10.68	1.05	macOS Monterey 12.3.1	X64	Intel Core i7-4870HQ CPU 2.50GHz (Haswell)

System.Tests.Perf_Int16.TryParse(value: "32767")

Result	Base	Diff	Ratio	Operating System	Bit	Processor Name
Same	70.54	68.96	1.02	alpine 3.12	Arm64	Unknown processor
Same	63.34	62.27	1.02	ubuntu 18.04	Arm64	Unknown processor
Slower	14.76	17.04	0.87	Windows 10	Arm64	Microsoft SQ1 3.0 GHz
Same	14.82	15.24	0.97	Windows 11	Arm64	Microsoft SQ1 3.0 GHz
Slower	8.23	15.67	0.53	Windows 11	Arm64	Unknown processor
Same	8.82	8.35	1.06	macOS Monterey 12.3	Arm64	Apple M1 Max
Faster	15.53	13.46	1.15	Windows 10	X64	Intel Xeon Platinum 8272CL CPU 2.60GHz
Same	14.63	15.18	0.96	Windows 10	X64	Intel Xeon CPU E5-1650 v4 3.60GHz
Same	11.72	11.29	1.04	Windows 11	X64	AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same	8.51	8.61	0.99	Windows 11	X64	AMD Ryzen 9 5900X
Same	19.32	18.25	1.06	Windows 11	X64	Intel Core i5-4300U CPU 1.90GHz (Haswell)
Faster	18.76	14.61	1.28	Windows 11	X64	Unknown processor
Noise	12.59	10.85	-	Windows 11	X64	Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Noise	14.69	12.57	-	Windows 11	X64	Intel Core i9-9900T CPU 2.10GHz
Same	12.68	12.79	0.99	alpine 3.13	X64	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same	35.12	32.00	1.10	centos 7	X64	Intel Xeon CPU E5530 2.40GHz
Same	12.53	12.93	0.97	debian 11	X64	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Noise	12.24	13.93	-	opensuse-leap 15.3	X64	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same	14.90	15.40	0.97	ubuntu 18.04	X64	Intel Xeon CPU E5-1650 v4 3.60GHz
Faster	27.86	22.71	1.23	ubuntu 18.04	X64	Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
Slower	25.73	32.50	0.79	Windows 10	Arm	Microsoft SQ1 3.0 GHz
Slower	74.33	84.39	0.88	ubuntu 18.04	Arm	ARMv7 Processor rev 3 (v7l)
Same	12.74	11.72	1.09	Windows 10	X86	Intel Xeon CPU E5-1650 v4 3.60GHz
Same	11.19	11.43	0.98	Windows 11	X86	AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same	24.99	23.37	1.07	Windows 11	X86	Intel Core i7-10510U CPU 1.80GHz
Same	10.48	9.79	1.07	Windows 7 SP1	X86	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same	16.05	15.28	1.05	macOS Monterey 12.2.1	X64	Intel Core i7-5557U CPU 3.10GHz (Broadwell)
Same	15.29	14.15	1.08	macOS Monterey 12.3.1	X64	Intel Core i7-4870HQ CPU 2.50GHz (Haswell)

System.Tests.Perf_SByte.TryParse(value: "-128")

Result	Base	Diff	Ratio	Operating System	Bit	Processor Name
Same	61.96	64.00	0.97	alpine 3.12	Arm64	Unknown processor
Same	56.14	55.93	1.00	ubuntu 18.04	Arm64	Unknown processor
Same	12.43	12.67	0.98	Windows 10	Arm64	Microsoft SQ1 3.0 GHz
Same	12.55	12.71	0.99	Windows 11	Arm64	Microsoft SQ1 3.0 GHz
Slower	7.65	13.19	0.58	Windows 11	Arm64	Unknown processor
Same	8.19	7.57	1.08	macOS Monterey 12.3	Arm64	Apple M1 Max
Noise	14.27	12.14	-	Windows 10	X64	Intel Xeon Platinum 8272CL CPU 2.60GHz
Same	12.65	12.13	1.04	Windows 10	X64	Intel Xeon CPU E5-1650 v4 3.60GHz
Same	10.44	9.99	1.05	Windows 11	X64	AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same	7.64	7.84	0.98	Windows 11	X64	AMD Ryzen 9 5900X
Same	16.78	15.83	1.06	Windows 11	X64	Intel Core i5-4300U CPU 1.90GHz (Haswell)
Faster	16.03	12.58	1.27	Windows 11	X64	Unknown processor
Noise	11.29	9.85	-	Windows 11	X64	Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Noise	12.86	11.07	-	Windows 11	X64	Intel Core i9-9900T CPU 2.10GHz
Same	11.62	12.50	0.93	alpine 3.13	X64	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Faster	34.09	28.03	1.22	centos 7	X64	Intel Xeon CPU E5530 2.40GHz
Same	11.89	11.74	1.01	debian 11	X64	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same	11.53	11.92	0.97	opensuse-leap 15.3	X64	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same	13.67	13.63	1.00	ubuntu 18.04	X64	Intel Xeon CPU E5-1650 v4 3.60GHz
Faster	24.34	20.20	1.21	ubuntu 18.04	X64	Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
Slower	22.42	31.26	0.72	Windows 10	Arm	Microsoft SQ1 3.0 GHz
Slower	65.23	75.78	0.86	ubuntu 18.04	Arm	ARMv7 Processor rev 3 (v7l)
Slower	11.50	14.33	0.80	Windows 10	X86	Intel Xeon CPU E5-1650 v4 3.60GHz
Same	10.60	10.63	1.00	Windows 11	X86	AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same	22.08	22.21	0.99	Windows 11	X86	Intel Core i7-10510U CPU 1.80GHz
Same	9.36	9.92	0.94	Windows 7 SP1	X86	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same	14.90	14.18	1.05	macOS Monterey 12.2.1	X64	Intel Core i7-5557U CPU 3.10GHz (Broadwell)
Faster	17.07	13.16	1.30	macOS Monterey 12.3.1	X64	Intel Core i7-4870HQ CPU 2.50GHz (Haswell)

Tagging subscribers to this area: @dotnet/area-system-runtime
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Various integer parsing benchmarks have regressed on arm (not on arm64).

Since arm is not covered by the perf lab, we don't have historical data in the Reporting System and all we know is that it happened between preview 2 and 3.

Repro:

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_monthly.py net7.0-preview2 net7.0-preview3 --filter System.Tests.Perf_Int32.Parse --architecture arm

System.Tests.Perf_Int32.Parse(value: "4")

Result	Base	Diff	Ratio	Operating System	Bit	Processor Name
Same	51.09	49.52	1.03	alpine 3.12	Arm64	Unknown processor
Same	44.74	42.88	1.04	ubuntu 18.04	Arm64	Unknown processor
Same	10.27	10.62	0.97	Windows 10	Arm64	Microsoft SQ1 3.0 GHz
Same	10.40	10.74	0.97	Windows 11	Arm64	Microsoft SQ1 3.0 GHz
Slower	8.29	11.18	0.74	Windows 11	Arm64	Unknown processor
Same	6.03	5.85	1.03	macOS Monterey 12.3	Arm64	Apple M1 Max
Same	11.31	10.11	1.12	Windows 10	X64	Intel Xeon Platinum 8272CL CPU 2.60GHz
Slower	9.58	13.24	0.72	Windows 10	X64	Intel Xeon CPU E5-1650 v4 3.60GHz
Same	8.40	8.12	1.03	Windows 11	X64	AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same	6.30	6.66	0.95	Windows 11	X64	AMD Ryzen 9 5900X
Faster	17.81	13.32	1.34	Windows 11	X64	Intel Core i5-4300U CPU 1.90GHz (Haswell)
Faster	13.77	11.00	1.25	Windows 11	X64	Unknown processor
Same	8.62	8.38	1.03	Windows 11	X64	Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Same	9.95	9.48	1.05	Windows 11	X64	Intel Core i9-9900T CPU 2.10GHz
Slower	9.92	15.37	0.65	alpine 3.13	X64	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Faster	26.16	23.34	1.12	centos 7	X64	Intel Xeon CPU E5530 2.40GHz
Same	9.61	9.78	0.98	debian 11	X64	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same	10.43	9.73	1.07	opensuse-leap 15.3	X64	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same	11.03	11.26	0.98	ubuntu 18.04	X64	Intel Xeon CPU E5-1650 v4 3.60GHz
Faster	18.41	15.63	1.18	ubuntu 18.04	X64	Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
Slower	17.64	23.96	0.74	Windows 10	Arm	Microsoft SQ1 3.0 GHz
Slower	47.96	58.39	0.82	ubuntu 18.04	Arm	ARMv7 Processor rev 3 (v7l)
Same	9.09	8.50	1.07	Windows 10	X86	Intel Xeon CPU E5-1650 v4 3.60GHz
Same	7.44	7.89	0.94	Windows 11	X86	AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same	17.27	17.77	0.97	Windows 11	X86	Intel Core i7-10510U CPU 1.80GHz
Same	7.38	7.61	0.97	Windows 7 SP1	X86	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same	11.96	12.18	0.98	macOS Monterey 12.2.1	X64	Intel Core i7-5557U CPU 3.10GHz (Broadwell)
Same	11.25	10.68	1.05	macOS Monterey 12.3.1	X64	Intel Core i7-4870HQ CPU 2.50GHz (Haswell)

System.Tests.Perf_Int16.TryParse(value: "32767")

Result	Base	Diff	Ratio	Operating System	Bit	Processor Name
Same	70.54	68.96	1.02	alpine 3.12	Arm64	Unknown processor
Same	63.34	62.27	1.02	ubuntu 18.04	Arm64	Unknown processor
Slower	14.76	17.04	0.87	Windows 10	Arm64	Microsoft SQ1 3.0 GHz
Same	14.82	15.24	0.97	Windows 11	Arm64	Microsoft SQ1 3.0 GHz
Slower	8.23	15.67	0.53	Windows 11	Arm64	Unknown processor
Same	8.82	8.35	1.06	macOS Monterey 12.3	Arm64	Apple M1 Max
Faster	15.53	13.46	1.15	Windows 10	X64	Intel Xeon Platinum 8272CL CPU 2.60GHz
Same	14.63	15.18	0.96	Windows 10	X64	Intel Xeon CPU E5-1650 v4 3.60GHz
Same	11.72	11.29	1.04	Windows 11	X64	AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same	8.51	8.61	0.99	Windows 11	X64	AMD Ryzen 9 5900X
Same	19.32	18.25	1.06	Windows 11	X64	Intel Core i5-4300U CPU 1.90GHz (Haswell)
Faster	18.76	14.61	1.28	Windows 11	X64	Unknown processor
Noise	12.59	10.85	-	Windows 11	X64	Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Noise	14.69	12.57	-	Windows 11	X64	Intel Core i9-9900T CPU 2.10GHz
Same	12.68	12.79	0.99	alpine 3.13	X64	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same	35.12	32.00	1.10	centos 7	X64	Intel Xeon CPU E5530 2.40GHz
Same	12.53	12.93	0.97	debian 11	X64	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Noise	12.24	13.93	-	opensuse-leap 15.3	X64	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same	14.90	15.40	0.97	ubuntu 18.04	X64	Intel Xeon CPU E5-1650 v4 3.60GHz
Faster	27.86	22.71	1.23	ubuntu 18.04	X64	Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
Slower	25.73	32.50	0.79	Windows 10	Arm	Microsoft SQ1 3.0 GHz
Slower	74.33	84.39	0.88	ubuntu 18.04	Arm	ARMv7 Processor rev 3 (v7l)
Same	12.74	11.72	1.09	Windows 10	X86	Intel Xeon CPU E5-1650 v4 3.60GHz
Same	11.19	11.43	0.98	Windows 11	X86	AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same	24.99	23.37	1.07	Windows 11	X86	Intel Core i7-10510U CPU 1.80GHz
Same	10.48	9.79	1.07	Windows 7 SP1	X86	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same	16.05	15.28	1.05	macOS Monterey 12.2.1	X64	Intel Core i7-5557U CPU 3.10GHz (Broadwell)
Same	15.29	14.15	1.08	macOS Monterey 12.3.1	X64	Intel Core i7-4870HQ CPU 2.50GHz (Haswell)

System.Tests.Perf_SByte.TryParse(value: "-128")

Result	Base	Diff	Ratio	Operating System	Bit	Processor Name
Same	61.96	64.00	0.97	alpine 3.12	Arm64	Unknown processor
Same	56.14	55.93	1.00	ubuntu 18.04	Arm64	Unknown processor
Same	12.43	12.67	0.98	Windows 10	Arm64	Microsoft SQ1 3.0 GHz
Same	12.55	12.71	0.99	Windows 11	Arm64	Microsoft SQ1 3.0 GHz
Slower	7.65	13.19	0.58	Windows 11	Arm64	Unknown processor
Same	8.19	7.57	1.08	macOS Monterey 12.3	Arm64	Apple M1 Max
Noise	14.27	12.14	-	Windows 10	X64	Intel Xeon Platinum 8272CL CPU 2.60GHz
Same	12.65	12.13	1.04	Windows 10	X64	Intel Xeon CPU E5-1650 v4 3.60GHz
Same	10.44	9.99	1.05	Windows 11	X64	AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same	7.64	7.84	0.98	Windows 11	X64	AMD Ryzen 9 5900X
Same	16.78	15.83	1.06	Windows 11	X64	Intel Core i5-4300U CPU 1.90GHz (Haswell)
Faster	16.03	12.58	1.27	Windows 11	X64	Unknown processor
Noise	11.29	9.85	-	Windows 11	X64	Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Noise	12.86	11.07	-	Windows 11	X64	Intel Core i9-9900T CPU 2.10GHz
Same	11.62	12.50	0.93	alpine 3.13	X64	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Faster	34.09	28.03	1.22	centos 7	X64	Intel Xeon CPU E5530 2.40GHz
Same	11.89	11.74	1.01	debian 11	X64	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same	11.53	11.92	0.97	opensuse-leap 15.3	X64	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same	13.67	13.63	1.00	ubuntu 18.04	X64	Intel Xeon CPU E5-1650 v4 3.60GHz
Faster	24.34	20.20	1.21	ubuntu 18.04	X64	Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
Slower	22.42	31.26	0.72	Windows 10	Arm	Microsoft SQ1 3.0 GHz
Slower	65.23	75.78	0.86	ubuntu 18.04	Arm	ARMv7 Processor rev 3 (v7l)
Slower	11.50	14.33	0.80	Windows 10	X86	Intel Xeon CPU E5-1650 v4 3.60GHz
Same	10.60	10.63	1.00	Windows 11	X86	AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same	22.08	22.21	0.99	Windows 11	X86	Intel Core i7-10510U CPU 1.80GHz
Same	9.36	9.92	0.94	Windows 7 SP1	X86	Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Same	14.90	14.18	1.05	macOS Monterey 12.2.1	X64	Intel Core i7-5557U CPU 3.10GHz (Broadwell)
Faster	17.07	13.16	1.30	macOS Monterey 12.3.1	X64	Intel Core i7-4870HQ CPU 2.50GHz (Haswell)

Author:	adamsitnik
Assignees:	-
Labels:	arch-arm32, area-System.Runtime, tenet-performance
Milestone:	-

[dakersnar]

I'm tempted to call this noise, as these tests are historically noisy on other configs. Unfortunately, we haven't been testing arm32 on perf reports since this one, so it is hard to say if this issue persisted or resolved itself. Given the history of noise, do you think it is safe to call this regression noise? @adamsitnik @tannergooding

[dakersnar]

Closing this, I'm assuming this is noise.

[adamsitnik]

@dakersnar could you please run these benchmark for .NET 6 and 7 and see what is the difference?

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 net7.0 --filter System.Tests.Perf_Int32.Parse --architecture arm

[tannergooding]

This is another example of where even if it was an actual regression, we're looking at ~5ns (somewhere between 5-15 cycles) where other platforms largely stayed the same or had measurable improvements in the inverse direction.

Arm32 is a platform that we only officially support on Linux and which doesn't even have SIMD support/acceleration today. This likely came about as part of the other parsing/formatting improvements and its not something worth spending significant time investigating.