Performance regression in ByteMark.BenchIDEAEncryption

[adamsitnik]

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f netcoreapp3.1 netcoreapp5.0 --filter ByteMark.BenchIDEAEncryption

By looking at the full historical data it looks that this particular regression has been introduced in the last few weeks

And it's not related to memory or GC:

Method	Runtime	Mean	Error	StdDev	Median	Min	Max	Ratio	Gen 0	Gen 1	Gen 2	Allocated
BenchIDEAEncryption	.NET Core 3.1	949.9 ms	3.77 ms	3.53 ms	949.2 ms	942.4 ms	955.5 ms	1.00	-	-	-	610.2 KB
BenchIDEAEncryption	.NET Core 5.0	1,452.8 ms	3.59 ms	3.36 ms	1,452.8 ms	1,446.5 ms	1,458.9 ms	1.53	-	-	-	610.2 KB

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@DrewScoggins this regression did not get detected by the bot. I am not sure why, perhaps it's too fresh?

cc @AndyAyersMS it might be some interesting low-level regression I guess?

Tagging subscribers to this area: @bartonjs, @vcsjones, @krwq
See info in area-owners.md if you want to be subscribed.

[danmoseley]

I assume this is Windows specific @adamsitnik ?

[vcsjones]

Note this isn't benchmarking anything in System.Security - its benchmarking a hand-rolled implementation of the IDEA algorithm.

[adamsitnik]

I assume this is Windows specific @adamsitnik ?

@danmosemsft It looks like it's not Windows-specific

The Ubuntu historical data

[danmoseley]

It looks like it's not Windows-specific

Ah, good info, I guess that means we can exclude machine configuration/the crypto work itself.

[bartonjs]

I guess that means we can exclude ... the crypto work itself.

It's not using the crypto stack, it's a custom algorithm implementation inside the performance repository just for use as a benchmark. So it's just low level things like array allocation, array read/write, field access, and GC.

https://github.com/dotnet/performance/blob/454476401e17ed7f4d8b899ecf7661eb6cd63bad/src/benchmarks/micro/runtime/Bytemark/ByteMark.cs

https://github.com/dotnet/performance/blob/8aed638c9ee65c034fe0cca4ea2bdc3a68d2a6b5/src/benchmarks/micro/runtime/Bytemark/idea.cs

[danmoseley]

Disregard, I missed your comment @vcsjones

[AndyAyersMS]

This is a jit specific benchmark, will relabel as codegen.

[AndyAyersMS]

History strongly implicates #40535. Wonder if #40871 has any impact here?

cc @briansull @echesakovMSFT @dotnet/jit-contrib