Skip to content

Add NativeAOT (and CoreRT) support for micro benchmarks #2328

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 24 commits into from
Mar 30, 2022

Conversation

adamsitnik
Copy link
Member

This PR contains 4 changes:

I have tested these changes locally on Windows for both NativeAOT and CoreRT and all benchmarks are working fine. I'll report back once I have Linux results.

I've added nativeaot7.0 and corert6.0 monikers to the supported framework list. I did not want to add another optional argument like --aot because it would make it impossible to compare multiple frameworks using -f:

py .\scripts\benchmarks_ci.py -f net6.0 corert6.0 net7.0 nativeaot7.0 --filter *

With this PR we will be able to compare .NET 6 with CoreRT vs .NET 7 with NativeAOT and ensure that there are no regressions.

It's also possible to specify ILCompilerVersion:

py .\scripts\benchmarks_ci.py -f nativeaot7.0 --filter * --bdn-arguments "--ilCompilerVersion $here" 

So we will be able to do things like Preview X vs Preview Y.

cc @jkotas @MichalStrehovsky @hez2010 @Beau-Gosse-dev @jeffhandley @agocke

@adamsitnik
Copy link
Member Author

@DrewScoggins could you please upload 0.13.1.1737 to our internal NuGet feed? Would you mind if I add one CI leg for NativeAOT microbenchmarks?

@jkotas
Copy link
Member

jkotas commented Mar 23, 2022

corert6.0

We have retired the corert repo in 2020. It is confusing that this called corert. I assume that the bits are not coming from corert repo. Can we call it nativeaot6.0?

@adamsitnik
Copy link
Member Author

We have retired the corert repo in 2020. It is confusing that this called corert. I assume that the bits are not coming from corert repo. Can we call it nativeaot6.0?

The corert6.0 is using the retired bits on purpose, I want to use it as a baseline for perf comparison.

Microsoft.DotNet.ILCompiler NuGet feed
corert6.0 6.0.0-* https://pkgs.dev.azure.com/dnceng/public/\_packaging/dotnet-experimental/nuget/v3/index.json
nativeaot7.0 7.0.0-* https://pkgs.dev.azure.com/dnceng/public/\_packaging/dotnet7/nuget/v3/index.json

I've already spotted one perf regression, which is most likely related to printing to console on Windows.

@jkotas
Copy link
Member

jkotas commented Mar 23, 2022

I am ok with having an option to use the .NET 6 experimental bits. My point is these bits are coming from NativeAOT experimental project in runtimelab. They are not coming from https://github.com/dotnet/corert repo. It is confusing to call them corert6.0.

@adamsitnik
Copy link
Member Author

@jkotas thanks for the explanation. I am going to rename it to nativeaot6.0

@DrewScoggins
Copy link
Member

No go ahead and add that leg. I assume this will only be on the CI side, and not the collection side, right?

@@ -18,6 +19,7 @@ namespace MicroBenchmarks.Serializers
[GenericTypeArguments(typeof(XmlElement))]
[GenericTypeArguments(typeof(SimpleStructWithProperties))]
[GenericTypeArguments(typeof(ClassImplementingIXmlSerialiable))]
[AotFilter("Currently not supported due to missing metadata.")]
Copy link

@hez2010 hez2010 Mar 24, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issues caused by missing metadata can be fixed with rd.xml. Does BDN support attaching rd.xml for NativeAOT benchmark tests, i.e. allowing to add a <RdXmlFile Include="rd.xml" /> to ItemGroup in the generated project?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hez2010 great point! I've opened #2338 to track it

@adamsitnik adamsitnik merged commit 6e1585f into dotnet:main Mar 30, 2022
@adamsitnik adamsitnik deleted the net7AOT branch March 30, 2022 07:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants