Added PowerShell.Benchmarks to SuperPMI collections

[TIHan]

This adds the PowerShell.Benchmarks, from the recently merged dotnet/performance#3000, to the SuperPMI collections.

[ghost]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

---

This adds the PowerShell.Benchmarks, from the recently merged dotnet/performance#3000, to the SuperPMI collections.

Author:	TIHan
Assignees:	-
Labels:	area-CodeGen-coreclr
Milestone:	-

[TIHan]

@kunalspathak this is ready.

[TIHan]

Seems to fail at the moment actually:

MSBUILD : error MSB1009: Project file does not exist.
Switch: /root/helix/work/workitem/u/performance/src/benchmarks/real-world/PowerShell.Benchmarks.csproj

[TIHan]

Current failure is the targetframework is set to net7, made a PR dotnet/performance#3420 to update it to net8.

[TIHan]

@BruceForstall This is ready. See successful pipeline: https://dev.azure.com/dnceng/internal/_build/results?buildId=2293539&view=logs&j=6e5fe0ba-f6d5-5f75-5db7-b07a5a8eee39&t=b89c5eaa-be13-560b-8e20-13db295b7d16

[20:47:32] Appending file 'D:\a_work\1\s\artifacts\helixresults\4f645e24-c233-4ce8-91b8-13247aa100ca\Partition6\realworld.run..windows.x64.checked.mch'
[20:47:32] Loaded 755, Saved 8

NativeAOT failure is not related.

[BruceForstall]

Something seems wrong. There should be more than 8 new, unique, functions. It doesn't look like the benchmark actually ran.

In the per-run log it looks like all the collected functions are either System.* or BenchmarkDotNet.

E.g., I see this failure:

[20:29:32] // BeforeAnythingElse
[20:29:32] 
[20:29:32] System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
[20:29:32]  ---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Management.Automation, Version=7.3.6.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. The system cannot find the file specified.
[20:29:32] File name: 'System.Management.Automation, Version=7.3.6.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'
[20:29:32]    at BenchmarkDotNet.Autogenerated.Runnable_2..ctor()
[20:29:32]    at BenchmarkDotNet.Autogenerated.Runnable_2.Run(IHost host, String benchmarkName) in C:\h\w\A4330904\w\A10F08E7\u\performance\artifacts\25dde84f-65d6-4200-8225-6408a8a18724\25dde84f-65d6-4200-8225-6408a8a18724.notcs:line 488
[20:29:32]    at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
[20:29:32]    at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
[20:29:32]    --- End of inner exception stack trace ---
[20:29:32]    at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
[20:29:32]    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
[20:29:32]    at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args) in C:\h\w\A4330904\w\A10F08E7\u\performance\artifacts\25dde84f-65d6-4200-8225-6408a8a18724\25dde84f-65d6-4200-8225-6408a8a18724.notcs:line 57
[20:29:32] // AfterAll
[20:29:32] No Workload Results were obtained from the run.
[20:29:32] // Benchmark Process 1084 has exited with code -1.

[TIHan]

Ok, looking into it.

[TIHan]

It's missing the runtimes folder for the partition. Will try to fix this.

[TIHan]

So, the runtimes folder exists when the benchmark is built, however, when the benchmark runs it cannot find those DLLs contained within runtimes. This is what the file/folder structure looks like when I run the same super_pmi py scripts locally:

@BruceForstall @kunalspathak You know of anything special that I may need to do here?

It does fail locally when I try to run through SuperPMI:

[18:14:35] // Benchmark: Scripting.InvokeMethod: Job-DKONAZ(EnvironmentVariables=DOTNET_JitName=superpmi-shim-collector.dll,DOTNET_ZapDisable=1,DOTNET_ReadyToRun=0,DOTNET_TieredCompilation=0, PowerPlanMode=00000000-0000-0000-0000-000000000000, Arguments=/p:EnableUnsafeBinaryFormatterSerialization=true, Toolchain=CoreRun, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [InvokeMethodScript='String'.GetType()]
[18:14:35] // *** Execute ***
[18:14:35] // Launch: 1 / 1
[18:14:35] // Execute: C:\work\runtime\artifacts\tests\coreclr\windows.x64.checked\Tests\9396c921-250d-4a54-9a58-ad2e7eb78d2d\CoreRun.exe 8ceeae80-bf2a-4444-8b00-e9c17f12a81c.dll --anonymousPipes 11616 6728 --benchmarkName "Engine.Scripting.InvokeMethod(InvokeMethodScript: \"'String'.GetType()\")" --job "EnvironmentVariables=DOTNET_JitName=superpmi-shim-collector.dll,DOTNET_ZapDisable=1,DOTNET_ReadyToRun=0,DOTNET_TieredCompilation=0, PowerPlanMode=00000000-0000-0000-0000-000000000000, Arguments=/p:EnableUnsafeBinaryFormatterSerialization=true, Toolchain=CoreRun, InvocationCount=1, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0" --benchmarkId 1 in C:\work\superpmi_payload\workitem\performance\artifacts\8ceeae80-bf2a-4444-8b00-e9c17f12a81c\bin\Release\net8.0\publish
[18:14:35] // BeforeAnythingElse
[18:14:35] 
[18:14:35] System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
[18:14:35]  ---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Management.Automation, Version=7.3.6.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. The system cannot find the file specified.
[18:14:35] File name: 'System.Management.Automation, Version=7.3.6.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'
[18:14:35]    at BenchmarkDotNet.Autogenerated.Runnable_1..ctor()
[18:14:35]    at BenchmarkDotNet.Autogenerated.Runnable_1.Run(IHost host, String benchmarkName) in C:\work\superpmi_payload\workitem\performance\artifacts\8ceeae80-bf2a-4444-8b00-e9c17f12a81c\8ceeae80-bf2a-4444-8b00-e9c17f12a81c.notcs:line 313
[18:14:35]    at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
[18:14:35]    at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
[18:14:35]    --- End of inner exception stack trace ---
[18:14:35]    at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)
[18:14:35]    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
[18:14:35]    at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args) in C:\work\superpmi_payload\workitem\performance\artifacts\8ceeae80-bf2a-4444-8b00-e9c17f12a81c\8ceeae80-bf2a-4444-8b00-e9c17f12a81c.notcs:line 57

[kunalspathak]

I am wondering if the benchmark runs properly after updating dotnet/performance#3420? If yes, you might want to download the payload of other real-world benchmark to see its folder structure and command line parameters for superpmi.

[TIHan]

@kunalspathak I think there is a bug when running benchmarks when specifying a custom corerun where it can't resolve the DLLs in the runtimes folder. I am able to reproduce this locally, and I can see the folder and the DLLs that they contain.

[kunalspathak]

there is a bug when running benchmarks when specifying a custom corerun where it can't resolve the DLLs in the runtimes folder

so this is outside of superpmi and just normally running the benchmarks?

[TIHan]

Running benchmarks normally works if I do this:
PowerShell.Benchmarks.exe --filter *

Supplying a --coreRun causes it to not find the assembly:
PowerShell.Benchmarks.exe --filter * --coreRun C:\work\runtime\artifacts\tests\coreclr\windows.x64.Checked\Tests\Core_Root\corerun.exe

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Management.Automation, Version=7.3.6.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. The system cannot find the file specified.
File name: 'System.Management.Automation, Version=7.3.6.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'
at BenchmarkDotNet.Autogenerated.Runnable_1..ctor()
at BenchmarkDotNet.Autogenerated.Runnable_1.Run(IHost host, String benchmarkName) in C:\work\superpmi_payload\workitem\performance\artifacts\33da4b6b-8187-44e4-8487-966bccb61ed8\33da4b6b-8187-44e4-8487-966bccb61ed8.notcs:line 339
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span1 copyOfArgs, BindingFlags invokeAttr) --- End of inner exception stack trace --- at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span1 copyOfArgs, BindingFlags invokeAttr)
at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)

[TIHan]

@dotnet/jit-contrib @BruceForstall This is ready. It was able to get a collection: https://dev.azure.com/dnceng/internal/_build/results?buildId=2315554&view=results

[BruceForstall]

It looks like there's a problem with the Linux collections, e.g.:

---> System.DllNotFoundException: Unable to load shared library 'advapi32.dll' or one of its dependencies.

also, the Linux collections only add about 140 MC versus about 4300 for Windows.

[TIHan]

@BruceForstall , this is ready again. I checked the OSX and Linux pipelines and they are successfully running the benchmarks and collecting. Now, the Windows benchmarks do run two more than Linux and OSX but this is due to the configuration of the tests themselves and not due to issues of missing assemblies. (Was able to replicate that locally)

https://dev.azure.com/dnceng/internal/_build/results?buildId=2317989&view=results

[BruceForstall]

Not related to this PR, but it looks like the ILLink benchmark is broken / not supported on almost ever platform:

[05:37:32] System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
[05:37:32]  ---> System.TypeInitializationException: The type initializer for 'ILLinkBenchmarks.Utilities' threw an exception.
[05:37:32]  ---> System.NotSupportedException: Unsupported architecture
[05:37:32]    at ILLinkBenchmarks.Utilities.RidString(OSPlatform OS, Architecture Arch, String Libc) in /root/helix/work/workitem/u/performance/src/benchmarks/real-world/ILLink/Utilities.cs:line 27
[05:37:32]    at ILLinkBenchmarks.Utilities..cctor() in /root/helix/work/workitem/u/performance/src/benchmarks/real-world/ILLink/Utilities.cs:line 11

and broken on linux/win-x64:

[05:35:56] System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.

[05:35:56]  ---> System.ComponentModel.Win32Exception (2): An error occurred trying to start process 'dotnet' with working directory 'C:\h\w\B5CD09CD\w\AC3C0947\u\performance\artifacts\27771c95-f0f8-4237-961d-49a50a992eb2\bin\Release\net8.0\publish'. The system cannot find the file specified.

[05:35:56]    at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)

[05:35:56]    at System.Diagnostics.Process.Start(ProcessStartInfo startInfo)

[05:35:56]    at ILLinkBenchmarks.Utilities.PublishSampleProject(String projectFilePath, String[] extraArgs) in C:\h\w\B5CD09CD\w\AC3C0947\u\performance\src\benchmarks\real-world\ILLink\Utilities.cs:line 67

[05:35:56]    at ILLinkBenchmarks.BasicBenchmark.LinkHelloWorldGlobalSetup() in C:\h\w\B5CD09CD\w\AC3C0947\u\performance\src\benchmarks\real-world\ILLink\BasicBenchmark.cs:line 43

[05:35:56]    at BenchmarkDotNet.Engines.EngineFactory.CreateReadyToRun(EngineParameters engineParameters)

[05:35:56]    at BenchmarkDotNet.Autogenerated.Runnable_0.Run(IHost host, String benchmarkName) in C:\h\w\B5CD09CD\w\AC3C0947\u\performance\artifacts\27771c95-f0f8-4237-961d-49a50a992eb2\27771c95-f0f8-4237-961d-49a50a992eb2.notcs:line 182

[05:35:56]    at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)

[05:35:56]    at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)

[05:35:56]    --- End of inner exception stack trace ---

[05:35:56]    at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)

[05:35:56]    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

[05:35:56]    at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args) in C:\h\w\B5CD09CD\w\AC3C0947\u\performance\artifacts\27771c95-f0f8-4237-961d-49a50a992eb2\27771c95-f0f8-4237-961d-49a50a992eb2.notcs:line 57

[TIHan]

Thanks @BruceForstall . Should we file a report to dotnet/performance since it may be a benchmark issue itself?

[BruceForstall]

Thanks @BruceForstall . Should we file a report to dotnet/performance since it may be a benchmark issue itself?

Not sure. We should open a bug against the SPMI collection though