Skip to content

Add Metadata inspection #2241

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed

Conversation

lihaMSFT
Copy link

@lihaMSFT lihaMSFT commented Jan 6, 2023

Hi,

This PR adds a runtime agnostic way to read metadata in order to list benchmarks in a BenchmarkDotNet assembly. This allows us to get benchmark names from a .NET Core assembly in .NET Framework since System.Reflection is not available.

@AndreyAkinshin
Copy link
Member

@lihaMSFT how MetadataExtensions is supposed to be used? At the moment, there are not usages of this class.

@lihaMSFT
Copy link
Author

lihaMSFT commented Jan 9, 2023

This is intended as a workaround for cross-toolchain benchmark discovery (#1572, #1460). We are using it to get a list of benchmarks which are then passed into the benchmark console app as command line arguments.

@lihaMSFT lihaMSFT closed this Jan 10, 2023
@adamsitnik
Copy link
Member

@lihaMSFT as mentioned offline this approach would not work well with benchmarks that have dynamically created list of arguments: https://benchmarkdotnet.org/articles/samples/IntroArgumentsSource.html#sample-introargumentssource

one of the alternatives would be to ask BDN to print the benchmarks tree:

dotnet run -c Release -f net7.0 --filter * --list tree

and parse the output:

BenchmarkDotNet
 └─Samples
    ├─Algo_Md5VsSha256
    │  ├─Md5
    │  └─Sha256
    ├─IntroArguments
    │  └─Benchmark
    ├─IntroArgumentsPriority
    │  ├─ManyArguments
    │  └─Benchmark
    ├─IntroArgumentsSource
    │  ├─SingleArgument
    │  └─ManyArguments
    ├─IntroArrayParam
    │  ├─ArrayIndexOf
    │  └─ManualIndexOf
    ├─IntroBasic
    │  ├─Sleep
    │  └─SleepWithDescription
    ├─IntroBenchmarkBaseline
    │  ├─Time50
    │  ├─Time100
    │  └─Time150
    ├─IntroCategories
    │  ├─A1
    │  ├─A2
    │  └─B1
    ├─IntroCategoryBaseline
    │  ├─Time50
    │  ├─Time100
    │  ├─Time550
    │  └─Time600
    ├─IntroColdStart
    │  └─Foo
    ├─IntroConfigSource
    │  └─Foo
    ├─IntroConfigUnion
    │  └─Foo
    ├─IntroCultureInfo
    │  └─Foo
    ├─IntroCustomMono
    │  └─Foo
    ├─IntroCustomMonoArguments
    │  └─Sample
    ├─IntroCustomMonoFluentConfig
    │  └─Foo
    ├─IntroCustomMonoObjectStyle
    │  └─Foo
    ├─IntroCustomMonoObjectStyleAot
    │  └─Foo
    ├─IntroDeferredExecution
    │  ├─Wrong
    │  └─Ok
    ├─IntroDisassembly
    │  ├─SumLocal
    │  └─SumField
    ├─IntroDisassemblyAllJits
    │  └─CallVirtualMethod
    ├─IntroDisassemblyDry
    │  └─Foo
    ├─IntroDisassemblyRyuJit
    │  ├─SumLocal
    │  └─SumField
    ├─IntroEnvVars
    │  └─Foo
    ├─IntroEventPipeProfiler
    │  └─Sleep
    ├─IntroEventPipeProfilerAdvanced
    │  └─RentAndReturn_Shared
    ├─IntroExceptionDiagnoser
    │  └─ThrowExceptionRandomly
    ├─IntroExport
    │  ├─Sleep10
    │  └─Sleep50Noisy
    ├─IntroExportJson
    │  ├─Sleep10
    │  └─Sleep20
    ├─IntroExportXml
    │  ├─Sleep10
    │  └─Sleep20
    ├─IntroFilters
    │  ├─A1
    │  ├─A2
    │  ├─A3
    │  ├─B1
    │  └─C1
    ├─IntroGcMode
    │  ├─Allocate
    │  └─AllocateWithStackalloc
    ├─IntroGenericTypeArguments<Int32>
    │  └─Create
    ├─IntroGenericTypeArguments<Char>
    │  └─Create
    ├─IntroHardwareCounters
    │  ├─SortedBranch
    │  ├─UnsortedBranch
    │  ├─SortedBranchless
    │  └─UnsortedBranchless
    ├─IntroHidingColumns
    │  └─AllocateArray
    ├─IntroInliningDiagnoser
    │  ├─IterationTest
    │  └─SplitJoin
    ├─IntroInProcess
    │  ├─Allocate
    │  └─AllocateWithStackalloc
    ├─IntroInProcessWrongEnv
    │  ├─Allocate
    │  └─AllocateWithStackalloc
    ├─IntroJobBaseline
    │  └─SplitJoin
    ├─IntroJoin1
    │  ├─A
    │  └─B
    ├─IntroJoin2
    │  ├─A
    │  └─B
    ├─IntroJsonExportObjectStyle
    │  ├─Sleep10
    │  └─Sleep20
    ├─IntroLargeAddressAware
    │  └─AllocateMoreThan2GB
    ├─IntroMemoryRandomization
    │  ├─Array_RandomizationEnabled
    │  └─Array_RandomizationDisabled
    ├─IntroMonitoring
    │  └─Foo
    ├─IntroMultimodal
    │  ├─Unimodal
    │  ├─Bimodal
    │  ├─Trimodal
    │  └─Quadrimodal
    ├─IntroNativeMemory
    │  ├─BitmapWithLeaks
    │  ├─Bitmap
    │  ├─AllocHGlobal
    │  └─AllocHGlobalWithLeaks
    ├─IntroNuGet
    │  └─SerializeAnonymousObject
    ├─IntroOrderAttr
    │  ├─Slow
    │  └─Fast
    ├─IntroOrderManual
    │  ├─Fast
    │  └─Slow
    ├─IntroOutliers
    │  └─Foo
    ├─IntroParams
    │  └─Benchmark
    ├─IntroParamsAllValues
    │  └─Benchmark
    ├─IntroParamsPriority
    │  └─Benchmark
    ├─IntroParamsSource
    │  └─Benchmark
    ├─IntroPercentiles
    │  ├─ConstantDelays
    │  ├─RandomDelays
    │  └─RareDelays
    ├─IntroPerfCollectProfiler
    │  └─WriteAllText
    ├─IntroPowerPlan
    │  ├─IterationTest
    │  └─SplitJoin
    ├─IntroRankColumn
    │  ├─Foo
    │  └─Bar
    ├─IntroRatioSD
    │  ├─Base
    │  ├─Slow
    │  └─Fast
    ├─IntroRatioStyle
    │  ├─Baseline
    │  ├─Bar
    │  └─Foo
    ├─IntroSetupCleanupGlobal
    │  └─Logic
    ├─IntroSetupCleanupIteration
    │  └─Benchmark
    ├─IntroSetupCleanupTarget
    │  ├─BenchmarkA
    │  ├─BenchmarkB
    │  ├─BenchmarkC
    │  └─BenchmarkD
    ├─IntroStaThread
    │  └─CheckForSTA
    ├─IntroStatisticalTesting
    │  ├─Sleep50
    │  ├─Sleep97
    │  ├─Sleep99
    │  ├─Sleep100
    │  ├─Sleep101
    │  ├─Sleep103
    │  └─Sleep150
    ├─IntroStatisticsColumns
    │  ├─Md5A
    │  ├─Md5B
    │  └─Sha256
    ├─IntroStopOnFirstError
    │  ├─FirstMethod
    │  └─SecondMethod
    ├─IntroTagColumn
    │  ├─Foo1
    │  ├─Foo12
    │  ├─Bar3
    │  └─Bar34
    ├─IntroTailcall
    │  └─Calc
    ├─IntroThreadingDiagnoser
    │  └─CompleteOneWorkItem
    ├─IntroUnicode
    │  └─Foo
    ├─IntroUnicodeFluentConfig
    │  └─Foo
    ├─IntroUnicodeObjectStyle
    │  └─Foo
    ├─IntroWasmCmdConfig
    │  └─Foo
    └─IntroWasmFluentConfig
       └─Foo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants