Run NativeAOT CLI tests in CI (win-x64, linux-x64, osx-arm64) by NikolaMilosavljevic · Pull Request #54719 · dotnet/sdk

NikolaMilosavljevic · 2026-06-11T17:28:12Z

Summary

PR #54297 added the dotnet-aot.Tests project (NativeAOT CLI entry-point tests) but did not hook them up to any pipeline, so they never run in CI. This change wires them up so they run for each platform that currently runs tests in CI.

These tests are run on the build agent (not Helix): the test assembly is published as a NativeAOT binary and executed on the agent. NativeAOT binaries must run on the architecture they were built for, so the runner scripts auto-detect the agent's host RID (via .dotnet/dotnet --info). On a cross-architecture leg (e.g. the macOS leg cross-builds osx-arm64 on an x64 hosted agent), the tests run for the host architecture (osx-x64).

Changes

test/dotnet-aot.Tests/run-aot-tests.ps1 / .sh — add an optional -Trx/--trx flag (plus results directory) that runs the AOT test binary with the MTP --report-trx options, emitting a TRX report under artifacts/TestResults/<config>/ so CI can publish results. Default-off, so local behavior is unchanged. The PowerShell script uses [System.IO.Path]::Combine for path joins so it runs under Windows PowerShell 5.1 (the CI powershell: task host), which does not support multi-argument Join-Path.
eng/pipelines/templates/jobs/sdk-build.yml — add a runNativeAotCliTests parameter (default false). When enabled, after the Helix queue step (so it never blocks the broader Helix test submission), publish + run the AOT test binary, then publish the TRX via PublishTestResults@2. The scripts auto-detect the host RID rather than being passed the build's target RID.
eng/pipelines/templates/jobs/sdk-job-matrix.yml — enable runNativeAotCliTests: true on the Windows (x64), Linux (x64), and macOS TestBuild legs — the platforms that run tests in CI.
eng/pipelines/templates/jobs/sdk-build.yml (Linux step) — set SHELL for the AOT test step. --cli-schema evaluates a shell-completion default that throws when SHELL is unset in the bare AzDO script: task.
test/dotnet-aot.Tests/AotParserTests.cs — running the tests under NativeAOT for the first time surfaced a real AOT issue from Unify the managed and NativeAOT CLI parsers into one shared implementation #54653: GetFileBasedAppEntryPointToken pulls in Microsoft.Build, which cannot be linked into a NativeAOT image (ILC reports the method "will always throw"). The three affected tests are skipped under AOT only (guarded by RuntimeFeature.IsDynamicCodeSupported, so the managed test run still exercises them), tracked by #54806.

Notes

This is distinct from runAoTTests/RunAoTTests, which runs the Blazor WebAssembly AoT tests on Helix.
Validated locally on Windows (under Windows PowerShell 5.1): the AOT publish + run succeeds with 0 failed, 52 passed, 9 skipped. The skipped tests are 6 integration tests that require the shipped dn binary in the SDK layout plus the 3 file-based-app tests quarantined by #54806.

Hook up test/dotnet-aot.Tests (added in dotnet#54297) to CI. These tests publish the test assembly as a NativeAOT binary and run it on the build agent (same architecture, native toolchain already present from the build), validating the AOT CLI entry-point code under NativeAOT compilation. - run-aot-tests.ps1/.sh: add optional -Trx/--trx (and results dir) to emit a TRX report via the MTP --report-trx options so CI can publish results. - sdk-build.yml: add runNativeAotCliTests parameter; after the Helix queue step (so it never blocks Helix submission), publish+run the AOT test binary with an explicit RID and publish the TRX results. - sdk-job-matrix.yml: enable runNativeAotCliTests on the Windows (x64), Linux (x64), and macOS (arm64) TestBuild legs. Note: this is distinct from runAoTTests/RunAoTTests, which runs the Blazor WebAssembly AoT tests on Helix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

This PR wires up the existing dotnet-aot.Tests (NativeAOT CLI entry-point tests) to actually run in CI on the build agents that already have the NativeAOT toolchain available (win-x64, linux-x64, osx-arm64). It adds optional TRX emission support to the test runner scripts and integrates execution + result publishing into the SDK build pipeline templates.

Changes:

Add optional --trx/-Trx + results directory support to the AOT test runner scripts so CI can publish TRX results.
Add a new pipeline switch (runNativeAotCliTests) and, when enabled, publish+run the AOT test binary on the build agent and publish the TRX.
Enable the switch for the CI TestBuild legs on Windows x64, Linux x64, and macOS arm64.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
test/dotnet-aot.Tests/run-aot-tests.sh	Adds optional TRX/report output arguments for CI publishing while keeping default local behavior unchanged.
test/dotnet-aot.Tests/run-aot-tests.ps1	Adds optional TRX/report output arguments for CI publishing while keeping default local behavior unchanged.
eng/pipelines/templates/jobs/sdk-job-matrix.yml	Enables the new NativeAOT CLI test run on the intended OS/arch TestBuild legs.
eng/pipelines/templates/jobs/sdk-build.yml	Adds the `runNativeAotCliTests` parameter and runs/publishes results after Helix queueing.

The ARM64 TESTBUILD job in .vsts-ci.yml was the only sdk-job-matrix invocation not passing oneESCompat, so its PublishBuildArtifacts@1/PublishPipelineArtifact@1 tasks emitted without the 1ES. prefix and were rejected by 1ES YAML validation when the job is enabled (enableArm64Job=true). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The separate enableArm64Job ARM64 TESTBUILD job became redundant after dotnet#54604 switched the main macOS PR leg to arm64 (both produce job TestBuild_macOS_arm64). It should not be enabled, so the oneESCompat fix is unnecessary; reverting to keep this PR scoped to the AOT test hookup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Two CI failures surfaced when the AOT test step ran: 1. Windows used Windows PowerShell 5.1, which does not support multi-argument Join-Path (PS 6+ only). Replaced the 3+ arg Join-Path calls with [System.IO.Path]::Combine. 2. The macOS leg cross-builds osx-arm64 on an x64 hosted agent, so the native arm64 binary could not execute (Bad CPU type). NativeAOT binaries must run on the host architecture, so the pipeline no longer passes the build's target RID; the scripts auto-detect the host RID via .dotnet/dotnet --info instead. Validated end-to-end under Windows PowerShell 5.1: 40 passed, 6 skipped, 0 failed, TRX emitted. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Running the AOT CLI tests in CI surfaced real failures from the new tests added in dotnet#54653: * GetFileBasedAppEntryPointToken pulls in Microsoft.Build, which cannot be linked into a NativeAOT image (ILC reports the method 'will always throw'). Skip the three affected AotParserTests under NativeAOT (guarded by RuntimeFeature.IsDynamicCodeSupported, so the managed test run still exercises them). Tracked by dotnet#54806. * InvokeCliSchema_RendersSchemaJsonFromAot evaluates a shell-completion default that requires SHELL to be set; the bare AzDO script task does not export it on Linux. Set SHELL on the AOT test step. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

NikolaMilosavljevic · 2026-06-16T18:36:05Z

Heads-up @baronfel: hooking these tests up to CI runs test/dotnet-aot.Tests under NativeAOT for the first time, which surfaced a real AOT issue from #54653.

ParseResultExtensions.GetFileBasedAppEntryPointToken → VirtualProjectBuilder.IsValidEntryPointPath pulls in Microsoft.Build, which can't be linked into a NativeAOT image. ILC flags it at publish time:

ILC: Method '...ParseResultExtensions.GetFileBasedAppEntryPointToken(ParseResult)'
  will always throw because: Failed to load assembly 'Microsoft.Build'

Since NativeEntryPoint calls this to detect/defer dotnet app.cs invocations, it affects the real dn, not just the tests. I've filed #54806 and quarantined the three affected AotParserTests under AOT only (guarded by RuntimeFeature.IsDynamicCodeSupported, so the managed test run still exercises them). Once file-based app detection is made AOT-safe, the skips should be removed.

I also set SHELL on the Linux AOT test step — --cli-schema evaluates a shell-completion default that throws when SHELL is unset in the bare AzDO script: task.

NikolaMilosavljevic · 2026-06-16T19:59:05Z

CI status note: latest red is an unrelated broken `main`, not this PR

The most recent CI run (build 1466921) is red on essentially every leg, but the failures are not caused by this PR.

Root cause: all legs fail early at the main 🟣 Build task with MSTEST analyzer errors (MSTEST0037 / MSTEST0023, treated as errors) in test/TemplateEngine/Microsoft.TemplateEngine.Authoring.Tasks.IntegrationTests/. These come from #54758 (migrating that project to MSTest.Sdk on MTP), whose commit is currently main's HEAD — so main itself is red and every PR targeting it is affected.

Evidence this isn't us:

All 22 build errors in the failing leg are in a single project: Microsoft.TemplateEngine.Authoring.Tasks.IntegrationTests.csproj. None reference any AOT file.
Our projects compile cleanly in the same log — dotnet-aot.dll and dotnet-aot.Tests.dll both build successfully.
This branch has no commits touching that TemplateEngine directory; it only appears here via the main merge from "Update branch".

Next step: once main is green again, I'll re-run CI here to confirm the NativeAOT CLI test legs (win-x64, linux-x64, osx-arm64) pass. No changes to this PR are needed for the current failures.

# Conflicts: # test/dotnet-aot.Tests/AotParserTests.cs

# Conflicts: # eng/pipelines/templates/jobs/sdk-job-matrix.yml

…tial test classes The "Run NativeAOT CLI Tests" CI step reported "Zero tests ran" (MTP exit code 8): the AOT test binary published successfully but discovered no tests. This was a pre-existing failure on main (introduced with the step in #54719) that our rebase inherited. Root cause is a version skew in the MSTest AOT source-generation pipeline. The repo flows the MSTest framework/adapter/platform at the 26360.5 build, but MSTest.Sdk (global.json) defaults MSTestSourceGenerationVersion to the stale 26325.12 build. The 26325 generator is incompatible with the 26360 framework and emits no source-generated reflection metadata, so reflection-free (NativeAOT) discovery finds zero test classes. Fixes (test/dotnet-aot.Tests): - Pin MSTestSourceGenerationVersion to 2.0.0-alpha.26360.5 so the generator matches the flowed framework/adapter/platform version. - Set MSTestSourceGenMode=ReflectionFree and register it as a CompilerVisibleProperty so the generator emits reflection metadata even when the adapter default does not apply in the inner AOT build. - Mark the five [TestClass] types partial, which MSTest source generation requires to augment test classes. Verified locally that a clean AOT-config compile now emits MSTestReflectionMetadata registrations for all five test classes. The native run itself is validated in CI (local ILC is unavailable for the preview.7 nativeaot runtime pack). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: b7c0dc09-64f3-40b8-9c5f-45b0fce2e461

…iscovery The test/dotnet-aot.Tests project publishes as NativeAOT and relies on the MSTest source generator for test discovery (reflection isn't AOT-safe). That requires MSTest.Sdk (the MSBuild SDK pinned in global.json) and the MSTest framework packages (MSTestPackageVersion, flowed by darc from microsoft/testfx) to be coherent. A dotnet/dotnet flow bumped MSTestPackageVersion to 4.4.0-preview.26360.5 while MSTest.Sdk stayed at 4.3.0-preview.26325.12, and PR #54719 then enabled the NativeAOT CLI test leg. The SDK/framework skew makes the source generator register zero tests, so the leg fails with MTP exit code 8 (Test run summary: Zero tests ran) on every main build. Normal reflection-based MSTest runs tolerate the skew, so only the AOT leg breaks. Fix: bump MSTest.Sdk to 4.4.0-preview.26360.5 (same testfx build as the MSTest framework), and track MSTest.Sdk as a darc dependency in eng/Version.Details.xml. Like Microsoft.DotNet.Arcade.Sdk and .Helix.Sdk, darc then updates its global.json msbuild-sdks version in lockstep with the MSTest framework packages, preventing the versions from drifting apart again. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 59570d89-a7e3-4993-baa9-3fb1f442a617

Copilot AI review requested due to automatic review settings June 11, 2026 17:28

NikolaMilosavljevic requested a review from MiYanni as a code owner June 11, 2026 17:28

Copilot started reviewing on behalf of NikolaMilosavljevic June 11, 2026 17:28 View session

Copilot AI reviewed Jun 11, 2026

View reviewed changes

NikolaMilosavljevic requested review from JeremyKuhne and baronfel June 11, 2026 17:40

NikolaMilosavljevic and others added 4 commits June 11, 2026 10:51

Merge branch 'main' into tests.hookup

fe48f22

NikolaMilosavljevic mentioned this pull request Jun 16, 2026

NativeAOT CLI: file-based app detection throws FileNotFoundException for 'Microsoft.Build' under AOT #54806

Closed

This was referenced Jun 16, 2026

Template engine tests hitting known parallelism issue in GetVisualStudioInstances #44878

Closed

Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.DeleteSubfolder has flaky test assertion #49233

Closed

NikolaMilosavljevic added 3 commits June 18, 2026 10:19

Merge branch 'main' into tests.hookup

d393ece

Merge remote-tracking branch 'upstream/main' into tests.hookup

b9c8d40

# Conflicts: # test/dotnet-aot.Tests/AotParserTests.cs

Merge remote-tracking branch 'upstream/main' into tests.hookup

bd01601

build-analysis Bot mentioned this pull request Jun 23, 2026

One process not started with aspire integ test #54921

Closed

NikolaMilosavljevic added 2 commits June 24, 2026 09:21

Merge branch 'main' into tests.hookup

b3e48a4

Merge branch 'main' into tests.hookup

7abd989

build-analysis Bot mentioned this pull request Jun 25, 2026

Flaky test: Nuget_reference_compat fails with ArgumentException "Destination is too short" #54960

Closed

Merge remote-tracking branch 'upstream/main' into tests.hookup

06a02bd

# Conflicts: # eng/pipelines/templates/jobs/sdk-job-matrix.yml

NikolaMilosavljevic requested a review from MichaelSimons as a code owner June 29, 2026 18:10

baronfel enabled auto-merge (squash) July 6, 2026 15:35

mthalman approved these changes Jul 14, 2026

View reviewed changes

baronfel merged commit 1c43067 into dotnet:main Jul 14, 2026
25 checks passed

dotnet-maestro Bot mentioned this pull request Jul 15, 2026

[main] Source code updates from dotnet/sdk dotnet/dotnet#7750

Merged

dotnet-milestone-bot Bot added this to the 11.0-preview7 milestone Jul 15, 2026

Evangelink mentioned this pull request Jul 15, 2026

Add dotnet test --list-tests json via SDK-side rendering #55299

Merged

dsplaisted mentioned this pull request Jul 15, 2026

Keep MSTest.Sdk in sync with MSTest framework to fix NativeAOT test discovery #55302

Merged

Navigation Menu

Uh oh!

Run NativeAOT CLI tests in CI (win-x64, linux-x64, osx-arm64) - #54719

Run NativeAOT CLI tests in CI (win-x64, linux-x64, osx-arm64)#54719
baronfel merged 12 commits into
dotnet:mainfrom
NikolaMilosavljevic:tests.hookup

NikolaMilosavljevic commented Jun 11, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

NikolaMilosavljevic commented Jun 16, 2026

Uh oh!

NikolaMilosavljevic commented Jun 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

NikolaMilosavljevic commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

NikolaMilosavljevic commented Jun 16, 2026

Uh oh!

NikolaMilosavljevic commented Jun 16, 2026

CI status note: latest red is an unrelated broken main, not this PR

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

NikolaMilosavljevic commented Jun 11, 2026 •

edited

Loading

CI status note: latest red is an unrelated broken `main`, not this PR