Repository navigation
[NativeAOT] Instrument NativeAOT applications at build time - #9400
Draft
tonyredondo wants to merge 155 commits into
Draft
tonyredondo wants to merge 155 commits into
tonyredondo wants to merge 155 commits into
Conversation
tonyredondo
added
type:enhancement
LLM ValidationLLM Validation Gate — dotnet-tracer-agent🟡 WARN
AnalysisChanged instruction file(s): Warning — 1 case(s) regressed, but none crossed the blocking bar.
Results
Cases
Per-dimension scores, token usage, latency, and estimated cost are in the CI job logs. |
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing This PR (9400) and master. ✅ No regressions detected |
BenchmarksBenchmark execution time: 2026-10-15 20:58:48 Comparing candidate commit 39bc463 in PR branch Found 3 performance improvements and 6 performance regressions! Performance is the same for 63 metrics, 0 unstable metrics, 75 known flaky benchmarks, 51 flaky benchmarks without significant changes.
|
tonyredondo
force-pushed
the
tony/nativeaot-tracer
branch
12 times, most recently
from
October 15, 2026 12:13
a090371 to
4eb9874
Compare
Handler static constructors first consult generic holders filled by a NativeAOT registry, then fall back to IntegrationMapper. The Invoke paths are unchanged, so JIT behavior and performance stay the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dd-trace aot instrument hosts Datadog.Tracer.Native in-process and drives it through an emulated CLR (ICorProfilerInfo and IMetaData* over dnlib), so assemblies are rewritten exactly like at runtime. Replaces the broken apply-aot command. Instrumentation can be verified with ILSpy, JIT preparation and ILVerify. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tion factories A registration can record that an integration method can't be bound, so the handler fails like it does when IntegrationMapper throws. The continuation holder keeps a factory, so the generator is still created from the end handler's static constructor. CallTargetAotRegistry has the runtime checks and the error log the generated registrations use. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ssemblies `dd-trace aot instrument` adds a registration type per rewritten method to the instrumented assembly, generic over the method's generic context, and calls it at the method entry. Its static constructor binds the integration methods the way IntegrationMapper does, so the handlers don't need dynamic code. Shapes with duck typing constraints aren't registered yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tions `dd-trace aot instrument` generates a DuckType AOT registry with the proxies the integration methods' duck typing constraints need, and the adapters create them like IntegrationMapper does. Generic targets get registrations for the closed instantiations found in the instrumented assemblies. The instrumented module initializer enables the AOT mode of duck typing. Also stop the DuckType AOT registry emitter from looking up its generated types in quadratic time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`dd-trace aot instrument --ducktype-map` adds the mappings dynamic duck typing recorded at runtime (DD_DUCKTYPE_DISCOVERY_OUTPUT_PATH) to the DuckType AOT registry, for the proxies created from the runtime type of a value. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eAOT instrumentation A publish passes the application and every assembly ILC compiles, most of them framework references without instrumentation: only the rewritten ones are written, and the closed instantiations are only looked for when a generic method was rewritten. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hout the native tracer An application instrumented at build time has its CallTarget instrumentation compiled in and no native tracer: Instrumentation.InitializeAot only starts the managed parts (tracer, security, diagnostic observers, Activity and OpenTelemetry) and never throws. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…'s module initializer The application's assembly is always written, with a module initializer that enables the DuckType AOT registry and calls Instrumentation.InitializeAot, each step isolated so a failure is logged and the application starts anyway. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Datadog.Trace.Aot.targets runs `dd-trace aot instrument` after ILC computes its inputs and swaps the application and the referenced assemblies the native tracer rewrote, plus the DuckType AOT registry. Only `dotnet publish` with PublishAot is affected; a failure publishes the application uninstrumented with a warning unless DatadogAotFailOnError is set. An opt-in test publishes an ASP.NET Core application and checks its spans reach the agent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Referencing Datadog.Trace.Aot instruments `dotnet publish -p:PublishAot=true` without any setting: the package brings the targets, the dd-trace runner, the native tracer of the build host, the full Datadog.Trace.dll ILC compiles when the application only references the manual API, and a catalog of the duck typing mappings Datadog.Trace looks up from runtime types. The instrumentation is incremental, and recorded mappings of assemblies the application doesn't use are skipped. An opt-in test publishes a manual API application with the package. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The native tracer gives no IL map for CallTarget, but the original instructions are still in the rewritten body, in order: they are aligned and keep their sequence points, the prologue gets the first line and a replaced ret gives its line to the leave that replaced it, as with the original PDB at runtime. Stack traces and debug information of instrumented methods have their lines again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… of validated registries
…ications don't have
…he startup results
… of System.Object
…uild a version of its own
tonyredondo
force-pushed
the
codex/ducktyping-nativeaot
branch
from
October 15, 2026 16:03
83572bc to
d31a301
Compare
tonyredondo
force-pushed
the
tony/nativeaot-tracer
branch
from
October 15, 2026 16:03
cc59c54 to
c3609ad
Compare
…duck typing targets proxies of their own
…yping proxy differently proxies of their own
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of changes
Adds NativeAOT support to the tracer: the
Datadog.Trace.AotNuGet package instruments applications published withPublishAot=trueat build time.Reason for change
NativeAOT applications have no JIT and no dynamic code: the profiler can't rewrite methods, and the tracer can't create its CallTarget adapters and duck typing proxies at runtime.
Implementation details
dd-trace aot instrumenthostsDatadog.Tracer.Nativeoffline (theICorProfilerInfosubset it uses, emulated over dnlib) and rewrites the application and its references before ILC, like the profiler does at runtime.[DuckType]/[DuckCopy]targets,IncludeDerivedTypes, open generic targets).Datadog.Trace.Aot.targetsplugs it intodotnet publish, and the application's module initializer starts the tracer (Instrumentation.InitializeAot).System.Objectserves for every type. Anobjectargument would otherwise get a proxy per type of the application (7,301 inCallTargetNativeTest), and ILC overflows the 1 MB stack of Windows on them.Datadog.Trace.Aotpackage of a pull request build has the build id in its version (<version>-prerelease.<BUILD_BUILDID>): NuGet keeps the first package of a version it extracts, so testing the package of a newer build used the old one. Local,masterand release builds keep<version>-prerelease.Producer<int, string>), which NativeAOT can't build from the generic proxy, get closed proxies (the search of generic instantiations also follows the code that calls the targets' constructors); and the classes of the application deriving from anIncludeDerivedTypestarget that declare a member the proxy binds (a new member, anewmember, a field with the same name) get proxies of their own, bound like dynamic duck typing binds them.Datadog.Trace.TestOptimization.IsSupported), the runtime debugger products (Dynamic Instrumentation, Exception Replay, Symbol Database; Code Origin stays) and their JSON models. TheSamples.HttpMessageHandlerbinary goes from 28.8 MB to 24.4 MB, and its startup is 3.4 ms faster.docs/development/NativeAOT.md,NativeAOT.Plan.md,NativeAOT.Tracker.md,NativeAOT.Samples.md,NativeAOT.Integrations.md.NativeAOT support
Per integration, from the e2e samples (JIT with the profiler compared with NativeAOT), a NativeAOT publish of the 94 applicable samples with the instrumentation reports and ILC warnings, and whether each library declares AOT compatibility. Details in
docs/development/NativeAOT.Integrations.md.When something fails
CallTargetInvokerException, which disables the integration for that typeIntegrationDisabledDueToErrortelemetry; the application keeps runningCheckAotDuckTypeDeclarationsfails in CI for the mappings the integration tests record)DuckTypeAotMissingProxyRegistrationExceptionwhere the proxy is createdDDAOT001lists the methods (an error withDatadogAotFailOnError)CallTargetInvokerExceptionon the first call, the integration disabled for that type)GenericInstantiationsMethod '...' will always throwIntegrations
Status: Supported (same spans and tags as under JIT); Supported, library partial (what the library runs in NativeAOT
has parity, the rest fails also without Datadog); Not supported (library) (the library doesn't run in NativeAOT);
Not verified (no NativeAOT run yet); Not applicable; Deferred (decided not to support yet). "AOT-annotated"
is whether the library's latest version declares trimming/AOT compatibility; a library that doesn't can still work.
Oracle.DataAccess(unmanaged) is .NET Framework onlyGetRecordsneedsStreamARN, also under JITMakeGenericTypeof value types)Logger<T>withMakeGenericTypein its DIMakeGenericTypeExtendedLogger) doesn't run in NativeAOT (DI by reflection, the library)MakeGenericTypeof value types)FFE_FLAGSand exposuresDD_TRACE_OTEL_ENABLEDand the resource; OTLP export of traces (JSON and protobuf, also on .NET 8 since H-73)[Trace],DD_TRACE_METHODSExecute,Query,QuerySingle,ExecuteScalar, sync and async); Dapper alone emits IL at runtime and doesn't run in NativeAOTEFOptimizeContext, queries precompiled when publishing)SaveChangeswith its transaction,Where/SingleAsync,CountAsync,OrderBy/ToListAsync, raw SQL); EF Core's NativeAOT support is experimental (no dynamic queries, noEnsureCreated/migrations at runtime)Reflection.Emitxunit.v3.core.aot); MSTest's is alpha. Test Optimization isn't compiled in NativeAOT applicationsFindings in the tracer
[ERR] ... Expected a null constant(GraphQL, Rebus): dnlib reads the local PDB constants of an enum type it can't resolve as class constants, which its writer rejectsEnumerateTags<TagWriter<__Canon>>: the OTLP export of traces in JSON threwInvalidProgramExceptionon .NET 8will always throwof Datadog.Tracewill always throwof the ASP.NET Core observers in the samples without ASP.NET Core (ILC 8)mongodb.queryspans had no command and the ASP.NET Core route tags were missingDDAOT001) when cutProducer<Null, Null>: no cluster id with other typesProducer<,>andConsumer<,>)Samples.HttpMessageHandler, locally: 26 ms without the tracer, 267 ms with it) is mostly its shutdown: two sleeps nothing interrupts (the DogStatsD worker and the trace serialization thread), also under JITobjectargument of the slow begin path (every proxy implements a generic virtual method, which ILC hashes recursively), and the macOS publish passed 5 minutesSystem.ObjectProducer<int, string>had no proxy (Kafka client handle, cluster id)ProduceAsync: the default delivery handler'sMakeGenericTypeofDeliveryReport<TKey, TValue>had no code, and the exception in the producer constructor's integration left the spans without bootstrap servers and cluster idDatadog.Trace.Kafka.rd.xml)IncludeDerivedTypesproxies served the derived classes bound to the base type's members (a member only a derived class has, or that hides the base's, was bound differently than under JIT)ILC warnings left
The paths that keep their ILC warning because NativeAOT didn't run them:
DiagnosticContextHelper.GetMdcProxy,GetMdlcProxyScopeContextLogsInjectionHelper.AddAttributeToJson4LayoutLoggingConfigurationFileLoaderLoadInstrumentation.OnMethodEndLog4NetCommon.TryAddAppenderToResponseObjectExtractorKnown issues
Bugs found and not fixed, detailed in
NativeAOT.KnownIssues.md:Task<TResult>(e.g.PairTask<TResult, TTag> : Task<TResult>) that isn't completed gets the continuation, aTask<TResult>, converted to the subclass withUnsafe.As: a cast or a member of the subclass then failsBindingFlagswhen it adapts the arguments (e.g. a proxy restricted to public methods selects a private one through duck chaining); with fallback names, such a method of the first name winsDD_DUCKTYPE_DISCOVERY_OUTPUT_PATH, tests) writes the map without its lock when it can't take it within 30 s or can't open the lock file; another process that read the map before and writes it after drops those mappingsTest coverage
RunNativeAotInstrumentationGate: native host and NativeAOT publish tests on Linux x64/arm64, macOS and Windows, among themCallTargetNativeTest(every CallTarget shape) published with NativeAOT, which must behave as with the profiler.CheckAotDuckTypeDeclarations: the Linux integration tests record the duck typing mappings their samples create, and the check fails when one isn't declared.NativeAOT.Samples.md).RunNativeAotPackageTests: once the package is built, the application of the package published with NativeAOT for .NET 8 and .NET 10 (manual API, AppSec, RASP, IAST, binary size, no trimming/AOT warning of Datadog.Trace, no method of Datadog.Trace compiled to throw).RunNativeAotPackageTestsalso compares the spans of the same application published with NativeAOT (with the package) and run under JIT (with the profiler and the tracer home): ASP.NET Core, HttpClient, the manual API, AppSec/RASP/IAST, SQLite with Dapper.AOT and errors, for .NET 8 and .NET 10.Samples.HttpMessageHandlerandSamples.FakeDbCommandpublished with NativeAOT for .NET 8 and .NET 10 on a hosted Windows agent (publish_nativeaot_samples),Baseline(without the tracer) andCallTarget+NativeAOTscenarios, in the execution benchmarks report. First CI results (.NET 8): FakeDbCommand 23 → 105 ms, HttpMessageHandler 115 → 359 ms, with the shutdown sleeps [Tracer] Wake up the trace serialization and DogStatsD threads at shutdown #9419 removes. No overhead threshold until [Tracer] Wake up the trace serialization and DogStatsD threads at shutdown #9419 is merged.Other details
Stacked on #8252 (base
codex/ducktyping-nativeaot). Draft, to run CI.