[JIT / Dynamic PGO] Silent data loss / premature termination in OpenXmlReader iterator under .NET 10 Release with Tiered PGO

[BenYtt]

Environment

Key	Value
OS	Windows 11 (win-x64, 10.0.26100)
Architecture	x64
.NET SDK	10.0.400 (Commit: 14fbf8d527)
.NET Runtime	10.0.11 (Commit: e2f47b0110)
DocumentFormat.OpenXml	3.1.1
ClosedXML	0.105.0
Configuration	Release (-c Release), Tiered PGO enabled (default) vs <TieredPGO>false</TieredPGO>

Summary

When iterating large OpenXML worksheets (DocumentFormat.OpenXml.OpenXmlReader) via an IEnumerable<(int, Row)> iterator (yield return state machine with LoadCurrentElement()), .NET 10 JIT Tiered PGO causes silent data loss where the enumeration prematurely terminates or skips rows without throwing any exceptions.

Original Signal

The defect was first identified in a 1,000,000-row unit test exercising OpenXmlReader streaming enumeration in Release mode:

Run	Configuration	TieredPGO	Result
1	net10.0 Release	Default (enabled)	Expected 1,000,000 — Actual 18,929 (FAIL)
2	net10.0 Release	Default (enabled)	Expected 1,000,000 — Actual 16,572 (FAIL)
3	net10.0 Release	<TieredPGO>false</TieredPGO>	Expected 1,000,000 — Actual 1,000,000 (PASS)

Isolated Behavior & Root Cause Analysis

In this standalone reproduction:

1. Under Debug or with <TieredPGO>false</TieredPGO>, row counts always match the baseline DOM traversal count (100% PASS across all dataset sizes and iterations).
2. Under Release with Tiered PGO enabled, Tier 0 / early Tier 1 invocations execute correctly. Once Tier 1 PGO optimization completes (triggered after sufficient loop iterations or method invocations, e.g. ~150k–300k elements processed), subsequent enumerations immediately yield 0 rows or terminate prematurely mid-stream.
3. The smallest dataset reproducing this reliably within 5 iterations is 35,000 data rows (3/5 failures, dropping to 0 rows by iteration 4). At 40,000+ data rows, 5/5 iterations fail immediately. At smaller row counts (e.g., 5,000 or 10,000 rows), the failure reproduces consistently once iteration count reaches the PGO recompilation threshold.
4. Setting <TieredPGO>false</TieredPGO> (or DOTNET_TieredPGO=0) completely eliminates the corruption across all sizes and iteration counts.

Root cause within the JIT codegen has not been isolated to a specific instruction sequence; the confirmed runtime mitigation is DOTNET_TieredPGO=0 / <TieredPGO>false</TieredPGO>.

Minimal Repro Project

Project File (dotnet10-tieredpgo-openxml-repro.csproj)

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <RootNamespace>dotnet10_tieredpgo_openxml_repro</RootNamespace>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <!-- Toggle TieredPGO to true (reproduces bug) or false (mitigates bug) -->
    <TieredPGO>true</TieredPGO>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="ClosedXML" Version="0.105.0" />
    <PackageReference Include="DocumentFormat.OpenXml" Version="3.1.1" />
  </ItemGroup>

</Project>

Source Code (Program.cs)

using System.Diagnostics;
using ClosedXML.Excel;
using DocumentFormat.OpenXml.Packaging;
using DocumentFormat.OpenXml.Spreadsheet;
using DocumentFormat.OpenXml;

var totalRows = args.Length > 0 ? int.Parse(args[0]) : 40000;
var iterations = args.Length > 1 ? int.Parse(args[1]) : 5;

Console.WriteLine($"=== Repro Harness (.NET {Environment.Version}) ===");
Console.WriteLine($"Target rows: {totalRows} (+1 header = {totalRows + 1} total rows expected)");
Console.WriteLine($"Iterations: {iterations}");
Console.WriteLine();

Console.Write($"Generating workbook with {totalRows} data rows... ");
var sw = Stopwatch.StartNew();
using var ms = BuildWorkbook(totalRows);
var bytes = ms.ToArray();
sw.Stop();
Console.WriteLine($"Done in {sw.ElapsedMilliseconds} ms ({bytes.Length:N0} bytes).");
Console.WriteLine();

// DOM baseline check (verifies physical XML structure is intact)
Console.Write("Calculating DOM baseline row count... ");
sw.Restart();
var domCount = CountRowsDom(new MemoryStream(bytes));
sw.Stop();
Console.WriteLine($"DOM Count = {domCount} (took {sw.ElapsedMilliseconds} ms).");
Console.WriteLine();

var mismatches = 0;
for (var iter = 1; iter <= iterations; iter++)
{
    sw.Restart();
    using var stream = new MemoryStream(bytes);
    var saxCount = CountRowsSax(stream);
    sw.Stop();

    var matches = (saxCount == domCount);
    if (!matches)
    {
        mismatches++;
    }

    var status = matches ? "MATCH" : $"MISMATCH (diff: {saxCount - domCount})";
    Console.WriteLine($"Iteration {iter,2}/{iterations}: SAX = {saxCount,8} | DOM = {domCount,8} | {status} ({sw.ElapsedMilliseconds,5} ms)");
}

Console.WriteLine();
Console.WriteLine($"Summary: {iterations - mismatches}/{iterations} passed, {mismatches} mismatches.");
return mismatches > 0 ? 1 : 0;

static MemoryStream BuildWorkbook(int totalRows)
{
    using var wb = new XLWorkbook();
    var ws = wb.AddWorksheet("Sheet1");
    ws.Cell(1, 1).Value = "Zone code";
    ws.Cell(1, 2).Value = "ISO A2 country code";
    ws.Cell(1, 3).Value = "Origin Postal code";
    ws.Cell(1, 4).Value = "City";
    ws.Cell(1, 5).Value = "Postal code from";
    ws.Cell(1, 6).Value = "Postal code to";

    for (var i = 0; i < totalRows; i++)
    {
        var r = i + 2;
        var from = i * 10;
        var to = from + 9;
        ws.Cell(r, 1).Value = $"Z{(i % 100):D3}";
        ws.Cell(r, 2).Value = "SE";
        ws.Cell(r, 3).Value = string.Empty;
        ws.Cell(r, 4).Value = $"City{(i % 50)}";
        ws.Cell(r, 5).Value = from.ToString("D8");
        ws.Cell(r, 6).Value = to.ToString("D8");
    }

    var ms = new MemoryStream();
    wb.SaveAs(ms);
    ms.Position = 0;
    return ms;
}

static int CountRowsSax(Stream xlsxStream)
{
    var count = 0;
    foreach (var row in StreamRows(xlsxStream))
    {
        count++;
    }
    return count;
}

static IEnumerable<(int RowNum, Row RowElement)> StreamRows(Stream xlsxStream)
{
    using var document = SpreadsheetDocument.Open(xlsxStream, isEditable: false);
    var workbookPart = document.WorkbookPart!;
    var sheet = workbookPart.Workbook.Sheets!.Elements<Sheet>().First();
    var worksheetPart = (WorksheetPart)workbookPart.GetPartById(sheet.Id!.Value!);

    using var reader = OpenXmlReader.Create(worksheetPart);

    while (reader.Read())
    {
        if (!reader.ElementType.Equals(typeof(Row)))
            continue;

        var rowElement = (Row)reader.LoadCurrentElement()!;
        var rowNum = (int)(rowElement.RowIndex?.Value ?? 0);

        yield return (rowNum, rowElement);
    }
}

static int CountRowsDom(Stream xlsxStream)
{
    using var document = SpreadsheetDocument.Open(xlsxStream, isEditable: false);
    var workbookPart = document.WorkbookPart!;
    var sheet = workbookPart.Workbook.Sheets!.Elements<Sheet>().First();
    var worksheetPart = (WorksheetPart)workbookPart.GetPartById(sheet.Id!.Value!);
    return worksheetPart.Worksheet.Descendants<Row>().Count();
}

Results Matrix

1. Row Count vs Reproduction Rate (Release Mode, TieredPGO=true, N=5)

┌───────────┬────────────────────────────────────┬─────────────────────┬────────────────────────────────────────────────────┬────────────┐
│ Data Rows │ Expected Total Rows (incl. Header) │ Mismatches / 5 Runs │              Run Details (SAX Counts)              │   Status   │
├───────────┼────────────────────────────────────┼─────────────────────┼────────────────────────────────────────────────────┼────────────┤
│ 1,000,000 │ 1,000,001                          │ 5 / 5               │ Iter 1: 29,538; Iter 2–5: 0                        │ FAIL       │
├───────────┼────────────────────────────────────┼─────────────────────┼────────────────────────────────────────────────────┼────────────┤
│ 500,000   │ 500,001                            │ 5 / 5               │ Iter 1: 42,028; Iter 2–5: 0                        │ FAIL       │
├───────────┼────────────────────────────────────┼─────────────────────┼────────────────────────────────────────────────────┼────────────┤
│ 250,000   │ 250,001                            │ 5 / 5               │ Iter 1: 20,983; Iter 2–5: 0                        │ FAIL       │
├───────────┼────────────────────────────────────┼─────────────────────┼────────────────────────────────────────────────────┼────────────┤
│ 100,000   │ 100,001                            │ 5 / 5               │ Iter 1: 31,343; Iter 2–5: 0                        │ FAIL       │
├───────────┼────────────────────────────────────┼─────────────────────┼────────────────────────────────────────────────────┼────────────┤
│ 50,000    │ 50,001                             │ 5 / 5               │ Iter 1: 45,071; Iter 2–5: 0                        │ FAIL       │
├───────────┼────────────────────────────────────┼─────────────────────┼────────────────────────────────────────────────────┼────────────┤
│ 40,000    │ 40,001                             │ 5 / 5               │ Iter 1: 39,937; Iter 2–5: 0                        │ FAIL       │
├───────────┼────────────────────────────────────┼─────────────────────┼────────────────────────────────────────────────────┼────────────┤
│ 35,000    │ 35,001                             │ 3 / 5               │ Iter 1–2: 35,001; Iter 3: 6,480; Iter 4–5: 0       │ FAIL       │
├───────────┼────────────────────────────────────┼─────────────────────┼────────────────────────────────────────────────────┼────────────┤
│ 30,000    │ 30,001                             │ 1 / 5               │ Iter 1–4: 30,001; Iter 5: 10,902                   │ PARTIAL    │
├───────────┼────────────────────────────────────┼─────────────────────┼────────────────────────────────────────────────────┼────────────┤
│ 20,000    │ 20,001                             │ 0 / 5               │ Iter 1–5: 20,001 (Reproduces on Iter 8 with N=20)  │ PASS (N=5) │
├───────────┼────────────────────────────────────┼─────────────────────┼────────────────────────────────────────────────────┼────────────┤
│ 10,000    │ 10,001                             │ 0 / 5               │ Iter 1–5: 10,001 (Reproduces on Iter 34 with N=50) │ PASS (N=5) │
└───────────┴────────────────────────────────────┴─────────────────────┴────────────────────────────────────────────────────┴────────────┘

2. Multi-Iteration PGO Threshold Analysis (TieredPGO=true)

┌───────────┬──────────────────┬────────────┬───────────────────────────┬────────────────────────────────┐
│ Data Rows │ Total Iterations │ Mismatches │ First Corrupted Iteration │             Notes              │
├───────────┼──────────────────┼────────────┼───────────────────────────┼────────────────────────────────┤
│ 20,000    │ 20               │ 13 / 20    │ Iteration 8 (18,585 rows) │ Iterations 9–20 yield 0 rows   │
├───────────┼──────────────────┼────────────┼───────────────────────────┼────────────────────────────────┤
│ 10,000    │ 50               │ 17 / 50    │ Iteration 34 (1 row)      │ Iterations 35–50 yield 0 rows  │
├───────────┼──────────────────┼────────────┼───────────────────────────┼────────────────────────────────┤
│ 5,000     │ 100              │ 87 / 100   │ Iteration 14 (1,392 rows) │ Iterations 15–100 yield 0 rows │
├───────────┼──────────────────┼────────────┼───────────────────────────┼────────────────────────────────┤
│ 3,000     │ 100              │ 52 / 100   │ Iteration 49 (206 rows)   │ Iterations 50–100 yield 0 rows │
├───────────┼──────────────────┼────────────┼───────────────────────────┼────────────────────────────────┤
│ 1,000     │ 100              │ 0 / 100    │ None (100/100 pass)       │ Below threshold for 100 calls  │
└───────────┴──────────────────┴────────────┴───────────────────────────┴────────────────────────────────┘

3. Mitigation Verification (Release Mode, <TieredPGO>false</TieredPGO>, N=5)

┌───────────┬─────────────────────┬─────────────────────┬─────────────────────────────────────┬────────┐
│ Data Rows │ Expected Total Rows │ Mismatches / 5 Runs │      Run Details (SAX Counts)       │ Status │
├───────────┼─────────────────────┼─────────────────────┼─────────────────────────────────────┼────────┤
│ 1,000,000 │ 1,000,001           │ 0 / 5               │ Iter 1–5: 1,000,001 (all match DOM) │ PASS   │
├───────────┼─────────────────────┼─────────────────────┼─────────────────────────────────────┼────────┤
│ 40,000    │ 40,001              │ 0 / 5               │ Iter 1–5: 40,001 (all match DOM)    │ PASS   │
├───────────┼─────────────────────┼─────────────────────┼─────────────────────────────────────┼────────┤
│ 35,000    │ 35,001              │ 0 / 5               │ Iter 1–5: 35,001 (all match DOM)    │ PASS   │
└───────────┴─────────────────────┴─────────────────────┴─────────────────────────────────────┴────────┘

Steps to Reproduce

1. Clone or save the project files (dotnet10-tieredpgo-openxml-repro.csproj and Program.cs).
2. Build in Release configuration:
dotnet build -c Release
3. Run the executable with 40,000 rows and 5 iterations:
./bin/Release/net10.0/dotnet10-tieredpgo-openxml-repro.exe 40000 5
4. Observe output: SAX count is corrupted on iteration 1 and drops to 0 on iterations 2–5.
5. Add <TieredPGO>false</TieredPGO> to .csproj, rebuild, and rerun to confirm all 5 iterations pass.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[EgorBo]

Seems fixed in .NET 11.0, probably the fix is not yet backported to 10.0 yet.

[EgorBo]

Actually, looks like it was accidentally fixed by #123856 and wasn't backported