[ci-scan] Test failure: KmacXof128Tests throw "invalid key length" under libraries-pgo

[github-actions]

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1484392
Build error leg or test failing: linux-x64 Release - System.Security.Cryptography.Tests

Error Details

Under PGO instrumentation (runtime-coreclr libraries-pgo, def 145), multiple KmacXof128Tests fail on linux-x64 with OpenSslCryptographicException: error:1C800069:Provider routines::invalid key length. The KMAC key passed to OpenSSL is rejected as having invalid length when the code is PGO-optimized, suggesting PGO may corrupt the key data or length passed to the native interop layer.

All KmacXof128 tests in the suite fail with the same OpenSSL error:

System.Security.Cryptography.Tests.KmacXof128Tests.CryptographicException_Instance_OutputSize [FAIL]
      Interop+Crypto+OpenSslCryptographicException : error:1C800069:Provider routines::invalid key length
      at System.Security.Cryptography.ConcurrentSafeKmac..ctor(String algorithmId, ReadOnlySpan`1 key, ReadOnlySpan`1 customizationString, Boolean xof)
      at System.Security.Cryptography.KmacXof128..ctor(ReadOnlySpan`1 key, ReadOnlySpan`1 customizationString)

Error Message

{
  "ErrorMessage": ["KmacXof128", "invalid key length"],
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Impact on platforms

• runtime-coreclr libraries-pgo (def 145) / linux-x64 Release / Helix: Ubuntu.2204.Amd64.Open

First build it occurred

• Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1474477
• Finished: 2026-06-21T19:12:08Z
• Follow-up build https://dev.azure.com/dnceng-public/public/_build/results?buildId=1484392 (Jun 27) also has this failure. 3 consecutive failures in window.

Duplicate search

• "KmacXof128" label:"Known Build Error" — 0 results
• "KmacXof128" "invalid key length" is:open — 0 results
• "KmacXof128" is:pr is:merged merged:>=2026-06-14 — 0 results
• "UNCHECKED_OBJECTREF" label:"Known Build Error" — 1 result, integrity-filtered AF: *(_UNCHECKED_OBJECTREF *)handle == NULL (HndCreateHandle called by getJitHandleForObject) #117138

No existing KBE matches this signature.

---

Filed by ci-failure-scan (detection only). ci-failure-fix walks open [ci-scan] KBEs and either opens a small fix PR or comments here to loop in owners — it never disables the test.

Note

🔒 Integrity filter blocked 4 items

The following items were blocked because they don't meet the GitHub integrity level.

• #129832 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #129876 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #114222 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #117138 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by CI Outer-Loop Failure Scanner · ● 101.8M · ◷

Report

Build	Repository	Test	Pull Request
1485769	dotnet/runtime	System.Security.Cryptography.Tests.KmacXof256Tests.GetHashAndReset_Minimal_Span	#129975

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	1	1

[github-actions]

Workflow artifact: ci-fix
Artifact kind: handoff
Linked KBE: #129939

Root-cause analysis (read-only hand-off)

Under PGO instrumentation (runtime-coreclr libraries-pgo, AzDO definition 145) every KmacXof128Tests case fails on linux-x64:

Interop+Crypto+OpenSslCryptographicException : error:1C800069:Provider routines::invalid key length
  at System.Security.Cryptography.ConcurrentSafeKmac..ctor(String algorithmId, ReadOnlySpan<byte> key, ReadOnlySpan<byte> customizationString, Boolean xof)
  at System.Security.Cryptography.KmacXof128..ctor(ReadOnlySpan<byte> key, ReadOnlySpan<byte> customizationString)

The same code passes in every non-PGO leg and only fails when PGO-optimized. OpenSSL rejecting the key length means the key span (pointer and/or length) handed across the interop boundary is being corrupted only under the PGO-optimized codegen — i.e. this points at a PGO/codegen defect (e.g. a mis-optimized span/length or a bad instrumentation-vs-optimized disagreement), not at a bug in the cryptography code itself. No product diff is safe to produce here, and a test-side change would only hide the miscompile, so this is a hand-off.

Evidence

• Failing build: 1484392 (leg linux-x64 Release - System.Security.Cryptography.Tests, Helix Ubuntu.2204.Amd64.Open).
• First failing build in window: 1474477 (2026-06-21); 3 consecutive failures in the scanned window.
• Affected pipeline: runtime-coreclr libraries-pgo (def 145) only.

Suggested reviewers / area contacts

• area-CodeGen-coreclr (PGO / dynamic-PGO): @JulieLeeMSFT, @dotnet/jit-contrib
• Cryptography context (to confirm the interop contract is correct): @bartonjs, @vcsjones

Note

This root-cause hand-off was generated by an AI/Copilot agent (ci-failure-fix). No code change was produced because the evidence points to a PGO/codegen miscompile that must not be worked around in the crypto or test code. Please verify before acting.

Note

🔒 Integrity filter blocked 6 items

The following items were blocked because they don't meet the GitHub integrity level.

• #128501 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #104998 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #104750 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #97491 search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• [ci-scan] Test failure: AsyncProfilerTests.StateMachineAsync_CallstackStressWithVaryingDepths under libraries-jitstress #129938 issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• [ci-scan] Test failure: System.Threading.Tests.MutexTests.CrossProcess_NamedMutex_ConcurrentCreateOrOpen #130125 issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by CI Outer-Loop Failure Fixer · ● 9.2M · ◷