Test failure System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelationBorderCasesInNonBacktracking(options: NonBacktracking)

[VincentBu]

Run: runtime-libraries-coreclr outerloop 20211021.5

Failed test:

net7.0-windows-Release-x64-CoreCLR_release-Windows.81.Amd64.Open

- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelationBorderCasesInNonBacktracking(options: NonBacktracking)

net7.0-windows-Release-x64-CoreCLR_release-Windows.10.Amd64.ServerRS5.Open

- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelation(options: NonBacktracking)
- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelation(options: Compiled)
- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelation(options: None)
- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelationBorderCasesInNonBacktracking(options: NonBacktracking)

net7.0-windows-Release-x64-CoreCLR_release-(Windows.Nano.1809.Amd64.Open)windows.10.amd64.serverrs5.open@mcr.microsoft.com/dotnet-buildtools/prereqs:nanoserver-1809-helix-amd64-08e8e40-20200107182504

- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelation(options: NonBacktracking)
- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelation(options: Compiled)
- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelation(options: None)
- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelationBorderCasesInNonBacktracking(options: NonBacktracking)

net7.0-windows-Release-x86-CoreCLR_release-Windows.10.Amd64.ServerRS5.Open

- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelation(options: NonBacktracking)
- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelation(options: Compiled)
- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelation(options: None)
- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelationBorderCasesInNonBacktracking(options: NonBacktracking)

net7.0-Linux-Release-arm-CoreCLR_release-(Debian.11.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-11-helix-arm32v7-20210304164347-5a7c380

- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelationBorderCasesInNonBacktracking(options: NonBacktracking)

net7.0-OSX-Release-arm64-CoreCLR_release-OSX.1100.ARM64.Open

- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelationBorderCasesInNonBacktracking(options: NonBacktracking)

One of error message:

Assert.NotEqual() Failure
Expected: Not 'ᲀ'
Actual:   'ᲀ'


Stack trace
   at System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelationBorderCasesInNonBacktracking(RegexOptions options) in /_/src/libraries/System.Text.RegularExpressions/tests/RegexCultureTests.cs:line 442

or

Assert.True() Failure
Expected: True
Actual:   False


Stack trace
   at System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelationBorderCasesInNonBacktracking(RegexOptions options) in /_/src/libraries/System.Text.RegularExpressions/tests/RegexCultureTests.cs:line 436

Tagging subscribers to this area: @eerhardt, @dotnet/area-system-text-regularexpressions
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Run: runtime-libraries-coreclr outerloop 20211021.5

Failed test:

net7.0-windows-Release-x64-CoreCLR_release-Windows.81.Amd64.Open

- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelationBorderCasesInNonBacktracking(options: NonBacktracking)

net7.0-windows-Release-x64-CoreCLR_release-Windows.10.Amd64.ServerRS5.Open

- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelation(options: NonBacktracking)
- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelation(options: Compiled)
- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelation(options: None)
- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelationBorderCasesInNonBacktracking(options: NonBacktracking)

net7.0-windows-Release-x64-CoreCLR_release-(Windows.Nano.1809.Amd64.Open)windows.10.amd64.serverrs5.open@mcr.microsoft.com/dotnet-buildtools/prereqs:nanoserver-1809-helix-amd64-08e8e40-20200107182504

- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelation(options: NonBacktracking)
- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelation(options: Compiled)
- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelation(options: None)
- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelationBorderCasesInNonBacktracking(options: NonBacktracking)

net7.0-windows-Release-x86-CoreCLR_release-Windows.10.Amd64.ServerRS5.Open

- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelation(options: NonBacktracking)
- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelation(options: Compiled)
- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelation(options: None)
- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelationBorderCasesInNonBacktracking(options: NonBacktracking)

net7.0-Linux-Release-arm-CoreCLR_release-(Debian.11.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-11-helix-arm32v7-20210304164347-5a7c380

- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelationBorderCasesInNonBacktracking(options: NonBacktracking)

net7.0-OSX-Release-arm64-CoreCLR_release-OSX.1100.ARM64.Open

- System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelationBorderCasesInNonBacktracking(options: NonBacktracking)

One of error message:

Assert.NotEqual() Failure
Expected: Not 'ᲀ'
Actual:   'ᲀ'


Stack trace
   at System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelationBorderCasesInNonBacktracking(RegexOptions options) in /_/src/libraries/System.Text.RegularExpressions/tests/RegexCultureTests.cs:line 442

or

Assert.True() Failure
Expected: True
Actual:   False


Stack trace
   at System.Text.RegularExpressions.Tests.RegexCultureTests.TestIgnoreCaseRelationBorderCasesInNonBacktracking(RegexOptions options) in /_/src/libraries/System.Text.RegularExpressions/tests/RegexCultureTests.cs:line 436

Author:	VincentBu
Assignees:	-
Labels:	arch-arm32, arch-arm64, arch-x86, area-System.Text.RegularExpressions, os-linux, os-mac-os-x, os-windows, arch-x64
Milestone:	-

[stephentoub]

I believe the issue here is different data coming from either NLS or older ICUs. I can repro the same failures by setting the environment variable:
DOTNET_SYSTEM_GLOBALIZATION_USENLS=1
and then running outerloop unit tests.

@veanes, @olsaarik, I think this highlights we'll need to revisit the pre-generation of the relation data (for reasons beyond the size ones we've already discussed). The data can vary based on the machine on which pregeneration was performed, but we ship a single S.T.RegularExpressions.dll binary that's used regardless of OS version and regardless of what globalization library is in use for a given process. For both size and correctness/consistency then, we'll likely need to remove the pregenerated data and come up with a scheme for computing it lazily at run-time.

cc: @danmoseley