Utf16.IsValid and Utf8/16.IndexOfInvalidSubsequence

[lilinus]

Fixes #118018
Fixes #118113

[Copilot]

Pull Request Overview

This PR adds new UTF-16 validation APIs to System.Text.Unicode, introducing Utf16.IsValid and Utf16.IndexOfInvalidSubsequence methods alongside similar functionality for UTF-8. It also refactors existing code to use these new standardized validation APIs instead of custom validation logic.

• Adds new Utf16 class with validation methods IsValid and IndexOfInvalidSubsequence
• Adds Utf8.IndexOfInvalidSubsequence method to complement existing Utf8.IsValid
• Refactors multiple codebases to use the new standardized validation APIs

Reviewed Changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
System.Private.CoreLib.Shared.projitems	Adds the new Utf16.cs file to the build
Utf16.cs	New class implementing UTF-16 validation methods
Utf8.cs	Adds IndexOfInvalidSubsequence method and updates class documentation
System.Runtime.cs	Adds public API surface for new UTF-16 and UTF-8 validation methods
HttpEncoder.cs	Refactors surrogate validation to use new Utf16.IndexOfInvalidSubsequence
Normalization.Icu.cs	Replaces custom validation logic with Utf16.IsValid
StringSearchValues.cs	Simplifies surrogate validation using Utf16.IsValid
Utf8UtilityTests.ValidateBytes.cs	Adds test coverage for new Utf8.IndexOfInvalidSubsequence method
Utf16UtilityTests.ValidateChars.cs	Adds test coverage for new Utf16 validation methods

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-runtime
See info in area-owners.md if you want to be subscribed.

[jeffhandley]

@lilinus Thanks for your patience on this; we are planning to review it this month. I've updated the branch and CI will re-run, but there was an ApiCompat error in the build previously indicating Utf8.IndexOfInvalidSubsequence wasn't available in the net8.0 target and it wasn't obvious to me why that was the case.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

[jozkee]

Other than the unsafe comment, this LGTM.

Is worth mentioning that

runtime/src/libraries/System.Private.CoreLib/src/System/Globalization/Normalization.Icu.cs

Line 221 in 167d862

private static bool HasInvalidUnicodeSequence(ReadOnlySpan<char> s)

from #118018 can't adopt the new APIs because it would make it more expensive by requiring two passes instead of one.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.