Includes type forwarding for System.Text.Unicode.Utf8 to the Microsoft.Bcl.Memory library #111292

AlexRadch · 2025-01-10T23:17:20Z

This commit introduces the System.Text.Unicode.Utf8 type to the Microsoft.Bcl.Memory library.

It includes type forwarding for Utf8 in Microsoft.Bcl.Memory.Forwards.cs, updates the documentation in PACKAGE.md to include Utf8 functionality, and adds corresponding test cases in Microsoft.Bcl.Memory.Tests.csproj.

The documentation now emphasizes Utf8 alongside Index, Range, and Base64Url, highlighting its role in converting data between UTF-8 and UTF-16 encodings.

This commit introduces the `System.Text.Unicode.Utf8` type to the `Microsoft.Bcl.Memory` library. It includes type forwarding for `Utf8` in `Microsoft.Bcl.Memory.Forwards.cs`, updates the documentation in `PACKAGE.md` to include `Utf8` functionality, and adds corresponding test cases in `Microsoft.Bcl.Memory.Tests.csproj`. The documentation now emphasizes `Utf8` alongside `Index`, `Range`, and `Base64Url`, highlighting its role in converting data between UTF-8 and UTF-16 encodings.

stephentoub · 2025-01-11T00:47:49Z

Thank you, but this is not the intent of that work item, which is to actually ship those Utf8 APIs downlevel, e.g. in addition to type forwarding, it also needs an implementation for netstandard.

Updated `PackageDescription` to include "Utf8" support. Added new `ItemGroup` for conditional compilation of UTF-8 and Unicode handling files for non-net8.0 frameworks. Modified visibility and implementations in `Ascii.Utility.Helpers.cs`, `Utf8.cs`, and `Utf8Utility` based on `MICROSOFT_BCL_MEMORY` define.

AlexRadch · 2025-01-11T12:53:28Z

Thank you, but this is not the intent of that work item, which is to actually ship those Utf8 APIs downlevel, e.g. in addition to type forwarding, it also needs an implementation for netstandard.

I added Utf8 APIs to Microsoft.Bcl.Memory library. I haven't added tests yet. I'll add them too.

Updated `Microsoft.Bcl.Memory.Tests.csproj` to include `UnicodeUtility.cs` and removed .NET 8.0 targeting condition. Modified `Utf8Tests.cs` by adjusting using directives and enhancing the `DecodeHex` method with conditional compilation for .NET 5.0+.

Added a new property `<DefineConstants>$(DefineConstants);MICROSOFT_BCL_MEMORY</DefineConstants>` to the project file to define a new compilation constant for the project.

src/libraries/Microsoft.Bcl.Memory/src/Microsoft.Bcl.Memory.csproj

src/libraries/Microsoft.Bcl.Memory/src/PACKAGE.md

AlexRadch · 2025-01-11T15:39:24Z

Thank you, but this is not the intent of that work item, which is to actually ship those Utf8 APIs downlevel, e.g. in addition to type forwarding, it also needs an implementation for netstandard.

@stephentoub I added Utf8 APIs to Microsoft.Bcl.Memory library for older .NET platforms. I've also added tests for older .NET platforms.

src/libraries/System.Private.CoreLib/src/System/Text/Ascii.Utility.cs

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8.cs

src/libraries/System.Private.CoreLib/src/System/Text/Ascii.Utility.cs

src/libraries/System.Private.CoreLib/src/System/Text/Ascii.Utility.Helpers.cs

src/libraries/System.Runtime/tests/System.Runtime.Tests/System/Text/Unicode/Utf8Tests.cs

src/libraries/System.Private.CoreLib/src/System/Text/Ascii.Utility.cs

src/libraries/Microsoft.Bcl.Memory/src/Microsoft.Bcl.Memory.Forwards.cs

src/libraries/Microsoft.Bcl.Memory/src/Microsoft.Bcl.Memory.csproj

Co-authored-by: Theodore Tsirpanis <teo@tsirpanis.gr>

- Added polyfill for System.Numerics.BitOperations for .NET Standard 2.0.

Removed the `DefineConstants` property from the project file, which included the constant `MICROSOFT_BCL_MEMORY`. This change may impact conditional compilation within the project.

teo-tsirpanis · 2025-01-19T15:21:25Z

(removing NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) as the reason it was added has been addressed)

teo-tsirpanis · 2025-02-23T22:26:20Z

@dotnet/area-system-memory could you review this? Thanks.

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8.cs

For downlevel frameworks we add `Rune.cs` to `Microsoft.Bcl.Memory`. (cherry picked from commit 79ee05d)

(cherry picked from commit bf6f989)

teo-tsirpanis · 2025-03-10T14:05:01Z

CI failures are unrelated.

jkotas · 2025-03-10T22:16:43Z

I do not have any additional comments. @dotnet/area-system-memory should review and signoff.

teo-tsirpanis · 2025-04-12T12:55:00Z

@dotnet/area-system-memory friendly reminder.

src/libraries/Microsoft.Bcl.Memory/src/Microsoft.Bcl.Memory.csproj

src/libraries/System.Private.CoreLib/src/System/Text/Rune.cs

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8.cs

src/libraries/System.Runtime/tests/System.Runtime.Tests/System/Text/Unicode/Utf8Tests.cs

teo-tsirpanis · 2025-04-17T22:37:55Z

PR feedback addressed in teo-tsirpanis@445a232.

(cherry picked from commit 445a232)

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8.cs

teo-tsirpanis · 2025-04-18T20:55:11Z

CI failures are unrelated.

am11 · 2025-04-19T11:32:09Z

src/libraries/Microsoft.Bcl.Memory/src/Polyfills/System.Numerics.BitOperations.netstandard20.cs

+        // C# no-alloc optimization that directly wraps the data section of the dll (similar to string constants)
+        // https://github.com/dotnet/roslyn/pull/24621
+
+        private static ReadOnlySpan<byte> TrailingZeroCountDeBruijn => // 32


nit: this can be a local in TrailingZeroCount (single use). Slightly better on netfx as well.

I can do it in a subsequent PR.

teo-tsirpanis · 2025-05-02T19:19:32Z

Can we merge this?

teo-tsirpanis · 2025-05-04T12:33:02Z

CI failures are unrelated.

stephentoub · 2025-05-07T04:48:42Z

/ba-g failures are known

ghost added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Jan 10, 2025

dotnet-policy-service bot added the community-contribution Indicates that the PR has been added by a community member label Jan 10, 2025

stephentoub added the NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) label Jan 11, 2025

teo-tsirpanis added area-System.Memory and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Jan 11, 2025

build-analysis bot mentioned this pull request Jan 11, 2025

The Operation will be canceled. The next steps may not contain expected logs. dotnet/dnceng#3008

Open

3 tasks

AlexRadch added 2 commits January 11, 2025 21:49

Add compilation constant for Microsoft BCL Memory

22be776

Added a new property `<DefineConstants>$(DefineConstants);MICROSOFT_BCL_MEMORY</DefineConstants>` to the project file to define a new compilation constant for the project.

akoeplinger reviewed Jan 11, 2025

View reviewed changes

src/libraries/Microsoft.Bcl.Memory/src/Microsoft.Bcl.Memory.csproj Outdated Show resolved Hide resolved

src/libraries/Microsoft.Bcl.Memory/src/PACKAGE.md Outdated Show resolved Hide resolved

build-analysis bot mentioned this pull request Jan 11, 2025

restarted. Azure DevOps can't recover from restarts. dotnet/dnceng#3879

Open

3 tasks

teo-tsirpanis reviewed Jan 11, 2025

View reviewed changes

build-analysis bot mentioned this pull request Jan 11, 2025

slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes." dotnet/dnceng#1883

Open

3 tasks

AlexRadch and others added 5 commits January 12, 2025 08:29

Apply suggestions from code review

79214bd

Co-authored-by: Theodore Tsirpanis <teo@tsirpanis.gr>

Update Microsoft.Bcl.Memory for framework compatibility

def40c4

- Added polyfill for System.Numerics.BitOperations for .NET Standard 2.0.

Space

a1caf07

Remove MICROSOFT_BCL_MEMORY from project constants

695e362

Removed the `DefineConstants` property from the project file, which included the constant `MICROSOFT_BCL_MEMORY`. This change may impact conditional compilation within the project.

Refactor Utf8Tests for .NET compatibility improvements

063359a

teo-tsirpanis mentioned this pull request Jan 12, 2025

[API Proposal]: Extend BlobBuilder so consumers can better control allocations #100418

Open

teo-tsirpanis removed the NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) label Jan 19, 2025

jkotas reviewed Feb 23, 2025

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8.cs Outdated Show resolved Hide resolved

Use Rune.DecodeFromUtf8 on all frameworks.

b5992c6

For downlevel frameworks we add `Rune.cs` to `Microsoft.Bcl.Memory`. (cherry picked from commit 79ee05d)

build-analysis bot mentioned this pull request Mar 4, 2025

[wasm][AOT] emcc : error - received SIGKILL (-9) #89402

Open

Fix compile errors.

90a3613

(cherry picked from commit bf6f989)

build-analysis bot mentioned this pull request Mar 10, 2025

Assertion failed in SafeFileHandle.OverlappedValueTaskSource.IOCallback #99517

Open

akoeplinger requested a review from jkotas March 10, 2025 16:17

jkotas requested review from a team and removed request for jkotas March 10, 2025 22:16

stephentoub approved these changes Apr 15, 2025

View reviewed changes

Address PR feedback.

412071a

(cherry picked from commit 445a232)

teo-tsirpanis reviewed Apr 18, 2025

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8.cs Show resolved Hide resolved

am11 reviewed Apr 19, 2025

View reviewed changes

Merge branch 'main' into BclMemoryUtf8

67cb05e

This was referenced May 4, 2025

X509Certificates.Tests.PfxTests.ReadSlhDsa_Pfx_Ietf_NotSupported failed in CI #115280

Closed

SocketsHttpHandler_HttpClientHandler_Http11_Cancellation_Test.GetAsync_CancelDuringResponseHeadersReceived_TaskCanceledQuickly hung in CI #115281

Open

teo-tsirpanis mentioned this pull request May 5, 2025

Add APIs to BlobBuilder for customizing the underlying byte array et al. #115294

Open

build-analysis bot mentioned this pull request May 6, 2025

ReadSlhDsa_Pfx_Ietf_NotSupported: CryptographicException : The specified network password is not correct. #115270

Closed

stephentoub merged commit c0ca2c2 into dotnet:main May 7, 2025
137 of 142 checks passed

eiriktsarpalis mentioned this pull request May 7, 2025

Proposal: Create OOB package for Rune, System.Text.Unicode.Utf8, and related APIs #52947

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Includes type forwarding for System.Text.Unicode.Utf8 to the Microsoft.Bcl.Memory library #111292

Includes type forwarding for System.Text.Unicode.Utf8 to the Microsoft.Bcl.Memory library #111292

AlexRadch commented Jan 10, 2025

stephentoub commented Jan 11, 2025 •

edited

Loading

AlexRadch commented Jan 11, 2025 •

edited

Loading

AlexRadch commented Jan 11, 2025 •

edited

Loading

teo-tsirpanis commented Jan 19, 2025

teo-tsirpanis commented Feb 23, 2025

teo-tsirpanis commented Mar 10, 2025

jkotas commented Mar 10, 2025

teo-tsirpanis commented Apr 12, 2025

teo-tsirpanis commented Apr 17, 2025

teo-tsirpanis commented Apr 18, 2025

am11 Apr 19, 2025

teo-tsirpanis Apr 19, 2025 •

edited

Loading

teo-tsirpanis commented May 2, 2025

teo-tsirpanis commented May 4, 2025

stephentoub commented May 7, 2025

Includes type forwarding for System.Text.Unicode.Utf8 to the Microsoft.Bcl.Memory library #111292

Includes type forwarding for System.Text.Unicode.Utf8 to the Microsoft.Bcl.Memory library #111292

Conversation

AlexRadch commented Jan 10, 2025

stephentoub commented Jan 11, 2025 • edited Loading

AlexRadch commented Jan 11, 2025 • edited Loading

AlexRadch commented Jan 11, 2025 • edited Loading

teo-tsirpanis commented Jan 19, 2025

teo-tsirpanis commented Feb 23, 2025

teo-tsirpanis commented Mar 10, 2025

jkotas commented Mar 10, 2025

teo-tsirpanis commented Apr 12, 2025

teo-tsirpanis commented Apr 17, 2025

teo-tsirpanis commented Apr 18, 2025

am11 Apr 19, 2025

Choose a reason for hiding this comment

teo-tsirpanis Apr 19, 2025 • edited Loading

Choose a reason for hiding this comment

teo-tsirpanis commented May 2, 2025

teo-tsirpanis commented May 4, 2025

stephentoub commented May 7, 2025

stephentoub commented Jan 11, 2025 •

edited

Loading

AlexRadch commented Jan 11, 2025 •

edited

Loading

AlexRadch commented Jan 11, 2025 •

edited

Loading

teo-tsirpanis Apr 19, 2025 •

edited

Loading