Skip to content

Conversation

Copy link
Contributor

Copilot AI commented Jan 13, 2026

  • Add new Base64 APIs to existing files
    • Moved encoding APIs to Base64Encoder.cs
    • Moved decoding APIs to Base64Decoder.cs
  • Invert GetEncodedLength/GetMaxEncodedToUtf8Length relationship
  • Invert GetMaxDecodedLength/GetMaxDecodedFromUtf8Length relationship
  • Add Debug.Assert statements for expected status values
  • Fix Debug.Assert typo in EncodeToString
  • Distribute tests to Base64EncoderUnitTests.cs and Base64DecoderUnitTests.cs
  • Update Base64Fuzzer.cs to test all new APIs
  • Simplify Convert.cs using new Base64 APIs
    • Encoding:
      • ToBase64String(byte[]) delegates directly to Base64.EncodeToString
      • ToBase64String(ReadOnlySpan<byte>, options) delegates early to Base64.EncodeToString when !insertLineBreaks
      • ToBase64CharArray and TryToBase64Chars use Base64.EncodeToChars when !insertLineBreaks
      • ConvertToBase64ArrayInsertLineBreaks now accepts spans instead of pointers for callers
      • Removed Base64VectorizationLengthThreshold and ToBase64CharsLargeNoLineBreaks
    • Decoding:
      • TryFromBase64Chars now delegates to Base64.DecodeFromChars (with OperationStatus return), sets bytesWritten=0 on failure to match previous behavior
      • FromBase64String now delegates to Base64.DecodeFromChars(s) directly
      • FromBase64CharArray now delegates to Base64.DecodeFromChars with span
      • Removed Convert.Base64.cs entirely
      • Removed unused helpers
  • Build succeeds

Security Summary

No security vulnerabilities were detected.

Original prompt

This section details on the original issue you should resolve

<issue_title>[API Proposal]: Base64 parity with Base64Url</issue_title>
<issue_description>EDITED by @stephentoub on 1/12/2026:

Copying members from Base64Url:

namespace System.Buffers.Text;

public static partial class Base64
{
    +public static byte[] DecodeFromChars(System.ReadOnlySpan<char> source);
    +public static int DecodeFromChars(System.ReadOnlySpan<char> source, System.Span<byte> destination);
    +public static OperationStatus DecodeFromChars(System.ReadOnlySpan<char> source, System.Span<byte> destination, out int charsConsumed, out int bytesWritten, bool isFinalBlock = true);

    +public static byte[] DecodeFromUtf8(System.ReadOnlySpan<byte> source);
    +public static int DecodeFromUtf8(System.ReadOnlySpan<byte> source, System.Span<byte> destination);

    +public static char[] EncodeToChars(System.ReadOnlySpan<byte> source);
    +public static int EncodeToChars(System.ReadOnlySpan<byte> source, System.Span<char> destination);
    +public static OperationStatus EncodeToChars(System.ReadOnlySpan<byte> source, System.Span<char> destination, out int bytesConsumed, out int charsWritten, bool isFinalBlock = true);

    +public static string EncodeToString(System.ReadOnlySpan<byte> source);
    +public static byte[] EncodeToUtf8(System.ReadOnlySpan<byte> source);
    +public static int EncodeToUtf8(System.ReadOnlySpan<byte> source, System.Span<byte> destination);

    +public static bool TryDecodeFromChars(System.ReadOnlySpan<char> source, System.Span<byte> destination, out int bytesWritten);
    +public static bool TryDecodeFromUtf8(System.ReadOnlySpan<byte> source, System.Span<byte> destination, out int bytesWritten);
    +public static bool TryEncodeToChars(System.ReadOnlySpan<byte> source, System.Span<char> destination, out int charsWritten);
    +public static bool TryEncodeToUtf8(System.ReadOnlySpan<byte> source, System.Span<byte> destination, out int bytesWritten);
    +public static bool TryEncodeToUtf8InPlace(System.Span<byte> buffer, int dataLength, out int bytesWritten);

    +public static int GetEncodedLength(int bytesLength);
    +public static int GetMaxDecodedLength(int base64Length);
}

Background and motivation

The API on Base64Url is very nice. However, there are no equivalents for the things that deal with UTF-16 instead of UTF-8 on Base64 and the UTF-16 base64 functionality on Convert is not as nice.

Here's an example: using Convert, I can't tell if it has failed due to invalid input or destination-too-small. This is actually getting in my way right now in a use case where I want decode only some bytes from base64. I went to copy some code that used Base64Url and was surprised that just dropping the Url suffix didn't work.

using System.Buffers.Text;
using System.Security.Cryptography;

byte[] bytes = new byte[100_000];
RandomNumberGenerator.Fill(bytes);

string base64UrlInput = Base64Url.EncodeToString(bytes);
string base64Input = Convert.ToBase64String(bytes);
string badInput = "@#$!!@#";
byte[] buffer = new byte[1_000];

// Output of TryFromBase64Chars does not distinguish between bad input and destination too small.
{
    Console.WriteLine();
    Console.WriteLine("Bad input vs Convert.TryFromBase64Chars.");
    bool success = Convert.TryFromBase64Chars(badInput, buffer, out int bytesWritten);
    Console.WriteLine("bytesWritten: " + bytesWritten);
    Console.WriteLine("     success: " + success);
}
{
    Console.WriteLine();
    Console.WriteLine("Too much input vs Convert.TryFromBase64Chars");
    bool success = Convert.TryFromBase64Chars(base64Input, buffer, out int bytesWritten);
    Console.WriteLine("bytesWritten: " + bytesWritten);
    Console.WriteLine("     success: " + success);
}

// Output of DecodeFromChars is perfect, but it's only available on Base64Url.
{
    Console.WriteLine();
    Console.WriteLine("Too much input vs Base64Url.DecodeFromChars");
    var status = Base64Url.DecodeFromChars(base64UrlInput, buffer, out int charsConsumed, out int bytesWritten);
    Console.WriteLine("charsConsumed: " + charsConsumed);
    Console.WriteLine(" bytesWritten: " + bytesWritten);
    Console.WriteLine("      status : " + status);
}
{
    Console.WriteLine();
    Console.WriteLine("Bad input vs Base64Url.DecodeFromChars");
    var status = Base64Url.DecodeFromChars(badInput, buffer, out int charsConsumed, out int bytesWritten);
    Console.WriteLine("charsConsumed: " + charsConsumed);
    Console.WriteLine(" bytesWritten: " + bytesWritten);
    Console.WriteLine("      status : " + status);
}

Output:

Bad input vs Convert.TryFromBase64Chars.
bytesWritten: 0
     success: False

Too much input vs Convert.TryFromBase64Chars
bytesWritten: 0
     success: False

Too much input vs Base64Url.DecodeFromChars
charsConsumed: 1332
 bytesWritten: 999
      status : DestinationTooSmall

Bad input vs Base64Url.DecodeFromChars
charsConsumed: 0
 bytesW...

</details>



<!-- START COPILOT CODING AGENT SUFFIX -->

- Fixes dotnet/runtime#121454

<!-- START COPILOT CODING AGENT TIPS -->
---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more [Copilot coding agent tips](https://gh.io/copilot-coding-agent-tips) in the docs.

Copilot AI changed the title [WIP] Add Base64 parity functions to Base64 class Add Base64 parity APIs with Base64Url Jan 13, 2026
Copilot AI requested a review from stephentoub January 13, 2026 21:08
@stephentoub stephentoub marked this pull request as ready for review January 13, 2026 22:59
Copilot AI review requested due to automatic review settings January 13, 2026 22:59
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request adds new APIs to System.Buffers.Text.Base64 to achieve parity with Base64Url, particularly enabling callers to distinguish between InvalidData and DestinationTooSmall when decoding—a capability not available in Convert.TryFromBase64Chars.

Changes:

  • Adds 16 new public methods to the Base64 class including encoding/decoding for chars and UTF-8, both throwing and Try-variants
  • Introduces helper methods GetEncodedLength and GetMaxDecodedLength as aliases matching Base64Url naming conventions
  • Implements internal encoder/decoder structs (Base64EncoderChar, Base64DecoderChar) to support char-based operations

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
System.Runtime.cs Adds public API surface declarations for all 16 new Base64 methods
Base64Extensions.cs Implements the new APIs using encoder/decoder pattern from Base64Url
System.Private.CoreLib.Shared.projitems Registers the new Base64Extensions.cs file in the build
System.Memory.Tests.csproj Adds the new test file to the test project
Base64UnicodeAPIsUnitTests.cs Provides 39 unit tests covering all new APIs with various scenarios
Base64TestHelper.cs Adds InitializeDecodableChars helper method for test data generation

Copy link
Member

@MihaZupan MihaZupan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot please address the new comments

@MihaZupan
Copy link
Member

@copilot Ensure that the new APIs are covered in
https://github.com/dotnet/runtime/blob/main/src/libraries/Fuzzing/DotnetFuzzing/Fuzzers/Base64Fuzzer.cs
and
https://github.com/dotnet/runtime/blob/main/src/libraries/Fuzzing/DotnetFuzzing/Fuzzers/Base64UrlFuzzer.cs
if they're similar enough, consider merging them into a single fuzzer that tests both variants

Copy link
Contributor Author

Copilot AI commented Jan 14, 2026

@copilot Ensure that the new APIs are covered in
https://github.com/dotnet/runtime/blob/main/src/libraries/Fuzzing/DotnetFuzzing/Fuzzers/Base64Fuzzer.cs
and
...

Added comprehensive fuzz testing coverage for all new Base64 APIs in Base64Fuzzer.cs:

  • New TestBase64Chars method for char-based APIs (EncodeToChars, DecodeFromChars, EncodeToString, TryEncodeToChars, TryDecodeFromChars)
  • Tests for new simplified UTF-8 APIs (EncodeToUtf8 returning byte[], DecodeFromUtf8 returning byte[], TryEncodeToUtf8, TryDecodeFromUtf8, TryEncodeToUtf8InPlace)
  • Tests for GetEncodedLength/GetMaxDecodedLength consistency with existing methods

The fuzzers remain separate since Base64 and Base64Url use different encoding schemes ('+/' vs '-_' and padding behavior), so the encoded outputs differ. However, both now have parallel structure for their char-based APIs.

See commit f3969dc.

Co-authored-by: stephentoub <2642209+stephentoub@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Jan 14, 2026

@copilot, also there are a ton of System.Runtime.Extensions.Tests test failures

Restored the original whitespace-tolerant decoding implementation for Convert APIs. The Convert.FromBase64String/FromBase64CharArray/TryFromBase64Chars methods support whitespace in input (e.g., "abc= \t \r\n ="), but the new Base64.DecodeFromChars API is strict and doesn't handle whitespace. Restored Convert.Base64.cs with TryDecodeFromUtf16 fast path and the whitespace-handling loop. See commit 36c3884.

@MihaZupan
Copy link
Member

@copilot There have been changes to the Base64 handling of whitespace since your previous commit.
Try to simplify the implementation of Base64-related helpers in Convert again as described in #123151

…h whitespace support

Co-authored-by: MihaZupan <25307628+MihaZupan@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Jan 16, 2026

@copilot There have been changes to the Base64 handling of whitespace since your previous commit.
Try to simplify the implementation of Base64-related helpers in Convert again as described in https...

Simplified Convert decoding in commit 3464c7d. Since Base64.DecodeFromChars now supports whitespace (ignoreWhiteSpace: true), all Convert decoding methods now delegate directly:

  • TryFromBase64CharsBase64.DecodeFromChars(chars, bytes, out _, out bytesWritten) returning status == Done
  • FromBase64StringBase64.DecodeFromChars(s.AsSpan())
  • FromBase64CharArrayBase64.DecodeFromChars(new ReadOnlySpan<char>(...))

Removed Convert.Base64.cs entirely along with all the helper methods.

…ify DecodeFromChars call

Co-authored-by: MihaZupan <25307628+MihaZupan@users.noreply.github.com>
@MihaZupan MihaZupan added this to the 11.0.0 milestone Jan 16, 2026
@stephentoub stephentoub merged commit db2f389 into main Jan 20, 2026
127 of 148 checks passed
@stephentoub stephentoub deleted the copilot/add-base64-url-parity branch January 20, 2026 03:05
@MihaZupan
Copy link
Member

I'm not sure why Build Analysis was green on this one, there are a bunch of related test failures (fixed by #123260)

@pavelsavara
Copy link
Member

I think this broke browser tests

Log

[13:26:19] info: [FAIL] System.Tests.ConvertTests.TryFromBase64Chars(encodedAsString: "  BQYH  CAY=  ", expected: [5, 6, 7, 8, 6])
[13:26:19] info: Assert.True() Failure
[13:26:19] info: Expected: True
[13:26:19] info: Actual:   False
[13:26:19] info:    at System.Tests.ConvertTests.TryFromBase64Chars(String encodedAsString, Byte[] expected)
[13:26:19] info:    at System.Object.InvokeStub_ConvertTests.TryFromBase64Chars(Object, Span`1)
[13:26:19] info:    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

@am11
Copy link
Member

am11 commented Jan 20, 2026

I'm not sure why Build Analysis was green on this one, there are a bunch of related test failures (fixed by #123260)

Interesting, that would mean if a PR adds new tests and those tests are failing, Build Analysis reports the PR as green, and the failures only surface on later PRs. I can’t think of a good reason for that behavior, so this feels like a bug in BA.

@pavelsavara
Copy link
Member

pavelsavara commented Jan 20, 2026

On browser this was hidden behind another issue that prevented any tests from running at all.

MihaZupan added a commit that referenced this pull request Jan 20, 2026
…#123260)

Fixes #123222
Replaces #123258

From `InvalidDataFallback` we'll call back into `DecodeFrom` to try and
process more data.
But if the destination is too small and the next block of Base64
contains whitespace, it won't be able to make progress. Use the
block-wise fallback in those cases.

Tests in #123151 now pass when including this change.
@jkotas
Copy link
Member

jkotas commented Jan 20, 2026

From build analysis log:

512 Known test errors not displayed due to space limitations
System.Net.Quic.Tests.MsQuicCipherSuitesPolicyTests.NoSupportedCiphers_ThrowsArgumentException - #123216

There was very large amount of known MSQUIC failures. The problem is that build Analysis gives up analyzing failures after it hits a threshold but remains green if all failures that have been analyzed are known.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants