Open
Conversation
andrewlock
added
area:tracer
andrewlock
force-pushed
the
andrew/add-streaming-base64-decoder
branch
from
e202a95 to
77915fa
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e202a958b6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
BenchmarksBenchmark execution time: 2026-02-23 13:22:21 Comparing candidate commit 3d24011 in PR branch Found 8 performance improvements and 3 performance regressions! Performance is the same for 161 metrics, 20 unstable metrics. scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces net6.0
scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody net6.0
scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs netcoreapp3.1
scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1
scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice netcoreapp3.1
scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool net6.0
scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark netcoreapp3.1
scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog netcoreapp3.1
scenario:Benchmarks.Trace.NLogBenchmark.EnrichedLog netcoreapp3.1
scenario:Benchmarks.Trace.SpanBenchmark.StartFinishScope net6.0
|
andrewlock
force-pushed
the
andrew/add-streaming-base64-decoder
branch
from
77915fa to
aa05f67
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a new Base64DecodingStream helper class that provides memory-efficient base64 decoding by processing the input string on-the-fly without allocating the full decoded byte array. The implementation uses different strategies for different target frameworks: on .NET Core it decodes directly into the caller's buffer using Convert.TryFromBase64Chars, while on .NET Framework it uses a chunked approach with ArrayPool buffers and the vendored Base64.DecodeFromUtf8InPlace method.
Changes:
- Added
Base64DecodingStreamclass for streaming base64 decoding without full array allocation - Implemented framework-specific optimizations (direct decode on NETCOREAPP, chunked decode with ArrayPool on .NET Framework)
- Added comprehensive test suite covering various input sizes, padding scenarios, buffer sizes, and error cases
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| tracer/src/Datadog.Trace/Util/Streams/Base64DecodingStream.cs | Core implementation of the streaming base64 decoder with conditional compilation for different target frameworks |
| tracer/test/Datadog.Trace.Tests/Util/Base64DecodingStreamTests.cs | Comprehensive test suite with 20+ test cases covering correctness, edge cases, padding, invalid input, and async operations |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing This PR (8226) and master. ✅ No regressions detected - check the details below Full Metrics ComparisonFakeDbCommand
HttpMessageHandler
Comparison explanationExecution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard. Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph). Duration chartsFakeDbCommand (.NET Framework 4.8)gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8226) - mean (75ms) : 72, 77
master - mean (75ms) : 70, 80
section Bailout
This PR (8226) - mean (79ms) : 78, 81
master - mean (79ms) : 76, 83
section CallTarget+Inlining+NGEN
This PR (8226) - mean (1,078ms) : 1035, 1121
master - mean (1,078ms) : 1039, 1118
FakeDbCommand (.NET Core 3.1)gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8226) - mean (116ms) : 113, 119
master - mean (116ms) : 113, 119
section Bailout
This PR (8226) - mean (118ms) : 115, 120
master - mean (117ms) : 114, 120
section CallTarget+Inlining+NGEN
This PR (8226) - mean (773ms) : 704, 841
master - mean (763ms) : 698, 827
FakeDbCommand (.NET 6)gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8226) - mean (104ms) : 101, 106
master - mean (103ms) : 100, 105
section Bailout
This PR (8226) - mean (104ms) : 102, 106
master - mean (105ms) : 102, 107
section CallTarget+Inlining+NGEN
This PR (8226) - mean (758ms) : 679, 836
master - mean (757ms) : 701, 813
FakeDbCommand (.NET 8)gantt
title Execution time (ms) FakeDbCommand (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8226) - mean (103ms) : 101, 106
master - mean (102ms) : 99, 105
section Bailout
This PR (8226) - mean (103ms) : 101, 106
master - mean (103ms) : 100, 105
section CallTarget+Inlining+NGEN
This PR (8226) - mean (688ms) : 661, 714
master - mean (667ms) : 653, 680
HttpMessageHandler (.NET Framework 4.8)gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8226) - mean (192ms) : 188, 196
master - mean (192ms) : 187, 196
section Bailout
This PR (8226) - mean (195ms) : 192, 198
master - mean (196ms) : 192, 199
section CallTarget+Inlining+NGEN
This PR (8226) - mean (1,140ms) : 1086, 1195
master - mean (1,139ms) : 1080, 1198
HttpMessageHandler (.NET Core 3.1)gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8226) - mean (276ms) : 272, 279
master - mean (276ms) : 270, 282
section Bailout
This PR (8226) - mean (276ms) : 270, 282
master - mean (276ms) : 272, 280
section CallTarget+Inlining+NGEN
This PR (8226) - mean (936ms) : 884, 988
master - mean (932ms) : 888, 976
HttpMessageHandler (.NET 6)gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8226) - mean (268ms) : 263, 274
master - mean (267ms) : 262, 273
section Bailout
This PR (8226) - mean (268ms) : 264, 272
master - mean (267ms) : 263, 271
section CallTarget+Inlining+NGEN
This PR (8226) - mean (925ms) : 897, 953
master - mean (920ms) : 896, 944
HttpMessageHandler (.NET 8)gantt
title Execution time (ms) HttpMessageHandler (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8226) - mean (267ms) : 261, 273
master - mean (266ms) : 260, 272
section Bailout
This PR (8226) - mean (267ms) : 263, 270
master - mean (267ms) : 263, 271
section CallTarget+Inlining+NGEN
This PR (8226) - mean (827ms) : 811, 843
master - mean (821ms) : 803, 838
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
andrewlock
force-pushed
the
andrew/add-streaming-base64-decoder
branch
from
c331178 to
33c73ce
Compare
andrewlock
force-pushed
the
andrew/add-streaming-base64-decoder
branch
from
33c73ce to
3d24011
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of changes
Adds
Base64DecodingStreamhelper classReason for change
In a few places (particularly Remote Config and Newtonsoft.JSON) we have code like this:
This takes an existing
string, decodes it from base64 to abyte[], then feeds thatbyte[]into aMemoryStream. For big strings, that's a potentially large extra array allocation we can avoid. Instead,Base64DecodingStreamacts effectively as aMemoryStreamover thestring, doing the decode either on the fly (.NET Core), or by using a pooled buffer to decode the input string in chunks.Implementation details
Basically asked 🤖 Claude to do this, then reviewed and tidied up and iterated. The general idea is:
Base64.DecodeFromUtf8InPlacebyte[]Span<T>It didn't seem worth trying to unify these two paths, given the lack of
Convert.TryFromBase64Charsin .NET Framework. We could useBase64.DecodeFromUtf8to write directly to the destination span instead, but we would still need to do the narrowing, and doing that in the destination span is a little risky, as it would mean we write a bunch of bytes which are then leftover junk. It's probably still fine, but I don't know that it's worth the complexity/risk.Warning
Rather than handle the case where you're passed a destination buffer that's <3 bytes, this implementation currently just throws. Otherwise we have to decode into a stackallocated buffer, and hang onto the "overflow" bytes to avoid returning 0 when we're not EOF, which is a bit of a pain. For the places we currently use it, I don't think this will be a problem, but if others disagree, we can handle the edge case too.
Test coverage
Added a variety of unit tests for the implementation
Other details
https://datadoghq.atlassian.net/browse/LANGPLAT-940
This will be part of a Remote config stack, but for now kept it agnostic