EDIT June 1, 2021 by @stephentoub: This was previously approved, but we'd like to make some tweaks: #35126 (comment)

Proposed API

New NonValidated property returning all headers received by the client without any validation is to be added to HttpHeaders collection.

public abstract class HttpHeaders // existing type
{
    public NonValidatedEnumerator NonValidated { get; } ; // new property

    public struct NonValidatedEnumerator : IEnumerable<KeyValuePair<string, HeaderStringValues>>, IEnumerable, IEnumerator<KeyValuePair<string, HeaderStringValues>>
    {
        public NonValidatedEnumerator GetEnumerator();
        public bool MoveNext();
        public KeyValuePair<string, HeaderStringValues> Current { get; }
        public void Dispose();
        ... // explicitly implemented interface members
    }
}

Usage example

foreach (var h in resp.Headers.NonValidated) { }
foreach (var h in resp.Content.Headers.NonValidated) { }

Original proposal

Today to enumerate the response headers from an HttpClient HTTP request, you can write code like this:

foreach (KeyValuePair<string, IEnumerable<string>> h in resp.Headers) { ... }
foreach (KeyValuePair<string, IEnumerable<string>> h in resp.Content.Headers) { ... }

There are a few issues, here:

1. Validation. TryAddWithoutValidation can be used to add request headers to be sent, and those headers will never be validated. And the handler can similarly use TryAddWithoutValidation to store response headers into the response headers collection. But enumerating the headers in the above manner will force validation of all the headers, which incurs measurable overhead.
2. Allocation. We're forced to allocate enumerables to hand back the response value(s), even if there's only one. And for an unvalidated header, there generally will be only one, because it won't have been validated/parsed.
3. Verbosity. Some developers are frustrated by the need to enumerate two different collections, with the headers split across response.Headers and response.Content.Headers.

We can address (1) and (2) by adding an API like this:

public abstract class HttpHeaders // existing type
{
    public NonValidatedEnumerator EnumerateWithoutValidation(); // new method

    public struct NonValidatedEnumerator : IEnumerable<KeyValuePair<string, HeaderStringValues>>, IEnumerable, IEnumerator<KeyValuePair<string, HeaderStringValues>>
    {
        public NonValidatedEnumerator GetEnumerator();
        public bool MoveNext();
        public KeyValuePair<string, HeaderStringValues> Current { get; }
        public void Dispose();
        ... // explicitly implemented interface members
    }
}

public readonly struct HeaderStringValues : IEnumerable<string>
{
    public Enumerator GetEnumerator();
    public struct Enumerator : IEnumerator<string>
    {
        public bool MoveNext();
        public string Current { get; }
        public void Dispose();
        ... // explicitly implemented interface members
    }
}

As a test, I looked to see what headers my Edge browser is sending when connecting to a particular service on the net, and what response headers I got in response, ~12 headers of varying lengths in each direction, some standard, some non-standard, and I put that into a benchmark that hits a local server. The benchmark makes the request and enumerates all the response and response content headers:

var req = new HttpRequestMessage(HttpMethod.Get, uri);
req.Headers.TryAddWithoutValidation(..., ...);
...

using (HttpResponseMessage resp = await client.SendAsync(req, default))
{
    foreach (var h in resp.Headers) { }
    foreach (var h in resp.Content.Headers) { }
}

With .NET Core 3.1, I get results like this:

We've already made very measurable perf improvements for .NET 5, so running against master I get:

When I then change the test to use an implementation of the above proposal:

foreach (var h in resp.Headers.EnumerateWithoutValidation()) { }
foreach (var h in resp.Content.Headers.EnumerateWithoutValidation()) { }

I get:

Note that very similar numbers are also possible without a new API but with a (small?) behavioral change: we could change HttpHeaders.GetEnumerator to a) not validate and b) to return arrays that are only valid until MoveNext is called again, such that we can reuse the same array for all headers in the enumeration.

Another variation on this would also address the 3rd cited issue around verbosity: we could instead expose such an API on HttpResponseMessage:

public class HttpResponseMessage
{
    public NonValidatedEnumerator EnumerateWithoutValidation();
    ...
}

in which case it internally would enumerate both Headers and Content.Headers. If we did that, to go along with it we could add:

public class HttpRequestMessage
{
    public bool TryAddWithoutValidation(string key, string value);
}

which would determine whether the specified header should go into Headers or Content.Headers.

If we did want to add new API here, there's a question around the HeaderStringValues type above. ASP.NET uses the StringValues type from Microsoft.Extensions.Primitives. A variety of options exist:

1. Just reuse that type, taking a dependency from System.Net.Http.dll onto Microsoft.Extensions.Primitives.dll and having devs that want to use this import the additional Microsoft. namespace.
2. Create a new type in a new System. namespace that's similar in nature, albeit with a few things cleaned up.
3. Do (2), and change ASP.NET APIs to use it instead of the Microsoft.Extensions.Primitives one.

cc: @scalablecory, @davidsh, @Tratcher, @samsp-msft