Add new conformance tests for headers and redirection

[xbhatnag]

Motivation

Improve conformance coverage for uncommon header values and infinite HTTP redirection taking inspiration from existing tests and HTTP RFC specs.

Modifications

Added 5 tests:

• Empty Header Values: Clients must allow headers with empty values. (RFC 9112 §5)
• Infinite Redirection: A client must not get stuck in a redirection loop and must instead throw an error after a maximum number of redirect follow attempts.
• Head-with-Content-Length: A client that makes a HEAD request should mark the request as completed upon receiving the headers, regardless of fields like Content-Length (RFC 9112 §6.3, rule 1)
• Client + Server multi-value headers: Clients must be able to send/receive headers with the same key and multiple values.

Result

These tests pass with the URLSession implementation of HTTPClient.

Test Plan

Manually verified some of this behavior with curl.

[guoye-zhang]

The invalid header value is actually stripped by HTTPRequest, not by URLSession.

One test we could do is whether multi-value fields are preserved, but URLSession won't pass it today.

[xbhatnag]

One test we could do is whether multi-value fields are preserved, but URLSession won't pass it today.

PTAL at the new multi-value tests, URLSession seems to pass those fine?

[xbhatnag]

One test we could do is whether multi-value fields are preserved, but URLSession won't pass it today.

PTAL at the new multi-value tests, URLSession seems to pass those fine?

Given our conversation above, merging multi-value fields is an optional feature that our tests allow, because some clients may choose to do so.

Later on when we write tests for cookies, we can also ensure that this merging does not apply to Set-Cookie.