The HTTP spec has never been clear about proper header encoding and some clients send UTF-8 or Latin1 data. The proxy needs to be as transparent as possible to such issues, or at least give the developer adequate control over the behavior.

Test scenarios:

• Request - Main issue, the developer rarely controls the client behavior
  • UTF-8 for known headers (Host, Referrer) and custom headers
  • Latin1 for known headers (Host, Referrer) and custom headers
• Response - Less of a problem since the developer has more control over these
  • UTF-8 for known headers (Location) and custom headers
  • Latin1 for known headers (Location) and custom headers
• Servers
  • Kestrel
    • Request: UTF-8. Requires an opt-in setting to get Latin1 ([Kestrel] Support for custom decoder in headers dotnet/aspnetcore#17400)
    • Response: ASCII only.
  • HttpSys - Uses UTF-8
  • IIS In-Proc - Same as HttpSys
  • IIS Out-of-Proc - Known to corrupt non-ascii headers when forwarding requests using WinHttp
• HttpClient
  • Request: Known to disallow sending non-ascii request headers
  • Response:
    • Custom Location decoding
    • Otherwise iso-8859-1, aka Latin1