Problem with Http.RequestString detection of response content encoding

[dam5s]

When using Http.RequestString to download "https://www.lemonde.fr/rss/une.xml", the resulting string has unexpected character encoding issues:

<title>Le parc locatif privé bousculé par lâessor des locations meublées</title>

instead of the output produced when using System.Net.Http directly:

<title>Le parc locatif privé bousculé par l’essor des locations meublées</title>

I created a small sample project to reproduce the issue and compare to System.Net -
https://github.com/dam5s/fsharp-data-requeststring-issue

I'm running it on .NET core 2.2 on Linux.
Let me know if more details are needed, thanks!

[ssimontis]

Hey there dam5s,

I think this is an encoding issue. If you dig into the .NET Core source code (I used JetBrains dotPeek to explore .NET Core App v2.2.3 of System.Net.Http, the class HttpContent defines the default encoding of a response as UTF-8, which is used if a CharSet is not specified in the HTTP ContentType response header. The XML file does not return a CharSet in the header; it is specified in the XML declaration on the first line of the file as UTF-8.

FSharp.Data.Http.fs defines the default encoding of a response as ISO-8859-1 which is basically UTF-8 forced into 1 byte. If the language has > 127 code points, proper output is not returned, which seems to be what you are experiencing. The source code has a [comment to this SO post on default encoding of HTTP POST requests http://stackoverflow.com/questions/708915/detecting-the-character-encoding-of-an-http-post-request/708942#708942 as justification for this, but it appears UTF-8 is used by most browsers now.

In short, I believe that FSharp.Data and .NET Core disagree upon the default encoding to use with responses. I have never tried to help out on an open source project before, I randomly picked your issue to investigate and this is my finding. I guess I need to find a representative of the project and contact them about a possible fix?

EDIT: Link fix

[dam5s]

It depends.

It seems the project is following the RFC to the letter, which then breaks websites/servers assuming that clients should use UTF-8 as the default encoding.

It seems browsers, curl and the System.Net.Http implementation disagree with that stance and do default to UTF-8 instead.

Does any maintainer have any input on that?

[ssimontis]

@ovatsus @tpetricek @colinbull Could you please take a look at this and advise on which behavior this project wishes to implement? Right now, it is following the RFC for POST encoding, but this puts it at odds with System.Net.Http's behavior. I am happy to fix the issue, I just need to know what you think is best for the project. Thanks!

[colinbull]

Ok, so my two cents is that there isn’t really a “correct” way to do this looking at the findings above. To this end I propose one of two things. Add a ‘http.setfallbackencoding’ function to the pipeline and let the user specify the encoding to use when the CharSet header is not specified. Or add an overload to request string that allows you to pass the default encoding as a parameter when CharSet is not specified. In terms of API design I think the latter is probably more in style with this lib.

[ssimontis]

@colinbull Thanks for getting back to us so quickly! I will take a look at this after work tonight.

[colinbull]

No worries, if you need any help just let me know.

…

On Wed, May 15, 2019 at 12:45 PM Scott Simontis ***@***.***> wrote: @colinbull <https://github.com/colinbull> Thanks for getting back to us so quickly! I will take a look at this after work tonight. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1251>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEO6SRSZU2253SCIRXROADPVPZWLANCNFSM4HBXGBSA> .

[suside]

According to RFC 4627 UTF-8 is the default encoding for JSON.
So it looks like HttpEncodings.ResponseDefaultEncoding should be split between:

1. text, RFC2616, ISO-8859-1
2. json, RFC4627, UTF-8

However System.Net.Http.HttpClient is using UTF-8 even for Content-type: text/plain

Workaround for now is to use Http.RequestString(..., responseEncodingOverride = "utf-8")

[github-actions]

🤖 This is an automated response from Repo Assist.

The root cause is that FSharp.Data defaults to ISO-8859-1 (Latin-1) when no charset is specified in the Content-Type response header, following RFC 2616 §3.7.1. Modern servers, browsers, and System.Net.Http.HttpClient all default to UTF-8 in practice, so this creates a real incompatibility for UTF-8 content served without an explicit charset.

Workaround (available today): pass responseEncodingOverride = "utf-8" to override the default:

Http.RequestString("(www.lemonde.fr/redacted), responseEncodingOverride = "utf-8")

Longer-term, a proper fix would default to UTF-8 when no charset is specified (matching HttpClient behaviour), or optionally split the default by content-type (UTF-8 for application/json, ISO-8859-1 for text/* per RFC). A PR along these lines would be welcome — the relevant code is HttpEncodings.ResponseDefaultEncoding in Http.fs.

Generated by Repo Assist

To install this workflow, run gh aw add githubnext/agentics/workflows/repo-assist.md@0d6e8cf9db90470cd5477c6a40b350fd9f9e1422. View source at https://github.com/githubnext/agentics/tree/0d6e8cf9db90470cd5477c6a40b350fd9f9e1422/workflows/repo-assist.md.

[dsyme]

Longer-term, a proper fix would default to UTF-8 when no charset is specified (matching HttpClient behaviour), or optionally split the default by content-type (UTF-8 for application/json, ISO-8859-1 for text/* per RFC). A PR along these lines would be welcome — the relevant code is HttpEncodings.ResponseDefaultEncoding in Http.fs.

Agreed we should do the same as HttpClient and do this.

[github-actions]

🤖 This is an automated response from Repo Assist.

I've just submitted a draft PR that implements this fix: branch repo-assist/fix-issue-1251-http-response-default-utf8.

The change is minimal: HttpEncodings.ResponseDefaultEncoding is changed from Encoding.GetEncoding("ISO-8859-1") to Encoding.UTF8, matching System.Net.Http.HttpClient behaviour. Doc comments in all four type providers are updated accordingly.

The responseEncodingOverride parameter remains available for callers who need to force a specific encoding.

All 2846 non-network tests pass.

Generated by Repo Assist

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/repo-assist.md@69c1ee19e39e6aaba35519aafefb5cbf314de323

Warning

⚠️ Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• www.google.com

[dsyme]

I'll cloise this as there is a draft PR, we can decide there on the change