CJK chars encoding error with HTML output

[lewisleedev]

HTML has some problems with CJK characters, and the simplest solution is to add in the tag. hledger by default does not include this encoding tag, so CJK characters (Korean, in my case) appear broken when rendered without adding the snippet.

Problem is that this snippet must be placed within the tag and the proper character encoding declaration should be set by the HTML document itself.

I'm still creating an issue for this problem since it affects the proper display of CJK characters in the generated HTML output but I also think that adding HTML meta tags in hledger's output might not be the right approach as a snippet of a document shouldn't really change the encoding for the whole document.

[simonmichael]

Thanks for reporting, I'll wait for a small example of the problem and workaround that you mentioned, since you understand it better.

[lewisleedev]

Sorry for the belated reply.

Example journal file follows:

2024-12-16 미드나잇롤러코스터클럽
    expenses:cafe                             4,500. KRW
    liabilities:samsung card                 -4,500. KRW

2024-12-16 里門蔘鷄湯
    expenses:restaurant                       9,000. KRW
    liabilities:samsung card                 -9,000. KRW

I also tried with the hanja(kanji, Chinese character) input and the same thing happened.

HTML output:

Rendered output, unmodified:

Adding <meta charset="utf8" /> to the output HTML file solves the issue.

Code added:

Rendered output w/ properly rendered CJK char:

[lewisleedev]

Also figured out just now: apparently enabling -t(tree) with HTML output shows this weird indentation character:

Sorry for the censored image, this was my real report with real data

This also was solved by adding the same meta tag.

It was created with the following command on Fedora linux(English locale, if that matters):

hledger is -MATt -O html > index.html

hledger is -MATt -O html -o index.html

This did not solve the issue.

Tried bunch of browsers but to no avail.

[simonmichael]

On my system (macos 15.1), all of the above display properly in safari, brave and firefox.

I probably have a system locale that supports UTF-8 decoding.

Can you tell us more about your OS and system locale/language setting ?

[simonmichael]

Ah, you said: Fedora GNU/Linux. And what does echo $LANG and locale -a look like in the terminal where you run hledger ?

[lewisleedev]

echo $LANG result:

en_US.UTF-8

locale -a result:

<SNIP>
en_US
en_US.iso88591
en_US.iso885915
en_US.utf8
en_ZA
<SNIP>

[lewisleedev]

To be browser specific, I used Zen browser v1.0.1-a17 (Firefox 132.0) and Brave browser v1.73.97 (Chromium 131.0.6778.108), both on the same Fedora machine.

Also tested this with a Windows 10 machine(set to use both English and Korean, display language is English), same with Zen browser and Brave browser and the exact same encoding error happens.

[simonmichael]

You seem to have everything set up correctly for command line use at least. All I can think of is that when you start web browsers from the GUI they are not seeing the same system locale. In that case starting the browser from your terminal might make a difference. But the Windows test makes this seem unlikely.

So I can't reproduce at the moment. Ideas, anyone ??

[simonmichael]

@thielema, have you noticed this in any of your HTML reports ?

[simonmichael]

@lewisleedev, when I trimmed your comment just now I noticed that the $LANG value and the installed locale have a different spelling. I remember that causing problems in my past testing. (Related: https://hledger.org/dev/hledger.html#troubleshooting)

[simonmichael]

(en_US.UTF-8 vs en_US.utf8)

[lewisleedev]

@lewisleedev, when I trimmed your comment just now I noticed that the $LANG value and the installed locale have a different spelling. I remember that causing problems in my past testing. (Related: https://hledger.org/dev/hledger.html#troubleshooting)

That seem to (also) affect terminal output? I have no problem with terminal output(using Wezterm, if it matters). Besides, considering this also happens on my Windows machine, I don't think locale setup is the issue here, at least $LANG part.

Also tested with an Android machine, en-US, Firefox 133.0.3. Same thing happens. w3m works fine so system wide encoding may not be the issue...

It's strange really, perhaps it's only MacOS that's working properly?

[Aankhen]

It works correctly for me in Firefox and Chrome on Windows, too. Nevertheless, I agree with @lewisleedev that hledger should add meta charset: without that, a browser opening the document from the filesystem (where there are no HTTP headers to glean the encoding from) has to guess what the encoding is. The fact that a lot of our browsers arrive at UTF-8 as the answer is fortunate, but it isn’t something hledger should rely on.

[simonmichael]

That does make sense and sounds simple to fix. I’ll do it soon unless someone beats me to it. It must be a default on Mac or something.

[lewisleedev]

where there are no HTTP headers to glean the encoding from

I actually figured it out thanks to that note. All this time I was using python -m http.server to essentially host my .html file when you were talking about opening it directly with the browser. I think I was giving it the wrong header.

[simonmichael]

@lewisleedev great! Does that mean we don't need to do anything in hledger ?

[Aankhen]

I would still suggest adding meta charset, because it is required to remove any ambiguity when the file is being opened directly from disk. It would be fair to say this isn’t a concern since most environments use UTF-8 anyway, but adding it wouldn’t hurt hledger and would deal with the edge cases.

[lewisleedev]

I would have to personally disagree with @Aankhen and say that meta tag in a snippet isn't the best idea. It's a snippet. It cqn either go in a file unaltered or can be inside another HTML page. Either way, user should be able to make their own decisions regarding the encoding without having to remove the meta tag.

I think browsers having uft8 as their default is definitely something that we can rely on. Besides, if for some reason user needs different encoding than utf8, this will be an issue.

[simonmichael]

Well now I'm glad I had all these local IT hassles because I was just about to push the charset UTF-8 meta tag in printHtml.

I see the point that our reports' HTML output is a HTML fragment, not a full HTML document. And "if it ain't broke don't fix it".

@thielema, I'm guessing you'll agree with @lewisleedev here ?

(Unrelated: I didn't find obvious users of Hledger.Write.Html.Blaze.printHtml, should it be removed ?)

[simonmichael]

Example of our HTML output with the meta tag added, just to make this concrete:

$ cat a.html
<meta charset="UTF-8"><link rel="stylesheet" href="hledger.css"><style>
table {border-collapse:collapse}
th, td {padding-left:1em}
th.account, td.account {padding-left:0;}
</style><table><tr><th style="border-bottom:double black">account</th><th style="border-bottom:double black">balance</th></tr><tr><td>ß</td><td align="right" class="amount">10 ß</td></tr><tr><td>проверка</td><td align="right" class="amount">10 проверка</td></tr><tr><td style="border-top:double black"><b>Total:</b></td><td style="border-top:double black" align="right" class="amount coltotal"><b>10 ß, 10 проверка</b></td></tr></table>

[thielema]

On Sat, 28 Dec 2024, Simon Michael wrote: (Unrelated: I didn't find obvious users of Hledger.Write.Html.Blaze.printHtml, should it be removed ?)

I added it for the proposed HTML export in hledger-web, which is based on blaze-html.

[Aankhen]

Either way, user should be able to make their own decisions regarding the encoding without having to remove the meta tag.

I understand what you’re saying, but hledger specifies that journal files are in UTF-8 and can only ever produce UTF-8 (modulo bugs or errors). Putting the HTML output in a non–UTF-8 document verbatim doesn’t make sense, which is why I’d say meta charset is both correct and advisable.

[thielema]

On Sat, 28 Dec 2024, Lewis Lee wrote: I would have to personally disagree with @Aankhen and say that meta tag in a snippet isn't the best idea. It's a snippet. It cqn either go in a file unaltered or can be inside another HTML page. Either way, user should be able to make their own decisions regarding the encoding without having to remove the meta tag.

I also encountered this question when doing FODS export. But since I observed that HTML export actually exports only the table and no HTML header, I did not add charset header to printHtml. Hledger documentation should state: * exported HTML is the bare HTML table, no standalone HTML, no HTML headers, thus no encoding information (btw. pandoc also requires --standalone option for some output formats) * HTML is exported in local encoding (not always UTF-8) (This is why FODS export declares the local encoding as XML encoding.) However, I added HTML view of balance reports experimentally to hledger-web and there I have to generate standalone HTML.

I think browsers having uft8 as their default is definitely something that we can rely on. Besides, if for some reason user needs different encoding than utf8, this will be an issue.

I think default encoding in HTTP was Latin-(1?) and the WWW default cannot be simply changed.