utf-8 encoding after yaml.load_file

[HenricoWitvliet]

I've got a file with utf-8 characters. yaml.load_file loads the character strings correctly. But the encoding, as given by Encoding(), returns unknown. Now I use Encoding(...) <-'UTF-8' to set the encoding.
It would be nice if the character strings had the utf-8 encoding bit set.

[RinatMenyashev]

same problem

[viking]

This same behavior occurs when using R core functions like readLines, at least in Linux. As far as I know, R does not do any kind of encoding detection. If you run example(Encoding), what is your output?

[HenricoWitvliet]

Since a yaml file is encoded in unicode, I would expect strings to be given this encoding. The character string that yaml.load_file returns in my example is utf-8 encoded. But I haven't tried an example yaml in utf-16, so I don't know if setting a bit in every string would be enough.

[viking]

Ah, I see. I didn't realize that all YAML documents are unicode, but the YAML specification agrees with you. The specification says that by default, the encoding is UTF-8. For UTF-16, the document must provide a byte-order mark:
http://yaml.org/spec/1.1/#id868742

It looks like LibYAML has an encoding property:
http://pyyaml.org/wiki/LibYAML#StylisticEventAttributes

I'll add this into the next update.

[viking]

As it turns out, R does not support UTF-16 at all in Encoding() as of version 3.0.2.

[yihui]

We just ran into the same problem. It will be nice if you can explicitly mark the encoding of character strings as UTF-8. Thanks! (We probably do not need to worry about UTF-16)

[viking]

I had forgotten about this issue, unfortunately. I will take a fresh look at it.

[yihui]

Thanks! FWIW, this is our current workaround: rstudio/rmarkdown#421 (Recursively mark the character elements of yaml.load() output as UTF-8)

[ofurkusi]

There seem to be two issues here, one with yaml.load_file and another with yaml.load.

When yaml.load_file calls readLines without explicitly defining the encoding as UTF-8, the contents of a valid UTF-8 encoded yaml file is read into a string with the encoding set to unknown (while in fact being UTF-8). On Windows, R treats the string as latin1 (I guess) so the characters are all garbled when displayed. By adding encoding="UTF-8 as a parameter to readLines the raw text input is read correctly and set as UTF-8 before being passed on to yaml.load.

While I suggest setting encoding="UTF-8 parameter for readLines in yaml.load_file it does not seem to be enough to fix the problem. Once yaml.load starts processing the text read by readLines, it messes the characters up again by reverting the encoding to unknown.

[yihui]

We were bitten by this issue again: rstudio/bookdown#142 Is there a chance that you could fix it? The fix should be fairly simple (mark the input and output strings as UTF-8), and I'm just not familiar with C.