UTF_16LE.encode does not encode string to UTF-16 LE correctly?

[itn3000]

Environment

rustc --version output:

rustc 1.27.0-nightly (0b72d48f8 2018-04-10)

and my encoding_rs version is 0.7.2.

Steps to reproduce

run the following program

extern crate encoding_rs;

use encoding_rs::UTF_16LE;

fn main() {
    let s = "aa";
    let (bytes, enc, unmappable) = UTF_16LE.encode(s);
    let (dec, enc, unmappable) = UTF_16LE.decode(&bytes);
    for i in dec.chars() {
        println!("{}", i as i32)
    }
    println!("{}", dec);
}

Expected

output following text

97
0
97
0
aa

Actual

output following text(24929 = 0x6161)

24929
慡

[hsivonen]

This is expected documented behavior.

The lack of UTF-16LE/BE encoders is mentioned in the rustdocs (also at the top of the docs for Encoding). The spec shows no encoders for them.

See also this tweet and the follow-up tweet.

[hsivonen]

let (bytes, enc, unmappable) = UTF_16LE.encode(s);

Note that the phenomenon you discovered is precisely why the enc part is in the return tuple.

[itn3000]

Thank you for response.
I got it.

[udoprog]

Got bit by this thinking that encoding_rs would actually abide by the configuration it was given. This seems more appropriate for some kind of error in my opinion rather than silently* ignoring the requested encoding.

*: when using different APIs like encode_from_utf8_without_replacement.

Note that I'm using encoding_rs in a non-browser context. So I can't take the position that other encodings just shouldn't be used. I do however respect that as the stance of this project!

[hsivonen]

Yes, the behavior arising the from the "output encoding" concept in the Encoding Standard is probably surprising (but documented!) when applied to situations other that URL parsing and HTML form submission. Still, as documented, this while this crate has non-Web secondary uses, the design follows the Encoding Standard to the point that secondary things like mappings from Windows code page numbers are outside this crate.

As noted earlier, for non-Web uses that for legacy reasons require UTF-16LE or UTF-16BE output to be generated to conform to format that requires either UTF-16LE or UTF-16BE (so that using UTF-8 doesn't work), there seems to be a need in the Rust ecosystem for a crate for that purpose. However, this crate is not that crate.

Notably, such functionality is unlikely to need to fit into the Encoding objects from this crate. Situations that accept the run-time dynamism of some &'static Encoding usually mean you can choose to use encoding_rs::UTF_8. Situations where only either UTF-16LE or UTF-16BE is required tend to be statically knows, so it would be OK to have a UTF-16LE/BE encoder crate that does not integrate with encoding_rs::Encoding.

[udoprog]

Right. The issue I raised is that certain APIs don't communicate output encoding AFAIU, like encode_from_utf8_without_replacement. And it would be less of a footgun if they did somehow.

Besides that, std and byteorder combined has everything you need to actually perform the encoding:

(Taken from my project)

fn encode_utf16<B>(buf: &mut Vec<u8>, s: &str)
where
    B: byteorder::ByteOrder,
{
    for c in s.encode_utf16() {
        buf.extend(std::iter::repeat(0x0).take(2));
        let s = buf.len() - 2;
        B::write_u16(&mut buf[s..], c);
    }
}