UTF8Encoding should support encoding/decoding of unpaired surrogates

[tmat]

According to RFC 3629 encoding/decoding unmatched surrogates should be disallowed:

"The definition of UTF-8 prohibits encoding character numbers between U+D800 and U+DFFF, which are reserved for use with the UTF-16 encoding form (as surrogate pairs) and do not directly represent characters."

However, this hasn't been followed by real world encoders/decoders. For example, the ECMA-335 standard encodes string arguments of custom attributes using UTF8 and the compilers allowed unpaired surrogates in the attribute argument. Another example is PDB - the file paths in PDB are stored as UTF8 encoded strings and unpaired surrogates are also allowed. The same for values of local string constants (e.g. const string surrogate = "\ud800").

To avoid breaking changes Roslyn needs to allow unpaired surrogates in the above cases and the MetadataReader should also use a variant of UTF8 encoding that is able to decode them. Currently Roslyn has a custom implementation of UTF8 encoder originating from CCI. In general, it seems that pragmatically a UTF16-UTF8 round-tripping is desirable in certain scenarios and UTF8Encoding should support it.

I propose to add a constructor to UTF8 Encoding that takes a bool allowUnpairedSurrogates (false by default) that can be used by both Roslyn and MetadataReader.

[joshfree]

/cc @KrzysztofCwalina

[KrzysztofCwalina]

cc @krwq

[tarekgh]

@tmat when encoding UTF16 "\ud800" to UTF8 bytes, what you expect to get as bytes if we allow the encoding? I am asking because there could be a workaround you can use but before I propose it I need to understand what you expect.

[tmat]

I expect it to write UTF8 like so (note allowUnpairedSurrogates flag that determines whether unpaired surrogates are encoded into the result or replaced with invalid character byte sequence):
http://source.roslyn.io/#Microsoft.CodeAnalysis/PEWriter/BlobUtilities.cs,15d9d2fb5564ddbc,references
and be able to read it and roundtrip the values.

An unpaired surrogate c would be encoded into 3 bytes:

ptr[0] = (byte)(((c >> 12) & 0xF) | 0xE0);
ptr[1] = (byte)(((c >> 6) & 0x3F) | 0x80);
ptr[2] = (byte)((c & 0x3F) | 0x80);

[tarekgh]

You can achieve this result by simple implementation of custom EncoderFallback and EncoderFallbackBuffer

https://msdn.microsoft.com/en-us/library/system.text.encoderfallback(v=vs.110).aspx
https://msdn.microsoft.com/en-us/library/system.text.encoderfallbackbuffer(v=vs.110).aspx

Encoding fallback mechanism is originally introduced to address the scenarios like yours.

[tmat]

Yes, i know. The point is I shouldn't need to write fallbacks, this should be a built-in functionality. Using fallbacks is complex. I'd rather keep the above code as is.

[tarekgh]

why? the default behavior is conforming with the standard. any other behavior is achieved by the fallback mechanism. why we need to add more configuration to the API to let it do that while there is already unified way to do it? also looks this is not main stream case either as we didn't get anyone raising this issue.

[tmat]

Are encoding fallbacks available on CoreCLR? What contracts are they included in?

[tarekgh]

Yes it is supported in System.Text.Encoding 4.0.10.0 contract

[tmat]

Can you point me to an example of how this is supposed to be used?

[tarekgh]

You may look at https://github.com/dotnet/corefx/blob/ceae1d38c93e3f53210ad91884dabc1bd6d30a90/src/System.Text.Encoding.Extensions/tests/Fallback.cs
it has simple example which you can modify to make it work the way you like. let me know if you want me help with that too.

[tmat]

I don't understand what GetNextChar is supposed to do. I need to return 3 bytes for the unpaired surrogate. Where do I return those bytes?

[tarekgh]

first you may not need to implement anything if you just used the replacement fallback. something like

        Encoding utf8 = Encoding.GetEncoding("utf-8", new EncoderReplacementFallback("\uFFFD"), new DecoderReplacementFallback("\uFFFD"));
        byte[] bytes = new byte[10];
        int count = utf8.GetBytes("\ud800", 0, 1, bytes, 0);

to answer your question just in case you still need to implement your fallback, the way fallback work is it'll pass you the character that cannot encode. this done by the call to the method EncoderFallbackBuffer.Fallback methods. if your Fallback methods implementation return true, this means you can handle the fallback for such character. this will cause EncoderFallbackBuffer.GetNextChar get called and in this method you need to return the char and not bytes. in your case you'll need to return '\uFFFD' to match your original implementation. GetNextChar will keep get called again till you return 0 that is because you can fallback using multiple characters and not only one character.
please let me know if there is anything unclear here.