Utf8Codec doesn't throw FormatException for single and paired UTF-16 surrogates

[sgrekhov]

Please see https://api.dartlang.org/stable/1.22.0/dart-convert/Utf8Codec/Utf8Codec.html

const Utf8Codec({
bool allowMalformed: false
})
Instantiates a new Utf8Codec.

The optional allowMalformed argument defines how decoder (and decode) deal with invalid or unterminated character sequences.

If it is true (and not overridden at the method invocation) decode and the decoder replace invalid (or unterminated) octet sequences with the Unicode Replacement character U+FFFD (�). Otherwise they throw a FormatException.

For testing I used malformed examples from http://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt. All is ok but 5.1 Single UTF-16 surrogates and 5.2 Paired UTF-16 surrogates

  Utf8Codec codec = new Utf8Codec(allowMalformed: false);
  // Single UTF-16 surrogates
  print(codec.decode([0xED, 0xA0, 0x80])); // �
  // Paired UTF-16 surrogates
  print(codec.decode([0xED, 0xA0, 0x80, 0xED, 0xB0, 0x80])); // 𐀀

FormatException be thrown here

Please note, that if allowMalformed is true, then Unicode Replacement character U+FFFD should be used

  Utf8Codec codec = new Utf8Codec(allowMalformed: true);
  // Single UTF-16 surrogates
  print(codec.decode([0xED, 0xA0, 0x80]).codeUnits); // prints [55296] which is U+D800, but should be U+FFFD

[lrhn]

We have allowed invalid UTF-8 here because Dart strings can contain unpaired surrogates.
That's inconsistent, and could cause incompatibilities with other decoders.

I think we should transform encoded surrogates to U+FFFD in decoding and unpaired surrogates to U+FFFD in encoding.

[rakudrama]

The cost of the incompatibility with TextDecoder.decode is that using TextDecoder to solve performance issues is unnecessarily complicated and not as efficent as it should be.

See #31370 for more details on the incompatibility.

[whesse]

This causes our test framework to output invalid utf8, in the case of the test log from the test that checks this property: co19/blob/master/LibTest/convert/Utf8Codec/decode_A03_t02.dart
This test log prints out the actual output of the decoder (an unpaired surrogate), and it gets stored to file, uploaded to cloud storage, and downloaded again as that invalid utf8 encoding of a single surrogate (the bytes [0xED, 0xA0, 0x80]). Our programs don't have a problem with this, but other external programs like editors do.

I agree that fixing this would be good.