Fix appending correct number of Unicode replacement characters

[pali]

When truncated UTF-8 sequence was followed by another truncated UTF-8
sequence optionally followed by UTF-8 invariant at the end of string and
first sequence in case it would have been not-truncated is longer as
current string then the whole sequence was replaced just by one Unicode
replacement characters, instead of two (for each invalid/truncated UTF-8
sequence).

This happened also in case Encode was called with STOP_AT_PARTIAL flag.

[dankogai]

Thank you!

[khwilliamson]

Note that this patch, while an improvement over the previous version, is not fully correct. The number of bytes consumed per failure in some instances is too few. For example, when replacing the bad bytes with a replacement character, the string \xc0\x80 should result in just one replacement character consuming both bytes; now it results in two replacement chars.

[chansen]

I have added comprehensive tests for Unicode Best Practices for Using U+FFFD in PR #87.

@khwilliamson, \xc0\x80 should be replaced with two replacement characters <U+FFFD U+FFFD>, \xc0 byte value can never be the first byte of a well-formed UTF-8 sequence.

[khwilliamson]

Actually, the patch may be correct but does not conform to "best practices." These are optional, but not required And the c0 80 emitting two characters is counterintuitive to me as being better than doing one. I'm consulting with Unicode people about why they have chosen this

[khwilliamson]

chansen is technically correct about the Unicode Best Practices, but Perl doesn't do that for overlong sequences, nor, it turns out, do some other UTF-8 decoders, including ICU, which is the closest thing there is to an official Unicode decoder. And a couple of these authors don't think that the Best Practices are correct. So far no one has defended the practices, and, have explicitly declined to defend them, if we don't hear from people by somewhat after the first of the year, I'll submit a formal proposal to modify the practices. In the meantime, Perl isn't going to change, and I don't think Encode need do so either. My patch that I sent to Pali, I believe brings Encode into line with core Perl, but it need not be applied just now. The current Encode version does meed the Standard in that malformed bytes are not ignored.

[chansen]

@khwilliamson, I don't think developers would appreciate yet another Unicode change!

Perl has never provided any routines that replaces ill-formed UTF-8 sequences, only routines for validation. ICU has always been constrained by it's legacy.

Current "Unicode Best Practices for Using U+FFFD" is mandatory by WHATWG's UTF-8 specification for HTML5 (since 2012). It's also implemented by Swift, V8 (Node.js), Java (JDK 7,8,9), Webkit (Safari, Chrome) and of course Perl's Unicode::UTF8.