Consider adding TextEncoder.containsLoneSurrogates() static

[annevk]

This came up in in the IDL for Wasm discussion at WebAssembly/interface-types#21 (comment) (bit before and after is also relevant context) and @Pauan suggested it might be faster than rustwasm/wasm-bindgen#1348 (comment).

cc @alexcrichton @lukewagner

[RReverser]

I was thinking of a different suggestion instead: adding fatal option to TextEncoder.

Currently it feels incosistent that TextDecoder has a mode to strictly check UTF-8 instead of producing replacement characters, but TextEncoder doesn't and produces them unconditionally, breaking safe roundtrip.

[annevk]

It's a little different though. The contract of an encoding is to provide a mapping between a byte sequence and a scalar value sequence. In particular this excludes a code point sequence.

From that perspective it makes more sense to defer that concern to a higher layer. If we don't want to do that anymore, we'd also have to support streaming I think, in case your input ends with a lone surrogate, but a subsequent call might provide the other lone surrogate that together form a scalar value.

[RReverser]

If we don't want to do that anymore, we'd also have to support streaming I think, in case your input ends with a lone surrogate, but a subsequent call might provide the other lone surrogate that together form a scalar value.

We already don't support streaming [well], because currently this will result in two replacement characters. It seems better to throw explicitly on such cases, and, as you said, defer handling of streaming to the upper layer.

[annevk]

I'm trying to say that if we want to support code point sequence as an input, and add fatal handling for lone surrogates, we also need to handle streaming them for consistency. Or we defer both problems to a higher level as the current design tries to.

[RReverser]

I'm trying to say that if we want to support code point sequence as an input

Ah I think now I understand what you're trying to say - that right now TextEncoder is not even intended to support UTF-16 code points at all and that conversion to replacement chars is actually a side effect and not intended behaviour of the encoder itself?

[annevk]

Correct, it happens at the IDL layer due to USVString.

[RReverser]

I see. I think I'd be still in favour of handling them in TextEncoder too, given how often it's going to be used with JS strings, even if it means adding support for streaming as well.

Otherwise with something like containsLoneSurrogates one would have to go through the string twice, which might have undesirable performance effects on large strings.

[hsivonen]

Firefox has SIMD-accelerated code for checking if a UTF-16 string is valid UTF-16. I guess it makes sense to expose it to the Web. It's less clear that TextEncoder is the right entry point as opposed to JS strings themselves. I can see how it would be more expedient to standardize a method at the WHATWG than at TC39, but it seems to me it ideally should be a JS string method.

Firefox currently doesn't have a mechanism to signal errors for unpaired surrogates in UTF-16 to UTF-8 conversion, because previously there has not been a use case for such detection. So implementing that approach would involve writing some new (non-glue) code.

I'd like to see if the event issue can be fixed quickly in browsers.

[hsivonen]

I'd like to see if the event issue can be fixed quickly in browsers.

Also, for the event issue, both checking the whole string for lone surrogates and making UTF-16 to UTF-8 conversion report unpaired surrogates are overkills: It's enough to check if the last code unit of the JS string is an unpaired surrogate.

[RReverser]

It's less clear that TextEncoder is the right entry point as opposed to JS strings themselves.

It would be symmetrical to TextDecoder though. Even if JS gains own support for string encoding/decoding in the future, it seems useful to have this level of consistency between existing APIs in WHATWG Encoding.

because previously there has not been a use case for such detection

Sure, but I think that this usecase (asking for only valid UTF-8 output) is common enough to warrant an implementation if/when API is agreed on. Firefox can use the existing SIMD-based check implementation internally, but I don't think that the actual shape of public-facing APIs should be driven by existing internal implementations...

[hsivonen]

Sure, but I think that this usecase (asking for only valid UTF-8 output) is common enough

s/valid UTF-8/lossless UTF-8/. I don't think it has been demonstrated that asking for losslessness is going to be a common use case. So far, experience with the Servo style engine in particular suggests that the Web, despite Hyrum's Law generally ruling everything, is remarkably free of relying on unpaired surrogates.

I don't think that the actual shape of public-facing APIs should be driven by existing internal implementations...

It shouldn't. I'm just mentioning the implementation distance for particular approaches.

[hsivonen]

Also, regarding whether this is going to be common, I think it's a notable data point that Firefox doesn't need error-signaling UTF-16 to UTF-8 conversion internally. (It does have error-signaling potentially-invalid UTF-16 to potentially-invalid UTF-8 comparison operation, but maybe it could get away with not having even that one.)

[RReverser]

I don't think it has been demonstrated that asking for losslessness is going to be a common use case.

I'm somewhat worried that the reason is that exactly that alternative is not exposed in the API and developers are not aware that it's currently lossy by default. (I personally wasn't aware of this until running into this issue.)

[Pauan]

@RReverser I was thinking of a different suggestion instead: adding fatal option to TextEncoder.

I think that would be a good thing to have, but that's orthogonal to this, because a fatal option will not help us.

We want to completely ignore strings which contain unpaired surrogates, so we definitely don't want runtime exceptions!

Otherwise with something like containsLoneSurrogates one would have to go through the string twice, which might have undesirable performance effects on large strings.

Indeed, I agree that is very unfortunate.

But for our use case we still wouldn't want fatal, instead we would want something else, such as an API which returns the encoded Uint8Array (if the string is valid) or null (if the string contains unpaired surrogates). That would avoid the double iteration.

---

@hsivonen So far, experience with the Servo style engine in particular suggests that the Web, despite Hyrum's Law generally ruling everything, is remarkably free of relying on unpaired surrogates.

Web code may not rely upon unpaired surrogates per se, but they definitely rely on the ability for invalid JS strings to roundtrip correctly (e.g. the double input event bug).

Unfortunately when encoding to UTF-8 you lose the ability to roundtrip.

[RReverser]

But for our use case we still wouldn't want fatal, instead we would want something else, such as an API which returns the encoded Uint8Array (if the string is valid) or null (if the string contains unpaired surrogates). That would avoid the double iteration.

@Pauan Hmm, that doesn't sound really orthogonal to my fatal suggestion. What you're describing sounds more like a simple error handling of the result:

let result;
try {
  result = strictEncoder.encode(someString);
} catch {
  result = null;
}
// here `result` is what you're describing, still without separate API or going through the string twice