surrogate Unicode code points in input stream

[jmdyck]

In tokenizer/unicodeCharsProblematic.test, the first 4 tests involve input streams containing U+DFFF or U+D800. In each case, the expected output includes:
(1) a parse error, and
(2) a character token in which the code point in question has been replaced by U+FFFD.

Looking at various versions of the HTML5 spec, I can't find justifications for the parse error or the replacement.

The closest I found was the paragraph:

Otherwise, if the number is in the range 0xD800 to 0xDFFF
or is greater than 0x10FFFF, then this is a parse error.
Return a U+FFFD REPLACEMENT CHARACTER character token.

This would appear to be precisely the justification, except that it occurs in the "Tokenizing character references" section, and "the number" is the interpreted value of the numeral in a character reference. (E.g., the paragraph applies to an input stream containing the 8 characters & # x D 8 0 0 ; , not to an input stream containing the single code point U+D800.)

[gsnedders]

The spec certainly used to say this is the now-gone "Pre-processing the input stream" section (this also handled stuff like U+0000 and permanent noncharacters). I believe this is now meant to be handled at the encoding level (no encoding can represent the surrogates in a valid encoding), so the parser places no such constraints on the input stream.

[jmdyck]

The spec certainly used to say this is the now-gone "Pre-processing the input stream" section (this also handled stuff like U+0000 and permanent noncharacters).

That section still exists (e.g., 12.2.2.5 in the living spec).

I believe this is now meant to be handled at the encoding level (no encoding can represent the surrogates in a valid encoding),

I'm doubtful. The "input byte stream" section says (twice) that the tokenizer's input stream consists of Unicode code points ("Overview of the parsing model" concurs), and the spec's definition of 'Unicode code point' explicitly includes "an isolated surrogate code point".

[gsnedders]

http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2013-September/040705.html

I believe this is a bug from over-aiming in the encoding changes. Surrogates can only get into the input stream from script — otherwise they are changed in the encoding layer. Anything that states otherwise is a bug.

[jmdyck]

I believe this is a bug from over-aiming in the encoding changes.

I don't know what that means.

Surrogates can only get into the input stream from script — otherwise they are changed in the encoding layer. Anything that states otherwise is a bug.

Even if that's true, input streams from script are still something that the tokenizer has to deal with, so isolated surrogates would still seem to be test-worthy.

[gsnedders]

I don't know what that means.

Trying to change the spec to simplify it, and over-simplifying it to the point of introducing normative changes.

Even if that's true, input streams from script are still something that the tokenizer has to deal with, so isolated surrogates would still seem to be test-worthy.

Oh, I don't disagree. But I'm not convinced this was a deliberate change in the spec, thus my above email.

[jmdyck]

Okay, so I think you're saying that the tests in question should stay in the test suite, but the decision to change their expected results (or not) should wait for a decision on whether the HTML spec needs to be changed on this point.

[gsnedders]

Yes, indeed. I'll always argue against removing something from the testsuite (unless it is an exact duplicate!).

[gsnedders]

http://html5.org/r/8182 makes the tests valid again.

[jmdyck]

http://html5.org/r/8182 makes the tests valid again.

Not entirely.

It does say that isolated surrogates are parse errors, and so the ParseErrors in the tests' expected output are now justified.

However, it doesn't say that they're replaced by U+FFFD at any point, which is something those test cases assume.

[jmdyck]

(Github won't let me reopen an issue?)

[nolanw]

My reading is the same as @jmdyck and I would happily accept a pull request.

[gsnedders]

(My comment obviously never posted along with the reopening; if it wasn't implied by my reopening it, I agree.)