Future of this repo

[gsnedders]

@whatwg/html-parser (i.e., @wycats @gsnedders @sideshowbarker @zcorpan @inikulin @hsivonen) + other parser maintainers I'm aware of (@lexborisov, @jdm who seems to review html5ever stuff now, @iabudiab) this repo has been pretty unloved for a while (see also: all the open PRs), so we should probably decide what we're doing with it.

As far as I'm aware, the encoding tests are only used by html5lib, the serializer tests are entirely unused (and in the browser case essentially supplanted by html/syntax/serializing-html-fragments/ in WPT), so outright dropping both of these costs us nothing.

The tree-construction tests could quite easily move to WPT, which would also make it less likely that we continue to grow WPT-only parsing tests.

The tokenizer tests are the hardest ones; as far as I'm aware there's various consumers, but these are tests that cannot be run within the browser setting (as the tokenizer isn't even exposed). With my WPT hat on, I don't really object to them living there, but it's also not obvious they should live there.

[sideshowbarker]

The tokenizer tests are the hardest ones; as far as I'm aware there's various consumers, but these are tests that cannot be run within the browser setting (as the tokenizer isn't even exposed). With my WPT hat on, I don't really object to them living there, but it's also not obvious they should live there.

I think we should move them, because there’s compelling value to moving them (together).

Rationale:

Given that we already have other test suites in WPT that cannot be run with the browser setting, and no WPT policy that prohibits adding more such test suites, at least it’s clear that there would be nothing blocking us from choosing to move the tokenizer tests into WPT along with the tree-construction tests.

Among the good reasons for doing that would be, if we’re going to move the tests at all, we should move the tokenizers tests together with the tree-construction tests — not split them up.

I guess it’s implicit that nobody would want to see the tests split up. So I guess the question comes down to whether we should move them into WPT or continue to keep them here.

I think it’s clear that one big advantage of moving them is that they’re more easily findable by the set of people who already knows about WPT or who are likely to discover WPT — which is a much bigger set than the people who already know about this repo or who are likely to find it.

And getting the tree-construction tests into WPT seems like an obvious big win that strongly argues for moving them out from here and into WPT. And if we want to keep the tests together, then that necessarily means moving the tokenizer tests along with the tree-construction tests.

[gsnedders]

FWIW, I agree with @sideshowbarker here; I was trying to avoid leading anyone to any specific decision in the OP.

On the whole my inclination is to migrate this all over in a few weeks unless anyone objects.

[jgraham]

I don't object.

[nolanw]

I use the encoding tests in my lil toy parser. That's no reason to keep them, just wanted to mention that the work wasn't unused :)

I don't know what WPT is (I'm pretty out of the loop), so a handy link to wherever the tests end up would be awesome.

[sideshowbarker]

Hi @nolanw, WPT is https://github.com/web-platform-tests/wpt/

[nolanw]

@sideshowbarker Thank you!

[Ms2ger]

Moving to wpt SGTM.

[craigbarnes]

If these tests are moved to WPT, does that imply changing the format at all?

[gsnedders]

@craigbarnes no, it doesn't

[hsivonen]

Since the non-browser test harnesses for the Validator.nu HTML Parser use the present input formats and are more sensitive to format changes that, as I understand it, the html5lib harness, I'd prefer to avoid format changes and I'd like to keep the non-scripted tree construction tests clearly separate from the scripted ones.

I'm OK with moving these to the WPT repo if it's up to the WPT harness to consume the tests in their current format for in-browser testing.

[zcorpan]

I'm ok with moving to wpt.

[untitaker]

these are tests that cannot be run within the browser setting (as the tokenizer isn't even exposed)

not to derail the thread too much, but couldn't you "run" them in the browser by:

1. constructing tree/dom from the raw input of the test using browser
2. constructing tree/dom from the expected output tokens of the test using some unrelated, non-browser "reference" tree builder implementation
3. compare the resulting doms