Test failures

[msabramo]

$ tox -e py27
...
Ran 27494 tests in 42.111s

FAILED (failures=415)

Travis CI has been broken since build #245 (commit e8e0046)

[gsnedders]

See #174.

[willharris]

Curious about this, and what the policy is about having the tests failing for months and months (6 months now). Why is this not being addressed, why is your own pull request still open months later?

[gsnedders]

Why is it not being addressed? Time. All of us only have so many hours in a day, and with all of us doing stuff in our spare time, sometimes fall to the way-side.

Ultimately, all the tests that are failing are because of spec changes that the testsuite has been updated to match but html5lib-python hasn't been. If it were down to me alone, I wouldn't have updated the submodule reference (and as such we'd appear to be passing everything). There's undoubtedly merit to being always up to date with expectations stored in our own repo, but I'd wait until all the code was written for that before updating the submodule reference.

At the end of the day, the combination of the HTML parsing algorithm now being widely implemented makes html5lib less interesting as an experimental implementation, and as a generic HTML parsing library it works in the vast majority of cases, so there's just less motivation than there once was.

Is the PR dead? Well, yes. I probably should've commented in there a while ago. Really we need something that not only has expected failures but also a way to keep them up to date; I think we're not far off having that done (but this is the case with many things around here — 99% done, but not quite!).

I know back when everything was in one repo we used to frequently have tests failing — and we just relied on the number of failures remaining constant as a sign we weren't breaking stuff (and yes, I know that doesn't really work), because at the end of the day updating the parser to the spec is in general a lot of tedious work (patches very much welcome, and far easier to do well than rewriting our test harness code to add expected failures and the like!).