IndexedDB test running issues

[brettz9]

I've reported an issue regarding some behaviors of the test runner at w3c/wpt-tools/issues/152 in regard to some IndexedDB tests (running on Chrome), but since it pertains to IndexedDB specifically, I'm not sure whether to mention it here (or on the IndexedDB repo for that matter).

@inexorabletash

[jgraham]

FWIW I couldn't reproduce your problems, although there is some arguably confusing output.

[inexorabletash]

@pwnall - can you take a look at the issue with the exceptions test?

[pwnall]

@brettz9 Can you please tell me how you're running the tests?

Sorry for the cluelessness. I'm using ./serve to bring up a WPT test server, and I got the following screens on Chrome 55 w/o the experimental web platform flag, and on Chrome 58 w/ the flag.

[brettz9]

@pwnall : I also ran it on Chrome 55 without the experimental web platform flag and with python serve (on Windows). While the first screen looks the same as mine with idbcursor-continueprimarykey-exceptions.htm (and that is not a problem), I'm speaking of the aggregate results page (where mine has different results from your second screenshot).

http://web-platform.test:8000/indexeddb//idbcursor-continueprimarykey-exceptions.htm had not been showing on the aggregate page listing in my earlier testing, but as per the screenshot, this particular problem is no longer occurring.

As far as http://web-platform.test:8000/indexeddb//idbcursor-continueprimarykey-exception-order.htm, however, it is, as per the following, correctly counting all 13 tests on the individual page, but note that as per the above screenshot, the aggregate page shows 0/1:

(Also, while that page lists a Harness status of "Error", as I can understand for an uncaught error, the aggregate shows it, as mentioned, within "Timeouts" rather than under "Errors"--I'd expect one or the other, e.g., if a timeout occurs because the test never completed but without producing an uncaught error.)

Regarding @jgraham 's observations in the other issue, I can understand a timeout being reported (though again, I'd expect "Error" instead if there were an uncaught error as reported on the individual page). I'm wondering whether the aggregate page gives up sooner than when simply loading the page independently to completion, and as such does not get the total test count of 13.

As far as @jgraham's other statement on reporting the subtest status as NOTRUN, if the runner is considering the total number of tests as 0, then I would expect it to report something other than "0/1" in the subttest pass rate (maybe 0/0) (but if the aggregate page is really working properly, I'd expect the aggregate page to get the total count of 13 and report 0/13).

I also find it confusing to be reporting "NOTRUN" instead of "Timeout"--if the latter is the known cause of not running (and as it shows as "Timeout" in the totals and in the JSON)--and again, I think "Error" would be even better than "Timeout" if an uncaught error is the known specific reason for the timeout as occurs when visiting the page on its own.

[brettz9]

And not to jumble the issue, but as it is confusing, it also required some digging into the Manifest file for me to see of expectations of lower case paths, despite the fact that the file system indeed stores the files under IndexedDB: