Form Controls

[mfreed7]

Description
Form controls are mentioned frequently as a source of developer frustration. While "form controls" is a quite general area, there are several recurring themes (e.g. see the most recent state of css survey results):

• Form element stylability, particularly for <select>, <input type=checkbox>, and <input type=radio>, but also generally for all input types.
• Pseudo-element interop, particularly for <input type=range>.
• Date/time input element support.
• CSS accent-color support.
• Form validation behavior.
• appearance:none/auto (unprefixed).

Specification
Other than accent-color, form element appearance is not (well) specified.

Tests

• Due to the lack of spec, there is also a lack of WPT testing for form element style-related features.
• Accent-color is fairly-well tested, except around the actual appearance with accent-color applied.
• Form submission is somewhat well tested, perhaps excepting validation behavior.
• Appearance tests (prefixed and unprefixed).

Rationale
Forms and forms-related-interop problems are very common developer complaints, e.g. state of CSS.

[jgraham]

So I'm in favour of improving form controls in general, but I'm concerned about the lack of spec/tests for the areas that are causing developer pain. Is there an area that meets the bar of:

• Causes significant compat problems or represents an area of difficulty for developers
• Has a spec with multi-vendor buy-in.
• Either has existing tests or for which tests could be written without major difficulty.

If not it seems like the first step in this area would be to agree on the path forward at the specification level and revisit including this in a future iteration of the Interop initative.

[jensimmons]

@mfreed7 What is your proposal for inclusion of a specific web technology to be included in Interop 2022?

"Improve form controls" is the mission of the Open UI project, and the HTML and CSS working groups. Interop 2022 is for encouraging browser rendering engines to implement features that have specifications, or to fix interop bugs that are particularly problematic for web developers.

[bkardell]

It strikes me that it might be easier for me to explain my comments/questions in the meeting and on the other issue with specific examples and questions.

"form element stylability" is too general thing probably, and is not well defined enough and seems mostly wound up in openui sutff. There might be something very specific worth looking at there, but I think it would need to be specific. However, unprefixed appearance:none/auto seems like it could be a specific example of something I would include in this bucket, and seems appropriate for interop 2022. There are a number of existing tests (linked above) there that have plenty of gaps we could close, and there might be more. Form submission and validation behaviors too have lots of tests and are very ragged, and maybe there should be more. input has many gaps in https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input#browser_compatibility. The example on #9 could go in a similar bucket for forms or interactive elemnts compat.

Inevitably, for some of these things there will be reasons like "the spec is unclear" or "the spec left this undefined, but there is a defacto standard, it seems".

This part has potential overlap with other venues, but I believe that's ok - not so much solving a new problem as filling observable gaps?

There are things, including PRs with tests like whatwg/html#5099 which point out things like this - is that 'in scope'? If so, I would say that also could go in a "forms" (or "interactive elements") bucket.

I think there are deltas on datalist too, but there are few tests. On twitter someone also suggested that there are "cross-browser differences in which events fire when. E.g., while typing each character in an input or keyboard scrolling through a select list." Not sure if those are known or have tests, but again, seems to fit a possible 'theme' here and if so would fit into what I'd expect would probably be appropriate.

[zcorpan]

For appearance (at least for the behavior of the property, not necessarily the precise rendering for all widgets), spec and test PRs are in review and I hope can be merged soon:

• Define which elements can be which kinds of widgets for CSS 'appearance' whatwg/html#7004
• [css-ui-4] Define how to compute the kind of widget to use for an element w3c/csswg-drafts#6537
• HTML: Compute kind of widget wpt#30267

[mfreed7]

Thanks for the comments here. I think I may have confused things by starting my list with "Form element stylability". I didn't intend this issue to be about forward-looking things that OpenUI is working on. As several have pointed out, that type of "new stuff" likely doesn't belong here.

But I do think perhaps some corner case form control stylability things could be included here? Things that provide developer frustration currently? Due to the poor state of form element standardization, perhaps many of these are going to end up being controversial. But just for a quick example, <input type=week value='2021-W45'> shouldn't match the appearance of <input type=text value='2021-W45'>. I.e. there should be a display that represents the week. This meets maybe 2.5 of the three bars:

• Causes significant compat problems or represents an area of difficulty for developers. (4 distinct comments on the State of CSS mentioned this area)
• Has a spec with multi-vendor buy-in. (spec, but appearance not specified)
• Either has existing tests or for which tests could be written without major difficulty. (we can write some, relatively easily)

I also just want to really +1 the comment from @bkardell. It seems like there's a collection of behaviors that do meet the bars above. It'll take some work to collect them, but that's the intention of this issue. @josepharhar from the Chromium team is willing to try to do that collection, if folks think this is a worthwhile area to include in Interop2022.

[jensimmons]

perhaps some corner case form control stylability things could be included here? Things that provide developer frustration currently?

Rather than spending more time debating whether or not we should perhaps consider a to-be-determined-in-the-future list of possibilities, I’d like to assert that yes, this is an area folks are willing to consider. There’s no one objecting to suggestions that concern form controls.

What’s needed is a specific list. What exactly is proposed? As we make decisions about what will be included in Interop 2022, we will be considering specific technology or bugs.

Without a list, we won’t be able to make any decisions. We can’t just “write a blank check”.

[mfreed7]

Rather than spending more time debating whether or not we should perhaps consider a to-be-determined-in-the-future list of possibilities, I’d like to assert that yes, this is an area folks are willing to consider. There’s no one objecting to suggestions that concern form controls.

Great! I guess I misunderstood your first post. As I said, @josepharhar will work on putting together a proposed list.

[josepharhar]

Here is a concrete list of things we could do, with specific actions bolded.

• <input type=range>
  • There were many entries in the spreadsheet of people asking for improved range stylability and pseudo elements.
  • The pseudo elements aren't specced, aren't tested, and aren't in every browser. We could spec and test these pseudo elements.
  • There is a datalist feature in the spec, I'm not sure if it is tested or if it's implemented in all browsers. We could make sure there are tests for range datalists.
• appearance
  • Merge the spec and test PRs in zcorpan's comment.
• date/time inputs
  • there were many entries in the spreadsheet asking for better date/time inputs. Most of the browsers have pickers for most of these implemented now, but there are still some that are missing: <input type=week> and <input type=month>. Firefox and safari fallback to <input type=text> for these. The spec seems a bit nebulous, and firefox passes all WPTs despite identical appearance to <input type=text>.
    • We could strengthen the spec for type=month and type=week to ensure there is some UI that helps the user pick a valid value.
    • We could make notref WPTs to ensure there is some custom UI instead of a plain text input for type=month and type=week, as well as other input types which are already implemented.

Other thoughts:

• I'm not sure what the issues with form validation are, but there was one entry in the spreadsheet asking for improvements.
• There were many entries asking for improved/stylable <select>. hopefully these are addressed by the new <selectmenu> element.
• There were multiple entires asking for checkboxes and radios. I'm not sure what exactly is wrong with them, but accent-color should at least make them all colorable when it is implemented by all browsers.
• accent-color was also mentioned multiple times, but I believe there are already decent WPTs and implementations being worked on in all browsers.
• There was one entry asking for <input type=number> styling. I found a stackoverflow post asking about styling the pseudo elements for the up and down buttons. I also noticed that chrome doesn't allow the user to type non-number characters into the number input, but this behavior doesn't exist in firefox or safari. We could write spec and tests for these behaviors, but they didn't get as much attention as the other compat issues.
• There were two entries asking for standard padding or layout of form controls. I'm not sure exactly what is included in the spec for this category or exactly what people want, but I at least know that zcorpan did some recent work in this area:
  • Default 'display' value for form controls whatwg/html#4082
  • Make all input elements display: inline-block by default whatwg/html#4840
  • HTML: test display and box-sizing UA style for form controls wpt#20148

Note: When I say "entries" or "spreadsheet", I'm talking about the rows of this state of CSS feedback spreadsheet: https://docs.google.com/spreadsheets/d/1W1scqBvz6xRERPGOCm_29cc71U9CJiIlfeNNyys-27M/edit#gid=1399297592

[gsnedders]

<input type=range>

• The pseudo elements aren't specced, aren't tested, and aren't in every browser. We could spec and test these pseudo elements.

To be clear, I presume you’re suggesting we (browser vendors collectively) do this work in the appropriate group (be in the WHATWG HTML workstream, the W3C CSS WG, or W3C Open UI CG?

On the whole, I think including anything which doesn’t already have a specification (in at least draft form) should probably be excluded, as otherwise we’re aiming to go from a blank page to an interoperably implemented specification within a single year, which seems very ambitious at best, and utterly unrealistic at worst.

[foolip]

This issue could probably be handled similarly to the suggestion in #4 (comment), including the well-spec'd and -tested things, and some more future-looking work happening, but not as part of a metric.

[bkardell]

Given the confusion in the description and discussion here - just for ease of following, can I suggest that we open a new issue with the specifics and close this one... it's too easy to start at the top and immediately read it differently than I think several of us are talking about which does not include future looking work. I don't think editing the OP is better.

[josepharhar]

It seems like maybe I should be providing a list of existing WPTs instead of suggesting spec or test changes...?

[jgraham]

Based on the confusion here, I think a new issue with a list of:

• Areas (from existing specs) that comprise the proposal.
• wpt tests covering those areas, plus any known missing cases or tests that need to be updated.

will be needed to assess the proposal for Interop 2022.

If there are additional areas where you think we aren't there yet, but where it might be possible to make enough progress on pre-implementation work (e.g. specs) in the next year to include in a future iteration of this scheme, open an additional issue for those parts, explaining the situation.

[jensimmons]

maybe I should be providing a list of existing WPTs instead of suggesting spec[…] changes?

That’s right.

This group does not have the ability to work on specifications. Web standards are created by working groups (and community groups) that have a rigorous process for ensuring the ideas generated by the group are dedicated to open source and can become part of the web. We cannot to put together an ad-hoc group to invent something new, without the legal contracts in place to protect the ownership of that invention.

For example, in order to be a member of the CSS Working Group, you have to either be an Invited Expert who signs a stack of documents pledging the legal ownership of the work you do to the W3C, or you are an employee of a corporation, to which you've pledged ownership of the work you are doing to them, and they've formally joined the W3C and agreed that the work there done by their employees is owned by the W3C. It's a big deal, and involves a lot of lawyers.

This group is not a working group for writing specifications. We are simply communicating about our separate efforts to implement specs that were created elsewhere, to share tests to see if we've done a good job with those implementations, and to make it easy to talk about these efforts with web developers.

This group can collaborate on manual testing and writing up a report of findings (as suggested to better understand the current state of viewport measurements #4). Such findings could then be taken up by a Working Group as a to-do list for spec work. But there's no need for Interop 2022 to do so for the topic of form controls, because the Open UI Community Group was already formed to do exactly that. It does not make sense to duplicate efforts in two places.

If you want to help make headway improving interop for form controls as part of Interop 2022, please refer to existing web standards (with links), and highlight areas where browsers do not match those standards (with tests). Opening a new issue with a much tighter scope is a good idea.

The larger question of improving developer experience of styling form controls is especially tricky, because the web standards for form controls are vague and leave a lot up to interpretation by the browser. Form controls are not 100% interoperable because of the long history of them being considered part of the browser UI or operating system UI, not part of a web page. Each browser decides for themselves how they want their color picker or calendar widget to look & work. And that’s not going to change anytime soon. Such controls are designed to be easily understood by the end user, in the context of using their device.

If you are interested in tackling the larger conundrum — figuring out how to make it possible for web developers to more easily create branded form controls consistent across browsers, despite the fact each browser will decide for itself how to make default controls — then I do suggest you get involved with Open UI. I know progress is slow, and perhaps you want things to happen much more quickly, but this is an incredibly complex space. It will take years to solve these needs, and Open UI is already making great progress.

[josepharhar]

As @mfreed7 mentioned, we've put together a specific list of form element interop areas, each with a spec, tests, and developer interest. All of these are "backwards looking" - they cover areas that are already standardized and have tests written. We would like to propose this list be used as the "Form Controls" item for Interop-2022.

Here is the proposed list:

• appearance
  • Spec
    • https://drafts.csswg.org/css-ui/#appearance-switching
    • Define which elements can be which kinds of widgets for CSS 'appearance' whatwg/html#7004
    • [css-ui-4] Define how to compute the kind of widget to use for an element w3c/csswg-drafts#6537
  • Tests
    • https://wpt.fyi/results/css/css-ui?q=appearance
    • https://wpt.fyi/results/html/rendering/widgets/appearance
    • https://wpt.fyi/results/css/css-ui?label=pr_head&max-count=1&pr=30267
  • Browser bugs
    • https://bugs.chromium.org/p/chromium/issues/detail?id=1248381
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1730147
    • https://bugs.webkit.org/show_bug.cgi?id=233430
    • https://bugs.webkit.org/show_bug.cgi?id=143842
  • Developer feedback
    • One entry asking for appearance support: https://docs.google.com/spreadsheets/d/1W1scqBvz6xRERPGOCm_29cc71U9CJiIlfeNNyys-27M/edit
    • https://twitter.com/jensimmons/status/1044354839046750209
• accent-color
  • Spec
    • https://drafts.csswg.org/css-ui/#propdef-accent-color
  • Tests
    • https://wpt.fyi/results/css/css-typed-om/the-stylepropertymap/properties?q=accent-color
    • https://wpt.fyi/results/css/css-ui?q=accent-color
  • Browser bugs
    • https://bugs.webkit.org/show_bug.cgi?id=227587
  • Developer feedback
    • 6 unique entries requesting support in state of css feedback: https://docs.google.com/spreadsheets/d/1W1scqBvz6xRERPGOCm_29cc71U9CJiIlfeNNyys-27M/edit
• <input type=month> and <input type=week>
  • Spec
    • https://html.spec.whatwg.org/multipage/input.html#week-state-(type=week)
    • https://html.spec.whatwg.org/multipage/input.html#month-state-(type=month)
  • Tests
    • https://wpt.fyi/results/html/semantics/forms/the-input-element/month.html
    • https://wpt.fyi/results/html/semantics/forms/the-input-element/week.html
  • Browser bugs
    • https://bugs.webkit.org/show_bug.cgi?id=200416
• input elements, especially type=month and type=week
  • Spec
    • https://html.spec.whatwg.org/multipage/input.html
    • https://html.spec.whatwg.org/multipage/input.html#week-state-(type=week)
    • https://html.spec.whatwg.org/multipage/input.html#month-state-(type=month)
    • https://html.spec.whatwg.org/multipage/form-control-infrastructure.html#textFieldSelection
  • Tests
    • https://wpt.fyi/results/html/semantics/forms/the-input-element
    • https://wpt.fyi/results/html/dom/reflection-forms.html
    • https://wpt.fyi/results/html/rendering/non-replaced-elements/form-controls
    • https://wpt.fyi/results/css/cssom/cssstyledeclaration-setter-form-controls.html
    • https://wpt.fyi/results/html/semantics/forms/textfieldselection
  • Browser bugs
    • https://bugs.chromium.org/p/chromium/issues/detail?id=521871
    • https://bugs.chromium.org/p/chromium/issues/detail?id=1234420
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1685926
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1738866
    • https://bugs.webkit.org/show_bug.cgi?id=233426
    • https://bugs.webkit.org/show_bug.cgi?id=233732
    • https://bugs.webkit.org/show_bug.cgi?id=233721
  • Developer feedback
    • 4 unique entries requesting support for date/time inputs: https://docs.google.com/spreadsheets/d/1W1scqBvz6xRERPGOCm_29cc71U9CJiIlfeNNyys-27M/edit
• <form rel>
  • Spec
    • https://html.spec.whatwg.org/multipage/links.html#linkTypes
  • Tests
    • https://wpt.fyi/results/html/semantics/forms/form-submission-target
  • Browser bugs
    • https://bugs.chromium.org/p/chromium/issues/detail?id=931209
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1509346
    • https://bugs.webkit.org/show_bug.cgi?id=194398
• form submission
  • Spec
    • https://html.spec.whatwg.org/multipage/form-control-infrastructure.html#plan-to-navigate
  • Tests
    • https://wpt.fyi/results/html/semantics/forms/form-submission-0
    • https://wpt.fyi/results/html/browsers/browsing-the-web/navigating-across-documents/replace-before-load
  • Browser bugs
    • https://bugs.chromium.org/p/chromium/issues/detail?id=1234409
    • https://bugs.webkit.org/show_bug.cgi?id=233427
  • Developer feedback: In 2019 and 2020 there were a series of form submission regressions I worked on, which produced many WPTs in form-submission-0. Clearly there is developer interest in supporting these edge cases as seen in the bug reports, but not all browsers pass the form-submission-0 tests.
    • https://bugs.chromium.org/p/chromium/issues/detail?id=977882
    • https://bugs.chromium.org/p/chromium/issues/detail?id=1084874
    • https://bugs.chromium.org/p/chromium/issues/detail?id=1085097
    • https://bugs.chromium.org/p/chromium/issues/detail?id=1091583
    • https://bugs.chromium.org/p/chromium/issues/detail?id=1092200
    • https://bugs.chromium.org/p/chromium/issues/detail?id=1092313
    • https://bugs.chromium.org/p/chromium/issues/detail?id=1055243
    • https://bugs.chromium.org/p/chromium/issues/detail?id=1093888
    • https://bugs.chromium.org/p/chromium/issues/detail?id=1037740
    • https://bugs.chromium.org/p/chromium/issues/detail?id=1094246
• Events on disabled form controls
  • Spec
    • https://html.spec.whatwg.org/multipage/form-control-infrastructure.html#enabling-and-disabling-form-controls%3A-the-disabled-attribute
  • Tests
    • https://wpt.fyi/results/dom/events/Event-dispatch-click.html
  • Browser bugs
    • https://bugs.chromium.org/p/chromium/issues/detail?id=588760
    • https://bugs.chromium.org/p/chromium/issues/detail?id=693521
    • https://bugs.chromium.org/p/chromium/issues/detail?id=1115661
    • https://bugs.chromium.org/p/chromium/issues/detail?id=1169799
    • https://bugs.chromium.org/p/chromium/issues/detail?id=1245511
  • Developer feedback
    • In addition to the user-reported chrome bugs, this article got a decent amount of attention: https://jakearchibald.com/2017/events-and-disabled-form-fields/
• Form validation
  • Spec
    • https://html.spec.whatwg.org/multipage/#the-constraint-validation-api
    • https://html.spec.whatwg.org/multipage/form-control-infrastructure.html#validitystate
  • Tests
    • https://wpt.fyi/results/html/semantics/forms/constraints
  • Browser bugs
    • https://bugs.chromium.org/p/chromium/issues/detail?id=1234417
    • https://bugs.chromium.org/p/chromium/issues/detail?id=521871
    • https://bugs.chromium.org/p/chromium/issues/detail?id=1234417
    • https://bugs.webkit.org/show_bug.cgi?id=233428
  • Developer feedback
    • One entry asking for improved validation support: https://docs.google.com/spreadsheets/d/1W1scqBvz6xRERPGOCm_29cc71U9CJiIlfeNNyys-27M/edit

[jensimmons]

Thanks for creating this list, @josepharhar. This makes discussing form controls very concrete. Excellent.

For WebKit bug tickets…

• appearance — actually, this is the ticket: https://bugs.webkit.org/show_bug.cgi?id=143842. Could you update the link in your comment?
• input elements, especially type=month and type=week — could we break this out into two items:
  • 1. implementing missing controls (type=month and type=week); and
  • 2. fixing bugs that are causing test failures of input elements. Then could you create a new WebKit ticket for the latter, much like you did the other buckets (like Form validation) of test failures. This will be helpful to separate the work into actionable chunks. Then let's ignore https://bugs.webkit.org/show_bug.cgi?id=200416 as a duplicate of work that's already been completed & shipped.

[gsnedders]

input elements, especially type=month and type=week

Looking at the data from chromestatus.com:

type	usage
hidden	48.53514%
text	45.42959%
submit	25.57569%
search	16.79710%
checkbox	11.37675%
button	6.47834%
email	5.17911%
radio	4.55302%
password	4.32131%
file	3.45936%
number	2.28061%
range	2.27417%
image	1.35899%
tel	1.08187%
url	0.26108%
date	0.19424%
reset	0.13829%
time	0.01665%
color	0.00930%
datetime-local	0.00632%
month	0.00141%
week	0.00012%

AFAICT, the reasons why month/week are so comparatively low aren't because of lack of interop, but because they are relatively niche. (And indeed, part of the reason why Safari on macOS lacks support for them is because macOS doesn't provide month or week pickers either.) Focusing on them especially seems perhaps unjustified, given I don't see much evidence of developer interest? It is, after all, easy to use the native ones then fallback.

The survey results linked do cite date/time inputs, but I don't think we have any clear signals as to what the problems are. Given those survey results are focused on CSS, one might assume it's about styling the inputs, which those tests don't cover (and isn't specified anywhere).

There's certainly interop issues there—I'm not questioning that—but it's the question of whether it's a priority, especially with any focus we want for Interop 2022.

At least personally, I think the rest of the things in the above comment (i.e., aside from input elements) are not unreasonable, but I do wonder if we should split them into smaller chunks than a singular "form controls" one.

[josepharhar]

Thanks for the feedback!

appearance — actually, this is the ticket: https://bugs.webkit.org/show_bug.cgi?id=143842. Could you update the link in your comment?

Thanks, I added this bug to the appearance section of my comment. I kept the new webkit bug I filed in there as well though because 143842 is marked as fixed and doesn't appear at first to be about the new spec behavior that will hopefully be changed soon.

input elements, especially type=month and type=week — could we break this out into two items

Done! I filed two new WebKit bugs for the the-input-element directory of WPTs and included them in my list.

Then let's ignore https://bugs.webkit.org/show_bug.cgi?id=200416 as a duplicate of work that's already been completed & shipped

This bug is still open and includes week and month in its description... should I try to close that one and open new one(s) that are just for week and month?

[annevk]

Sam makes a pretty compelling case to exclude week and month controls. I also couldn't find anything in the CSS survey that specifically relates to those.

[SebastianZ]

Regarding events on disabled form controls:
https://bugzilla.mozilla.org/show_bug.cgi?id=1220048

Sebastian

[foolip]

I have now labeled (almost) all tests listed in #11 (comment) with the interop-2022-forms label:
https://wpt.fyi/results/?label=master&label=experimental&product=chrome&product=firefox&product=safari&aligned&q=label%3Ainterop-2022-forms

@josepharhar can you go through that and see if I seem to have missed anything? I excluded crash and tentative tests, and also the tests from web-platform-tests/wpt#30267 are still in flight.

As we review this test list, looking at tests that don't pass or fail in the usual manner is worthwhile, mostly timeouts and harness errors:
https://wpt.fyi/results/?label=master&label=experimental&product=chrome&product=firefox&product=safari&aligned&q=label%3Ainterop-2022-forms%20%28status%3A%21ok%26status%3A%21pass%26status%3A%21fail%29

Most likely some of the tests could be improved to fail faster and with a clearer reason.

[josepharhar]

Yeah, timeouts are a pretty common failure case for navigation-related tests

If there is a way to make them fail without timing out, that would be awesome. I tried to do it myself but it seems like the test is rapidly restarting and I couldn't figure out how to detect the case where history.back() was used to start the test. If not, I guess that's OK as long as they are still included in interop2022.

I believe I've filed a Safari bug on those.

Can you find the bug? It would be great to link it to it on wpt.fyi

[domenic]

https://bugs.webkit.org/show_bug.cgi?id=227299

[foolip]

In web-platform-tests/wpt-metadata#2401 @ziransun is suggested that we drop a number of tests that involve week/month. They are:

https://wpt.fyi/results/html/semantics/forms/constraints/form-validation-checkValidity.html
https://wpt.fyi/results/html/semantics/forms/constraints/form-validation-reportValidity.html
https://wpt.fyi/results/html/semantics/forms/constraints/form-validation-validity-stepMismatch.html
https://wpt.fyi/results/html/semantics/forms/the-input-element/input-valueasdate-stepping.html
https://wpt.fyi/results/html/semantics/forms/the-input-element/input-valueasnumber.html
https://wpt.fyi/results/html/semantics/forms/the-input-element/input-valueasnumber-stepping.html
https://wpt.fyi/results/html/semantics/forms/the-input-element/input-valueasdate.html

I've gone through them, and for all of them the subtests all pass in Chrome and Firefox, and the only failures in Safari are related to week/month. That makes it a pretty clear case for dropping the tests, so I'll merge that PR to unlabel the tests. (If there had been a bunch of other subtests that we did want, we'd need to split the tests.)

[foolip]

A number of tests for this area came up in my analysis in #48. Quoting:

/dom/events/Event-dispatch-click.html is TIMEOUT in Chrome, but that's the expected failure mode when the event doesn't fire.

The 6 form-submit* tests in /html/browsers/browsing-the-web/navigating-across-documents/replace-before-load/ are ERROR in Safari. There's a NoSuchFrameException raised in Python so probably there's some WebDriver issue underneath this.

The 2 form-requestsubmit-during-* tests in the same directory are TIMEOUT in Safari. Needs investigation.

5 form-validation-* tests in /html/semantics/forms/constraints/ have MISSING tests because two different tests are created (and fail) in Safari than in Chrome and Firefox. These tests are about week/month and like web-platform-tests/wpt-metadata#2401 should be excluded, but the tests need to be fixed to run the same subtests in all browsers first to see if there are other failing tests that aren't about week/month.

/html/semantics/forms/constraints/infinite_backtracking.html is TIMEOUT in Firefox, presumably because it has the bug the test is probing for.

/html/semantics/forms/form-submission-0/form-double-submit-multiple-targets.html is TIMEOUT in Safari.

/html/semantics/forms/form-submission-0/form-double-submit-to-different-origin-frame.html is TIMEOUT in Firefox and Safari, but also fails in Chrome, tracked by https://crbug.com/1087077. I would guess one of the events never fire in Firefox/Safari and that the timeout is legit.

The rel-* tests in /html/semantics/forms/form-submission-target/ are TIMEOUT in Safari, but all of the subtests pass. This very unusual and needs investigation.

/html/semantics/forms/textfieldselection/select-event.html is TIMEOUT in Safari after the first subtest times out. Looks like the "select" event never fires and the test is OK.

/html/semantics/forms/textfieldselection/selection-not-application.html has fewer subtests in Safari, the week/month ones aren't created. This actually seems fine from a metrics point of view if we score each browser individually.

/html/semantics/forms/textfieldselection/selection-start-end.html is an overall TIMEOUT in Safari, with a bunch of NOTRUN subtests. What's odd is that no subtest has timed out. Needs investigation.

/html/semantics/forms/textfieldselection/textfieldselection-setRangeText.html is an overall TIMEOUT in Safari after one of the subtests is TIMEOUT due to an event not being fired. The test seems OK, but it's not obvious how to score this.

/html/semantics/forms/textfieldselection/textfieldselection-setSelectionRange.html is TIMEOUT in Safari because one subtest is TIMEOUT. Also tricky for scoring.

/html/semantics/forms/the-input-element/range-restore-oninput-onchange-event.html is TIMEOUT in Safari, both harness and subtests. An event isn't fired, test seems OK.

I would encourage folks to search for the name of their browser in this comment and see if anything needs investigation/fixing.

A bunch of the issues are about week/month input. web-platform-tests/wpt-metadata#2401 didn't remove all of those tests, but for what remains tests will have to be split. @josepharhar is this something you might look into? (I didn't check who wrote the tests.)

[josepharhar]

https://bugs.webkit.org/show_bug.cgi?id=227299

Thanks! I linked this bug here: web-platform-tests/wpt-metadata#2409

I've gone through them, and for all of them the subtests all pass in Chrome and Firefox, and the only failures in Safari are related to week/month. That makes it a pretty clear case for dropping the tests, so I'll merge that PR to unlabel the tests. (If there had been a bunch of other subtests that we did want, we'd need to split the tests.)

Sounds good to me

The 6 form-submit* tests in /html/browsers/browsing-the-web/navigating-across-documents/replace-before-load/ are ERROR in Safari. There's a NoSuchFrameException raised in Python so probably there's some WebDriver issue underneath this.

Oh, I didn't notice this, I thought they were just timeout. I now see that they say "MISSING". When you open the test on wpt.live, the page constantly reloads very fast in safari, maybe that's why there are other issues? As opposed to an actual infra problem?

/html/semantics/forms/textfieldselection/selection-start-end.html is an overall TIMEOUT in Safari, with a bunch of NOTRUN subtests. What's odd is that no subtest has timed out. Needs investigation.

I'll try fixing this test if I can.

/html/semantics/forms/textfieldselection/selection-not-application.html has fewer subtests in Safari, the week/month ones aren't created. This actually seems fine from a metrics point of view if we score each browser individually.

A bunch of the issues are about week/month input. web-platform-tests/wpt-metadata#2401 didn't remove all of those tests, but for what remains tests will have to be split. @josepharhar is this something you might look into? (I didn't check who wrote the tests.)

Isn't selection-not-applicable.html the only one? I'll try splitting it into two tests.

[josepharhar]

I just wrestled with selection-start-end.html for a while, and I'm happy to just let it keep timing out until webkit implements the corresponding behavior.

I made a PR to separate week and month cases from selection-not-application.html: web-platform-tests/wpt#32479

[foolip]

The 6 form-submit* tests in /html/browsers/browsing-the-web/navigating-across-documents/replace-before-load/ are ERROR in Safari. There's a NoSuchFrameException raised in Python so probably there's some WebDriver issue underneath this.

Oh, I didn't notice this, I thought they were just timeout. I now see that they say "MISSING". When you open the test on wpt.live, the page constantly reloads very fast in safari, maybe that's why there are other issues? As opposed to an actual infra problem?

Taking form-submit-button-click.html as an example, the test is indeed MISSING, but above that you'll see "Harness status" and "ERROR". If you toggle "Show Details" you'll see this text squashed into a narrow box:

ERROR message: Traceback (most recent call last):
  File "/Users/runner/work/1/s/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 406, in run_func
    self.result = True, self.func(self.protocol, self.url, self.timeout)
  File "/Users/runner/work/1/s/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 496, in do_testharness
    result = protocol.base.execute_script(
  File "/Users/runner/work/1/s/tools/wptrunner/wptrunner/executors/executorwebdriver.py", line 50, in execute_script
    return method(script)
  File "/Users/runner/work/1/s/tools/webdriver/webdriver/client.py", line 20, in inner
    return func(self, *args, **kwargs)
  File "/Users/runner/work/1/s/tools/webdriver/webdriver/client.py", line 791, in execute_async_script
    return self.send_session_command("POST", "execute/async", body)
  File "/Users/runner/work/1/s/tools/webdriver/webdriver/client.py", line 659, in send_session_command
    return self.send_command(method, url, body, timeout)
  File "/Users/runner/work/1/s/tools/webdriver/webdriver/client.py", line 623, in send_command
    raise err
webdriver.error.NoSuchFrameException: no such frame (404): 

Opening http://wpt.live/html/browsers/browsing-the-web/navigating-across-documents/replace-before-load/form-submit-button-click.html in STP 137 indeed just keeps reloading. I assume the problem is that the form is submitted and reloads the page. When wptrunner then tries to poll the page for test results (I think) the original page is gone.

Is there any way for this test to fail before the navigation happens, or is the navigation happening the bug itself that the test is supposed to detect? If it is, then I think we'll have to live with the ERROR, that it's a surprising manifestation of a real bug.

A bunch of the issues are about week/month input. web-platform-tests/wpt-metadata#2401 didn't remove all of those tests, but for what remains tests will have to be split. @josepharhar is this something you might look into? (I didn't check who wrote the tests.)

Isn't selection-not-applicable.html the only one? I'll try splitting it into two tests.

Thanks for splitting! There are also week/month subtests in these tests:

• form-validation-validity-rangeOverflow.html
• form-validation-validity-rangeUnderflow.html
• form-validation-validity-valid.html
• form-validation-validity-valueMissing.html
• form-validation-willValidate.html

[josepharhar]

Is there any way for this test to fail before the navigation happens, or is the navigation happening the bug itself that the test is supposed to detect? If it is, then I think we'll have to live with the ERROR, that it's a surprising manifestation of a real bug.

I tried querying window.history to see if the main page was navigated backwards to, which I think is the case where the test keeps restarting, but I couldn't quite figure it out.

Thanks for splitting! There are also week/month subtests in these tests:

web-platform-tests/wpt#32499

[jensimmons]

For reference, this is what we considered and decided:

Technology considered	Decision
appearance	Include
accent-color	Investigate
<form rel>	Include
Events on disabled form controls	Include
input type=month + type=week	Exclude
Input elements bugs	Include
Form submission bugs	Include
Form validation bugs	Include