Agenda 2024-01-30

[jgraham]

• Discuss 2024 goals and roadmap: Add Interop 2024 Goals and Roadmap #19
  • Work out how scoring corresponds to roadmap
• Change the repetition rule to avoid the occasional conflict with the WPT Infra meeting.

[gsnedders]

I never got any email for any invite for this?

The last iCalendar file that got sent to me with "mobile" in the title still had a repetition end date of 2023-19-12. Not sure if the event got modified and Google Calendar just didn't send iMIP emails for this?

Also Agenda+: change the repetition rule to avoid the occasional conflict with the WPT Infra meeting.

[gsnedders]

Interop Mobile Investigation Minutes 2024-01-30

Present: James Graham, James Scott, Panos Astithas, Sam Sneddon, Jonathan Bedard

Goals & Roadmap

James G: made a proposal, based on last meeting, #19
... carry over Safari work that didn't happen
... and investigate failures [see PR]
... if anyone has any comments

Sam: Mobile Safari is only meaningful for stable and beta, because that's the cadence Apple ships it.
... Ref tests on Mobile Safari would be meaningless due to the inability to change the viewport size.
... The general aim has been to get WebKitTestRunner working and submitting results to the dashboard, as that allows tip-of-tree results, not tied to any stable/beta release.
... Though obviously there are differences to Mobile Safari.

James G: That seems fine, if you can propose a change to the PR.

Sam: Might still be useful to run Mobile Safari, albeit without reftests given results there would be misleading.

James G: that would work for wpt.fyi, but not Interop.
... aside from Safari/WKTR, any other comments?

Panos: Do we think getting the diff of mobile/desktop to zero is realistic?

James G: The key word there is "unexpected". If we have a justification, that's fine.
... the difference for Firefox (GeckoView) isn't huge, Chrome has a larger diff (but sets no special timeout for Android).

James S: Weizhong tried to increase the timeout, but then hit other problems.

Panos: I think it timed out the overall test job?
... I hadn't noticed the "unexpected", maybe worth drawing more attention to it.

James G: It's probably not reasonable to expect it to reach zero. We expect differences. We don't expect tests to time out, reftests to need more fuzziness, but we should get down to reflecting product differences.

Sam: And to be clear, we do intend to get WKTR running on macOS too, given otherwise there'd be a much larger gap between WKTR on iOS and STP on macOS.

Panos: To check I understand, the plan is to have WKTR, GeckoView, Chrome on Android for experimental, then Mobile Safari, Firefox on Android, Chrome on Android for stable?

James G: I was mostly focusing on experimental, and not thinking too much about stable. Adding stable would increase load. I'd personally be happy with just having a mobile experimental view, but there is some value in having a stable view.

Panos: Maybe this is a question for the Interop group, what they need?

James G: My sense is people mostly look at the experimental results, but of course stable is what's shipping to users.

Panos: To take this to the hypothetical, would this be the platforms for stable?

Sam: [re-summarizes about Safari and reftests], WKTR has some complication on stable because open source + public SDKs isn't necessarily defended in CI

Jonathan: I think that's fair, and WKTR is probably the only sensible direction.

Panos: So is this just in 2024, or might Safari be able to change viewport size in future?

Jonathan: That seems unlikely, because there's little benefit outside of very specific testing situations.

Sam: Web developers probably want to be testing actual device behavior, not something that's different, so it's a very niche request. I'm very doubtful on changing.

Panos: Would it be any different on iPad?

Sam: Maybe? Stage manager allows some amount of resizing, but not sure how much.

Jonathan: But also only in one dimension, I think.

Panos: OK, so that might get us slightly closer.

Jonathan: Is the overall goal to just test on a different form factor, or to test for differences? [??? maybe not correctly minuted]

Panos: I think the goal is to be able to compare end-user experience of mobile browsers, and be able to check that they work versus a standardized set of tests. Not against changing tests, in light of Chrome/Firefox on Android v. desktop changes.

Jonathan: How much do we care about browsers or engines here? Because if we care about browsers this leads to much harder questions about what exactly we're testing here and what the scope is.

James G: For Mobile Firefox, we're using the GeckoView test runner which is not actual mobile Firefox, but most of the time I'd expect it to be very similar to actual Mobile Firefox. But there are cases that are interesting to people, like the interaction with the URL bar with viewport units. But we don't have that many tests there because it's very hard to actually figure out how to test this in general, because there are differences in how you make browser chrome appear/disappear.

... for Safari, I have the view that there's a substantially larger difference between WKTR and Safari, for example if there's system components that are only being used by one of them.

... as for changing the tests, we can't rewrite every reftest to not assume a fixed viewport, that's probably multiple years of effort. We probably can't change that at that point.

... if we want specific types of tests for mobile, we might want to look at adding a new test type of reftest for mobile.

Jonathan: The differences between Safari and WKTR: there's many switches that exist in WKTR and not Safari; Safari has many features that don't exist in WKTR, but they tend not to be web standards related things (password autofill, etc); and we've viewed WPT as testing the engine and not the browser. And I don't think that's how the others are viewing this, rather viewing engine and browser as more tightly coupled?

Panos: There are aspects like the credential management spec which are linked to features like that.

... I think in the short-term we view the goal as experimental, because stable will require a lot more investigation as to where differences are.

Jonathan: We've probably waited far too long for testing mobile, we probably should've done this a decade ago; and for example, the way Apple thinks of things like visionOS as an extension of mobile, and as we move into the future, we're moving into a world in which desktop is the special-case, not mobile.

... It sounds like this year we'll be using WKTR but not Safari, and we'll take a look at the differences between WKTR and Safari at some point in the future because there's interest in testing the browser, and not the engine, in the future?

James G: it sounds like the goal for this year therefore is to get WKTR on wpt.fyi on both macOS and iOS. for step 3, (looking at unexpected differences), it makes more sense to compare macOS to iOS WKTR.

Jonathan: Of course we can also compare WKTR to Safari on macOS, and look at the difference.

Panos: Is the plan to replace STP with WKTR on macOS?

Sam: In addition.

Sam: ...STP typically is on a branch, though without many changes, whereas WKTR will be from ToT, so there will be some changes.

Jonathan: Though there isn't really any reason why we can't build WKTR for the STP branches.

James: Someone from Apple should submit a change to the PR, but it sounds like we have consensus on the rest of this?

[rough consensus]

... Would be nice to have more r+s votes in favor?

Reschedule

Sam: The infra meeting is the first Tuesday; my proposal is we change this meeting to "second and fourth Tuesday".

James: ...I literally can't see a way to do this in Google Calendar.

Panos: The other way I've done this is for WHATNOT is have multiple calendar events.

Scoring (Goals & Roadmap Redux)

Panos: We should come up with some plan for scoring. Maybe 1/5th for each.

James G: 4 & 5 probably should be merged? Make them 4a, 4b, then 1/4 for each.

James G: If nobody has any other comments, no more agenda issues, and the meeting ends ~now, so byeeee!