Add support for running WPT suite

[delan]

Running our WPT jobs on self-hosted runners could speed up builds by several minutes, but saving that kind of time requires the full capacity of our three servers.

Let’s see if we can squeeze every last drop of time savings out of them. Our current best improvement is 17min → 10min, with 36 runners across GitHub plus four servers.

• Create an ubuntu2204-wpt image that correctly runs all tests (ff6bfad)
• Add support for reserving multiple runners (723f823)
• Add support for autoscaling or policy overrides (454fd1a...334eeee)
• Add support for dynamic scheduling (334eeee...39f8c60)

dependency discrepancy issues:

• 20 tests break if Droid Sans Fallback is installed servo#35029
• /css/css-fonts/font-variant-emoji-1.html and /css/css-fonts/font-variant-emoji-2.html break unless Noto Color Emoji is installed servo#35030
• /css/css-fonts/test-synthetic-italic.html depends on the presence of fonts-dejavu-extra servo#41043
• 25 tests depend on the presence of fonts-noto-cjk servo#41044

[delan]

• 1 runner (<120s build, 24GiB 16vcpu) = longest WPT job 66/70min, 19 stable unexpected
• 8 runners (181s build, 12GiB 8vcpu, one server) = longest WPT job 27/32min, 16 stable unexpected
  • /!\ oom kill on 1 runner /!\
• 16 runners (182s build, 12GiB 8vcpu, 8+8) = longest WPT job 17/22min, 20 stable unexpected
• 20 runners (<120s build, 12GiB 8vcpu, 4+8+8) = longest WPT job 13/18min, 19 stable unexpected
• 20 runners (<120s build, 12GiB 4vcpu, 4+8+8) = longest WPT job 19/24min, 19 stable unexpected
• 18 runners (<120s build, 12GiB 8vcpu, 6+6+6) = longest WPT job 12/17min, 19 stable unexpected
• 36 runners (<120s build, 6GiB 6vcpu, 12+12+12) = longest WPT job 13/18min, 21 stable unexpected
  • /!\ oom kill on 2 runners /!\
  • the commit says 8vcpu, but it’s actually 6vcpu
• 36 runners (<120s build, 6GiB 6vcpu, 12+12+12) = longest WPT job 12/17min, 20 stable unexpected
  • /!\ oom kill on 2 runners /!\
• 16 runners (180s build, GitHub-hosted) = longest WPT job 24/28min, success
• 20 runners (<120s build, GitHub-hosted) = longest WPT job 19/21min, success
• 20 runners (GitHub-hosted) = longest WPT job 19min, success
• 20 runners (GitHub-hosted) = longest WPT job 19min, success
• 20 runners (GitHub-hosted) = longest WPT job 30min(!), success
• 20 runners (GitHub-hosted) = longest WPT job 19min, success
• 20 runners (GitHub-hosted) = longest WPT job 19min, success

[delan]

With aa4d2af:

• 8 runners (18GiB 8vcpu, 4+4) = longest WPT job 22/26min, 19 stable unexpected
• 8 runners (GitHub-hosted) = longest WPT job 44/48min, success

[delan]

Fixed some image issues:

• 2253d1f (20 tests break if Droid Sans Fallback is installed servo#35029)
• 8ad69ca (/css/css-fonts/font-variant-emoji-1.html and /css/css-fonts/font-variant-emoji-2.html break unless Noto Color Emoji is installed servo#35030)

[delan]

• 12 runners (18GiB 8vcpu, 4+4+4, --timeout-multiplier 1) = longest WPT job 12/16min, 4 stable unexpected
• 12 runners (18GiB 8vcpu, 4+4+4, --timeout-multiplier 1) = longest WPT job 12/16min, success
• 12 runners (GitHub-hosted, --timeout-multiplier 1) = longest WPT job 24/28min, 1 stable unexpected
• 16 runners (18GiB 8vcpu, 4+4+4+4, --timeout-multiplier 1) = longest WPT job 11/15min, success

[delan]

.oO( what if we could use a combination of github-hosted and self-hosted runners, with autoscaling to make use of existing resources? )

• problem: github-hosted runners are much slower than self-hosted runners, so giving them equal shares of work would be inefficient

.oO( what if we could use work stealing or some other kind of dynamic scheduling? )

• decouple wpt runner chunks from github actions matrix
• split the wpt runner into a high number of chunks, say c = 200
• external server hands out chunk numbers: main...chunker
• each runner does a loop like this
  • while let Some(chunk_id) = POST("https://ci0.servo.org/chunk?build_id=<guid>&total_chunks=<c>")
  • mach test-wpt $chunk_id of $total_chunks

unfortunately this significantly slows down the build, which seems to be in part because every mach test-wpt invocation takes ~10 seconds to start:

• github-hosted, unmodified = 42min
• self-hosted, unmodified = 13min–17min/22min
• self-hosted, 400 chunks = 33min
• self-hosted, 200 chunks = 28min
• self-hosted, 100 chunks = 25min

[delan]

.oO( what if we could use a combination of github-hosted and self-hosted runners, with autoscaling to make use of existing resources? )

all builds in this comment are unmodified d91f4b4 with the example config (8GiB 8vcpu).

ci0	ci1	ci2	github :)

• 20 github + 6 self (104 chunks) = longest WPT job 17/22min, success
• 20 github + 12 self (104 chunks) = 8min–14min/18min, 1 stable unexpected

ci0	ci1	ci2	jupiter

• 20 github + 16 self (104 chunks) = 8min–14min/18min, minor failures
• 20 github + 16 self (216 chunks) = 8min–15min/17min, minor failures
• 20 github + 16 self (216 chunks, 2x/0.75x) = 6min–12min/14min, minor failures
  • the longest self-hosted job was only 8min, so they are being underutilised

[delan]

idea: forbid github-hosted runners to claim the last 1/3 of chunks (54d3ae6), since their test timeouts are so long.

• 20 github + 16 self (216 chunks, 2x/0.75x) = 5min–10min/13min, minor failures

[delan]

idea: reduce the apt install set for github-hosted wpt jobs, and use a simpler 20+15 runner config instead of 20+16.

• 20 github + 15 self (216 chunks, 2x/0.75x) = 5min–9min/11min, minor failures

[delan]

added some secretless authentication (25d926c).

• 20 github + 15 self (216 chunks, 2x/1x) = 5min–10min

grabbed the faster checkout code from linux.yml, and added a missing apt dependency.

• 20 github + 15 self (216 chunks, 2x/1x) = 5min–11min

[delan]

bumped up the runners to 24GiB 16vcpu.

• 20 github + 15 self (216 chunks, 2x/1x, 24GiB 16vcpu) = 5min–10min
• 20 github + 15 self (216 chunks, 2x/1x, 24GiB 16vcpu) = 4min–10min
• 20 github + 15 self (216 chunks, 2x/1x, 24GiB 16vcpu) = 4min–10min

[delan]

end-to-end workflow, with policy override, reservation, and chunker.

• 20 github + 12 self (192 chunks, 2x/1x, 24GiB 16vcpu) = /13min, build2.txt

what if we use servo-ubuntu2204 rather than servo-ubuntu2204-wpt, to reduce churn?

• 20 github + 12 self (192 chunks, 2x/1x, 24GiB 16vcpu) = /13min

what if we turn up --processes, or further turn up --timeout-multiplier, on self-hosted jobs?

• 20 github + 12 self (192 chunks, timeout 2x/1x, processes 1x/2x, 24GiB 16vcpu) = /13min, build3.txt
• 20 github + 12 self (192 chunks, timeout 2x/0.5x, processes 1x/1x, 24GiB 16vcpu) = /12min, build4.txt, bad

[delan]

questions to answer:

• what happens if two wpt builds happen concurrently?
• what happens if the monitor services are restarted prematurely?
• what happens if the monitor services are completely down?
• what happens if a policy override is made but not used?