Skip to content

Comments

Use new runner for testsuite#3656

Merged
jdetter merged 36 commits intomasterfrom
jdetter/use-new-runner
Nov 17, 2025
Merged

Use new runner for testsuite#3656
jdetter merged 36 commits intomasterfrom
jdetter/use-new-runner

Conversation

@jdetter
Copy link
Collaborator

@jdetter jdetter commented Nov 13, 2025

Description of Changes

Speed improvements:

CI Job Before After Change
CI/Unreal Engine 29m 19m 34% faster
CI/Testsuite 21m 11m 47% faster
C#/Unity 20m 16m 20% faster
CI/Build and test wasm bindings 10m 3m 70% faster
CI/Smoketests (linux) 17m 12m 29% faster

CI/Unreal Engine - switch to serial test execution
CI/Testsuite - partially switch to serial test execution

I also haven't seen an unexpected Operation cancelled, so I think that problem might be behind us.

API and ABI breaking changes

No user facing changes

Expected complexity level and risk

2 - this touches a lot of our CI. It has been solid so far but if we start getting random CI failures this could be the cause.

Testing

  • All CI passes

(ok as of this update the internal tests are failing but that isn't related to this PR 😅 )

Future Work

There are 2 remaining tests that could be optimized by moving them to faster runners:

  1. (~45m) The windows smoketests (difficult because it requires Windows). @bfops has brought up the idea of only running this test on PRs that are in the merge queue and I think that would be the best 80/20 here. There really is not a great reason to be running this on every commit anyway.
  2. (36m) The internal tests (easier because it's on Linux)

I would also consider moving from instantiating container images to instantiating VM snapshots. This would allow us a lot more flexibility and we wouldn't have to have as much container config in our workflow files. Based on guides I've seen online this is also somewhat common. Also managing docker within docker has been kind of a nightmare so it would be easier to run containers inside of VMs to remove a layer of containerization there.

Testsuite inconsistencies

We have at least 2 different jobs that are inconsistent/flaky: the "testsuite" and the "unreal engine tests".

I'd like @JasonAtClockwork to make the unreal engine tests run sequentially so that we have less random failures. I know this will increase the testing time but due to this PR we should have extra headroom there.

I've also pinged Joshua about one of the tests that I've seen sometimes fails randomly. This is either an issue in the test or an issue in SpacetimeDB itself, I'm not sure yet.

These inconsistencies are present both in our custom runner and in the Github runners, so merging this PR will not make this situation any worse than it already is.

@jdetter jdetter force-pushed the jdetter/use-new-runner branch from d54b8b9 to 62a4f7e Compare November 13, 2025 02:04
@jdetter jdetter force-pushed the jdetter/use-new-runner branch from c80af6b to ac437f8 Compare November 13, 2025 02:12
Updated

Runner fix

Don't install dotnet in the container for smoketests

Rerun

Use container settings

Rerun again

Try again

Yet another run

Dont use runner user

Try again

Test ephemeral runners
@jdetter jdetter force-pushed the jdetter/use-new-runner branch 3 times, most recently from 9e5a16d to f532af7 Compare November 13, 2025 15:09
Another CI run

Remove comment

Concurrency test

Concurrency test 2

concurrency test 3

Test

test 2

Attempt fix

Add more jobs to the new runner

Use container

Remove a small diff

Another diff fix

Enable cancel in progress
@jdetter jdetter force-pushed the jdetter/use-new-runner branch from b8b82f5 to 7ba5a74 Compare November 13, 2025 20:42
@jdetter jdetter force-pushed the jdetter/use-new-runner branch 2 times, most recently from 8b6c3ac to ca96586 Compare November 14, 2025 16:54
@jdetter jdetter force-pushed the jdetter/use-new-runner branch from ca96586 to 2ddb8a1 Compare November 14, 2025 17:15
@jdetter jdetter marked this pull request as ready for review November 17, 2025 15:54
@jdetter jdetter requested a review from bfops November 17, 2025 15:54
Copy link
Collaborator

@bfops bfops left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this LGTM, though I worry about you being the only person who understands these changes. If something went wrong, I wouldn't know how to debug it.

@bfops bfops added release-any To be landed in any release window no runtime change This change does not affect the final binaries labels Nov 17, 2025
@jdetter jdetter force-pushed the jdetter/use-new-runner branch 2 times, most recently from de5c553 to 7213e91 Compare November 17, 2025 18:56
@jdetter jdetter enabled auto-merge November 17, 2025 19:26
@jdetter jdetter disabled auto-merge November 17, 2025 19:30
@jdetter jdetter enabled auto-merge November 17, 2025 19:50
@jdetter jdetter disabled auto-merge November 17, 2025 19:50
@jdetter jdetter enabled auto-merge November 17, 2025 20:17
@jdetter jdetter added this pull request to the merge queue Nov 17, 2025
Merged via the queue into master with commit 05ecc80 Nov 17, 2025
25 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no runtime change This change does not affect the final binaries release-any To be landed in any release window

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants