Synapse Workers in Complement tests are slow (in CI) #12848
Description
Complement tests against Workerised Synapse are slow; a full suite takes about 52 minutes (before being split into 4 alphabetical groups to make it tolerable whilst still giving us the coverage immediately).
Complement starts a new homeserver for each test, which likely explains why SyTest doesn't have this problem (though apparently it's possible for state to leak between tests in SyTest).
Currently, that involves about 13 workers (Synapse processes) being started up, a Postgres, a Redis and an Nginx. We would do well to figure out what improvements we can make to cut down the time it takes to run a full test suite.
N.B. GitHub Actions runners have 2 cores if I remember. This problem is not so evident on a machine with many cores (a local run for me takes about 11 minutes).
However, the time that CI takes is probably the most important to consider (it should be no worse than existing tests; let's call it less than 25 minutes, which is the runtime of 'trial (3.7, postgres, all, 10)'), so that's where '52 minutes' comes from.