xdist multi-machine CI integration

[simon-weber]

This is the same topic as #283: I'm looking into running xdist with remote machines on Jenkins.

Our current setup uses distributed-nose, which works by splitting the suite into pieces, running those separately, then aggregating the results. The problem is that it gets difficult to create well-balanced static shards as we parallelize more (even when distributing by expected test runtime). Xdist's dynamic dispatching fixes this.

So, as far as I can tell, these are my options for running xdist in an automated multi-machine setup:

1. reserve all workers ahead of time (either through jenkins or some other way), then use xdist as-is
2. modify xdist so workers can connect back dynamically, rather than be connected to ahead of time
3. modify xdist to decouple it from workers (eg have the master+workers communicate via pubsub)

Is there anything I'm missing? Any thoughts on the feasibility/advantages/disadvantages of building the 2nd/3rd options on top of xdist? I'm not too familiar with how it works under the hood.

[simon-weber]

I dug around the code for a bit. Options 2 and 3 look tricky given how execnet is used. I came up with something else that might work:

4. modify xdist to a) start running tests after collection is done on the first worker b) continually retry to connect other workers

Then something like this might work:

• run the master, configuring the workers as local ports in a predefined range
• run a parallel task, giving each shard one port + the master's ip
  • each shard does setup then starts a remote ssh tunnel from master:port to itself

Thoughts?

[nicoddemus]

Hi @simon-weber, thanks for the detailed description.

About your 4) option, it does seem like a good idea, I think you would need a separate scheduler to accomplish that. There are 4 schedulers currently implemented, you might use them as basis for your implementation.

Also, execnet supports remote execution through the --tx option (althought I have never used it myself). Did you try that?

[simon-weber]

Great. I'll let you know how it goes.

I haven't played too much with the --tx options aside from abusing the interpreter param. Was there something particular you had in mind?

[nicoddemus]

I think it can be used to start remote SSH sessions, although I have never used it myself.

[simon-weber]

Gotcha. Here's a messy proof of concept: master...venmo:worker_retry. It seems to work with local socket servers: I was able to start the master, wait, then launch the workers separately later on. All the workers ended up having tests distributed to them. I was pleasantly surprised that all the plumbing of adding workers at runtime was already handled!

I'm going to try to hook up the jenkins side next. In the meantime, what would you want to see in a proper PR? This could potentially all go into a scheduler, though I think it'd require adding more hooks (to delegate the dsession and workermanage changes). It also seems like this could be separate from scheduling; ie, it might make sense as a flag in xdist that would apply regardless of the scheduler.

[nicoddemus]

Hi @simon-weber,

Thanks for sharing it! Indeed the patch looks small. Is that the complete set of changes though? Seems really small in comparison to what you want to accomplish.

Please continue with your tests and we can discuss the next steps then; my initial gut feeling is that I would prefer to add new hooks and have this functionality as a separate plugin, at least for now. @RonnyPfannschmidt what do you think?

[RonnyPfannschmidt]

as is all this allows is missing nodes, without an accompanying test that demonstrates how this can work im not sure this is sensible

[simon-weber]

Yup, this does seem to be all that's needed. I've got this working in jenkins now - which was trickier than expected - and I'm planning to have it run head to head with our normal builds for a week or so.

(If anyone finds this and is interested in the jenkins implementation, feel free to reach out to me.)

[nicoddemus]

Closing for now, but please keep us posted @simon-weber!

[fruch]

@simon-weber any more info on how you set it up ?

[simon-weber]

I ran the configuration described in #299 (comment) for a bit. It worked ok, but wasn't very reliable since reconnecting the workers after a failure was an issue.

I eventually switched to https://github.com/Shopify/ci-queue, which was more suited for this use case.