Add utility slave

[marcoabreu]

This PR adds utility slaves.

The idea of utility slaves is to offload miscellaneous lifecycle-tasks that are not actually part of the task but rather involved to keep the pipeline alive. We should not process these tasks on the same slaves as actual workload since this has multiple implications:

1. mxnetlinux-cpu slaves are very powerful and only have 3 executors. We don't need 72 CPU cores to load a groovy script
2. Lifecycle tasks like publishing the GitHub status to PRs share the same queue as regular workload. During high load, this might result in a PR not being notified that a job has been queued (or finished) until a mxnetlinux-cpu worker is free. This might confuse the user.
3. Since a lifecycle task might be only executed when it reaches the top of the queue, the follow-up jobs will only be added to the queue after the lifecycle job has been processed. Only at this point, the real jobs will be enqueued. This means that it has two wait for it's part twice. Especially considering our auto scaling gets triggered every 5 minutes, it could mean that the real workload misses multiple scaling cycles since the lifecycle job hasn't passed yet and thus the "real" jobs haven't been scheduled yet.

Basically what this replaces is the concept of a high-priority queue. Since we don't have that system in Jenkins (and neither do we want to use expensive slaves for that), we can instead make these slaves with a lot of executors and a special label.

This job is NOT intended to run ANY custom workload but only to execute Jenkins operations that require a node context. Usually, these tasks would run on the Jenkins master, but I disabled that for security reasons.

The slave would be a c5.large with 2vCPUs.

[KellenSunderland]

LGTM, but I'd recommend we hold off on large changes in the CI for a few days to ensure things are stable for the release process. What do you think?

[marcoabreu]

This patch is exactly to improve the stability. We have to merge these changes before the release or we won't be able to support a patch release.

[KellenSunderland]

The intent of all software patching is to improve things in one dimension or another. Large changes at a critical time rarely achieve solely their intent as no-one writes perfect code without testing (at least I don't). Did we do any shadow testing of this change in the dev account? I'm ok with approving this as long as you're highly-confident that it wont cause issues.

[KellenSunderland]

Code looks ok with me.

[marcoabreu]

Yes, this PR (and the Backend changes) have been tested on the dev account :)

With patch I'm not talking about 1.3.1 or 1.4 but 1.4.1. we would have no CI coverage if we don't apply the pipeline changes before the branch is cut :/

[marcoabreu]

Thanks for your review!