Investigate use of jclouds for on-demand infrastructure

[smlambert]

See if we can leverage https://jclouds.apache.org/ to dynamically allocate Jenkins nodes for use by longer running jobs, and container testing.

Related:

1. https://wiki.jenkins.io/display/JENKINS/JClouds+Plugin
2. @karianna mentioned the security team has recommendations related to this type of infra model, which will be shared at some point.

[karianna]

Security team fully supports this idea! Will be easiest to start with any Docker-based builds but then use jclouds to bring up/down entire hosts/vms.

[sxa]

In principle I've always loved this way of doing things and advocated it, but we need to weigh it up against the possible negative impacts - mostly performance related. This is more a brain dump for cfuture consideration/discusion than anything else. DISCLAIMER: I haven't used jclouds :-)

• Time to do the initial provision to a "ready" system from the providers
• Time to setup the system (e.g. ansible) vs using pre-setup templates which would to be kept patched and adjusted in line with playbook changes
• Will likely require more time to do a clone since it won't be using an existing workspace (and java is quite large)
• Ease of debugging when a problem occurs

In the case of docker we'd need to know what the host systems would be and again the performance impact on the builds (time and network bandwidth if that's a concern) of having to potentially pull things down regularly, and again keeping the images up to date (same issue as for templates)

[karianna]

Well thought out points @sxa555

In principle I've always loved this way of doing things and advocated it, but we need to weigh it up against the possible negative impacts - mostly performance related. This is more a brain dump for future consideration/discussion than anything else. DISCLAIMER: I haven't used jclouds :-)

• Time to do the initial provision to a "ready" system from the providers
• Time to setup the system (e.g. ansible) vs using pre-setup templates which would to be kept patched and adjusted in line with playbook changes

I wonder if we can have immutable templated hosts (like AWS's AMIs). I suspect not all platforms support this type of idea (especially bare metal). I suspect we'll have to do this on a best efforts basis.

• Will likely require more time to do a clone since it won't be using an existing workspace (and Java is quite large)

You're right, again I wonder if we can prefetch a clone, fetch it in parallel or have it attached via a device.

• Ease of debugging when a problem occurs

This is a big concern and I think we'd need to make sure that all logs were streamed off and that failed builds kept the host/container alive.

In the case of docker we'd need to know what the host systems would be and again the performance impact on the builds (time and network bandwidth if that's a concern) of having to potentially pull things down regularly, and again keeping the images up to date (same issue as for templates)

I'm hoping that docker impact will be small, but I'm no expert on the disk I/O impact in particular (hello compiling native code off disk).

[smlambert]

All of this good discussion is why the title is "Investigate...", to try some stuff out and make a call whether it looks useful/promising to proceed. :)