scaling issues in deploy

[tianon]

With #5 and #16, our builds.json and sources.json generators are now really fast (and respectful of our upstream registry rate limits), but we're still very slow at deploying images (to the point that deploying often fails before it gets through the full set 😭). The deploy job is currently effectively "generate a bunch of crane cp commands and run them", which isn't terribly efficient:

meta-scripts/jenkins.jq

Lines 13 to 17 in 80bb556

	| if length == 1 then
	@sh "crane copy \(.) \($target)"
	else
	@sh "crane index append --tag \($target) " + (map("--manifest " + @sh) | join(" ")) + " --flatten"
	end

I'm working on a new version of this that is more native to our process so that things like auth tokens can be shared between API invocations more correctly/effectively, but wanted to file an issue to track the problem (especially because it's now the main blocker to our scaling up much more).