feat: build packages in parallel #17723

kuisathaverat · 2020-04-15T10:26:48Z

What does this PR do?

It split the package by architecture to build it in parallel.

Why is it important?

It will reduce the build time for packages

kuisathaverat · 2020-04-15T10:27:25Z

/packaging

urso · 2020-04-15T11:12:15Z

.ci/packaging.groovy

+        axes {
+          axis {
+            name 'PLATFORMS'
+            values '+linux/armv7', '+linux/ppc64le', '+linux/s390x', '+linux/mips64', 'darwin'


Maybe I'm missing something, but don't we build windows packages?

not from the betas-ci, we only have this jobs https://beats-ci.elastic.co/job/elastic+beats+master+package/
I've check https://github.com/elastic/beats/blob/master/dev-tools/mage/platforms.go#L29-L70 and there is a bunch of platforms we do not build on the beats-ci, Should we build them?

checking the logs, I think does no matter what you put on PLATFORMS env var you always build all the packages

This is a log where PLATFORMS is +linux/s390x
https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/pipelines/packaging/branches/PR-17723/runs/1/nodes/155/log/?start=0

but after build all the cross-compile Docker images I think it is compiling packages that are not requested.

[2020-04-15T10:42:45.027Z] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=s390x-linux-gnu-gcc, CXX=s390x-linux-gnu-g++, GOARCH=s390x, GOARM=, GOOS=linux, PLATFORM_ID=linux-s390x] [2020-04-15T10:42:45.027Z] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=aarch64-linux-gnu-gcc, CXX=aarch64-linux-gnu-g++, GOARCH=arm64, GOARM=, GOOS=linux, PLATFORM_ID=linux-arm64] [2020-04-15T10:42:45.027Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=o64-clang, CXX=o64-clang++, GOARCH=amd64, GOARM=, GOOS=darwin, PLATFORM_ID=darwin-amd64] [2020-04-15T10:42:45.027Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=x86_64-w64-mingw32-gcc, CXX=x86_64-w64-mingw32-g++, GOARCH=amd64, GOARM=, GOOS=windows, PLATFORM_ID=windows-amd64] [2020-04-15T10:43:26.771Z] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, PLATFORM_ID=linux-amd64] [2020-04-15T10:43:26.771Z] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=gcc, CXX=g++, GOARCH=386, GOARM=, GOOS=linux, PLATFORM_ID=linux-386] [2020-04-15T10:43:29.276Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=aarch64-linux-gnu-gcc, CXX=aarch64-linux-gnu-g++, GOARCH=arm64, GOARM=, GOOS=linux, PLATFORM_ID=linux-arm64] [2020-04-15T10:43:29.276Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=386, GOARM=, GOOS=linux, PLATFORM_ID=linux-386] [2020-04-15T10:46:05.836Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, PLATFORM_ID=linux-amd64] [2020-04-15T10:46:19.465Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=i686-w64-mingw32-gcc, CXX=i686-w64-mingw32-g++, GOARCH=386, GOARM=, GOOS=windows, PLATFORM_ID=windows-386] [2020-04-15T10:47:15.806Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=s390x-linux-gnu-gcc, CXX=s390x-linux-gnu-g++, GOARCH=s390x, GOARM=, GOOS=linux, PLATFORM_ID=linux-s390x] [2020-04-15T10:51:47.176Z] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=s390x-linux-gnu-gcc, CXX=s390x-linux-gnu-g++, GOARCH=s390x, GOARM=, GOOS=linux, PLATFORM_ID=linux-s390x] [2020-04-15T10:51:47.176Z] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=aarch64-linux-gnu-gcc, CXX=aarch64-linux-gnu-g++, GOARCH=arm64, GOARM=, GOOS=linux, PLATFORM_ID=linux-arm64] [2020-04-15T10:51:47.176Z] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, PLATFORM_ID=linux-amd64] [2020-04-15T10:51:47.177Z] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=gcc, CXX=g++, GOARCH=386, GOARM=, GOOS=linux, PLATFORM_ID=linux-386] [2020-04-15T10:51:48.682Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=x86_64-w64-mingw32-gcc, CXX=x86_64-w64-mingw32-g++, GOARCH=amd64, GOARM=, GOOS=windows, PLATFORM_ID=windows-amd64] [2020-04-15T10:51:48.989Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=o64-clang, CXX=o64-clang++, GOARCH=amd64, GOARM=, GOOS=darwin, PLATFORM_ID=darwin-amd64] [2020-04-15T10:51:49.289Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, PLATFORM_ID=linux-amd64] [2020-04-15T10:51:49.577Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=386, GOARM=, GOOS=linux, PLATFORM_ID=linux-386] [2020-04-15T10:55:11.890Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=aarch64-linux-gnu-gcc, CXX=aarch64-linux-gnu-g++, GOARCH=arm64, GOARM=, GOOS=linux, PLATFORM_ID=linux-arm64] [2020-04-15T10:55:39.158Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=s390x-linux-gnu-gcc, CXX=s390x-linux-gnu-g++, GOARCH=s390x, GOARM=, GOOS=linux, PLATFORM_ID=linux-s390x] [2020-04-15T10:55:47.588Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=i686-w64-mingw32-gcc, CXX=i686-w64-mingw32-g++, GOARCH=386, GOARM=, GOOS=windows, PLATFORM_ID=windows-386] [2020-04-15T11:00:17.233Z] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, PLATFORM_ID=linux-amd64] [2020-04-15T11:00:17.233Z] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=s390x-linux-gnu-gcc, CXX=s390x-linux-gnu-g++, GOARCH=s390x, GOARM=, GOOS=linux, PLATFORM_ID=linux-s390x] [2020-04-15T11:00:17.233Z] >> Building using: cmd='../../heartbeat/build/mage-linux-amd64 golangCrossBuild', env=[CC=x86_64-w64-mingw32-gcc, CXX=x86_64-w64-mingw32-g++, GOARCH=amd64, GOARM=, GOOS=windows, PLATFORM_ID=windows-amd64] [2020-04-15T11:00:17.233Z] >> Building using: cmd='../../heartbeat/build/mage-linux-amd64 golangCrossBuild', env=[CC=o64-clang, CXX=o64-clang++, GOARCH=amd64, GOARM=, GOOS=darwin, PLATFORM_ID=darwin-amd64] [2020-04-15T11:00:19.419Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=x86_64-w64-mingw32-gcc, CXX=x86_64-w64-mingw32-g++, GOARCH=amd64, GOARM=, GOOS=windows, PLATFORM_ID=windows-amd64] [2020-04-15T11:00:19.420Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=o64-clang, CXX=o64-clang++, GOARCH=amd64, GOARM=, GOOS=darwin, PLATFORM_ID=darwin-amd64] [2020-04-15T11:04:10.993Z] >> Building using: cmd='../../heartbeat/build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=386, GOARM=, GOOS=linux, PLATFORM_ID=linux-386] [2020-04-15T11:04:15.760Z] >> Building using: cmd='../../heartbeat/build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, PLATFORM_ID=linux-amd64] [2020-04-15T11:04:47.868Z] >> Building using: cmd='../../heartbeat/build/mage-linux-amd64 golangCrossBuild', env=[CC=aarch64-linux-gnu-gcc, CXX=aarch64-linux-gnu-g++, GOARCH=arm64, GOARM=, GOOS=linux, PLATFORM_ID=linux-arm64] [2020-04-15T11:05:04.160Z] >> Building using: cmd='../../heartbeat/build/mage-linux-amd64 golangCrossBuild', env=[CC=s390x-linux-gnu-gcc, CXX=s390x-linux-gnu-g++, GOARCH=s390x, GOARM=, GOOS=linux, PLATFORM_ID=linux-s390x] [2020-04-15T11:08:25.855Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=386, GOARM=, GOOS=linux, PLATFORM_ID=linux-386] [2020-04-15T11:08:47.841Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, PLATFORM_ID=linux-amd64] [2020-04-15T11:09:34.544Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=aarch64-linux-gnu-gcc, CXX=aarch64-linux-gnu-g++, GOARCH=arm64, GOARM=, GOOS=linux, PLATFORM_ID=linux-arm64] [2020-04-15T11:09:47.741Z] >> Building using: cmd='../../heartbeat/build/mage-linux-amd64 golangCrossBuild', env=[CC=i686-w64-mingw32-gcc, CXX=i686-w64-mingw32-g++, GOARCH=386, GOARM=, GOOS=windows, PLATFORM_ID=windows-386] [2020-04-15T11:11:54.243Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=i686-w64-mingw32-gcc, CXX=i686-w64-mingw32-g++, GOARCH=386, GOARM=, GOOS=windows, PLATFORM_ID=windows-386] [2020-04-15T11:12:01.802Z] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=gcc, CXX=g++, GOARCH=386, GOARM=, GOOS=linux, PLATFORM_ID=linux-386] [2020-04-15T11:12:04.365Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=s390x-linux-gnu-gcc, CXX=s390x-linux-gnu-g++, GOARCH=s390x, GOARM=, GOOS=linux, PLATFORM_ID=linux-s390x] [2020-04-15T11:13:26.020Z] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=aarch64-linux-gnu-gcc, CXX=aarch64-linux-gnu-g++, GOARCH=arm64, GOARM=, GOOS=linux, PLATFORM_ID=linux-arm64] [2020-04-15T11:17:35.398Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=s390x-linux-gnu-gcc, CXX=s390x-linux-gnu-g++, GOARCH=s390x, GOARM=, GOOS=linux, PLATFORM_ID=linux-s390x] [2020-04-15T11:18:24.276Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, PLATFORM_ID=linux-amd64] [2020-04-15T11:18:24.276Z] >> Building using: cmd='../../journalbeat/build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=386, GOARM=, GOOS=linux, PLATFORM_ID=linux-386] [2020-04-15T11:18:24.276Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=386, GOARM=, GOOS=linux, PLATFORM_ID=linux-386] [2020-04-15T11:20:42.369Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=aarch64-linux-gnu-gcc, CXX=aarch64-linux-gnu-g++, GOARCH=arm64, GOARM=, GOOS=linux, PLATFORM_ID=linux-arm64] [2020-04-15T11:22:49.461Z] >> Building using: cmd='../../journalbeat/build/mage-linux-amd64 golangCrossBuild', env=[CC=s390x-linux-gnu-gcc, CXX=s390x-linux-gnu-g++, GOARCH=s390x, GOARM=, GOOS=linux, PLATFORM_ID=linux-s390x] [2020-04-15T11:23:43.333Z] >> Building using: cmd='../../journalbeat/build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, PLATFORM_ID=linux-amd64] [2020-04-15T11:24:41.950Z] >> Building using: cmd='../../journalbeat/build/mage-linux-amd64 golangCrossBuild', env=[CC=aarch64-linux-gnu-gcc, CXX=aarch64-linux-gnu-g++, GOARCH=arm64, GOARM=, GOOS=linux, PLATFORM_ID=linux-arm64] [2020-04-15T11:24:52.326Z] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=s390x-linux-gnu-gcc, CXX=s390x-linux-gnu-g++, GOARCH=s390x, GOARM=, GOOS=linux, PLATFORM_ID=linux-s390x] [2020-04-15T11:24:54.897Z] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=gcc, CXX=g++, GOARCH=386, GOARM=, GOOS=linux, PLATFORM_ID=linux-386] [2020-04-15T11:24:57.437Z] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, PLATFORM_ID=linux-amd64] [2020-04-15T11:24:59.987Z] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=aarch64-linux-gnu-gcc, CXX=aarch64-linux-gnu-g++, GOARCH=arm64, GOARM=, GOOS=linux, PLATFORM_ID=linux-arm64] [2020-04-15T11:31:30.088Z] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=s390x-linux-gnu-gcc, CXX=s390x-linux-gnu-g++, GOARCH=s390x, GOARM=, GOOS=linux, PLATFORM_ID=linux-s390x] [2020-04-15T11:31:30.088Z] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=aarch64-linux-gnu-gcc, CXX=aarch64-linux-gnu-g++, GOARCH=arm64, GOARM=, GOOS=linux, PLATFORM_ID=linux-arm64] [2020-04-15T11:31:30.088Z] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, PLATFORM_ID=linux-amd64] [2020-04-15T11:31:30.088Z] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=gcc, CXX=g++, GOARCH=386, GOARM=, GOOS=linux, PLATFORM_ID=linux-386] [2020-04-15T11:31:31.895Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=x86_64-w64-mingw32-gcc, CXX=x86_64-w64-mingw32-g++, GOARCH=amd64, GOARM=, GOOS=windows, PLATFORM_ID=windows-amd64] [2020-04-15T11:31:31.895Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, PLATFORM_ID=linux-amd64] [2020-04-15T11:31:32.179Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=s390x-linux-gnu-gcc, CXX=s390x-linux-gnu-g++, GOARCH=s390x, GOARM=, GOOS=linux, PLATFORM_ID=linux-s390x] [2020-04-15T11:31:32.179Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=o64-clang, CXX=o64-clang++, GOARCH=amd64, GOARM=, GOOS=darwin, PLATFORM_ID=darwin-amd64] [2020-04-15T11:36:53.973Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=386, GOARM=, GOOS=linux, PLATFORM_ID=linux-386] [2020-04-15T11:37:15.932Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=aarch64-linux-gnu-gcc, CXX=aarch64-linux-gnu-g++, GOARCH=arm64, GOARM=, GOOS=linux, PLATFORM_ID=linux-arm64] [2020-04-15T11:37:24.636Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=i686-w64-mingw32-gcc, CXX=i686-w64-mingw32-g++, GOARCH=386, GOARM=, GOOS=windows, PLATFORM_ID=windows-386] [2020-04-15T11:43:27.014Z] >> Building using: cmd='../../packetbeat/build/mage-linux-amd64 golangCrossBuild', env=[CC=i686-w64-mingw32-gcc, CXX=i686-w64-mingw32-g++, GOARCH=386, GOARM=, GOOS=windows, PLATFORM_ID=windows-386] [2020-04-15T11:43:27.014Z] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=gcc, CXX=g++, GOARCH=386, GOARM=, GOOS=linux, PLATFORM_ID=linux-386] [2020-04-15T11:43:27.279Z] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=aarch64-linux-gnu-gcc, CXX=aarch64-linux-gnu-g++, GOARCH=arm64, GOARM=, GOOS=linux, PLATFORM_ID=linux-arm64] [2020-04-15T11:43:27.279Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=i686-w64-mingw32-gcc, CXX=i686-w64-mingw32-g++, GOARCH=386, GOARM=, GOOS=windows, PLATFORM_ID=windows-386] [2020-04-15T11:43:27.539Z] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, PLATFORM_ID=linux-amd64] [2020-04-15T11:43:27.539Z] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=s390x-linux-gnu-gcc, CXX=s390x-linux-gnu-g++, GOARCH=s390x, GOARM=, GOOS=linux, PLATFORM_ID=linux-s390x] [2020-04-15T11:45:49.679Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=x86_64-w64-mingw32-gcc, CXX=x86_64-w64-mingw32-g++, GOARCH=amd64, GOARM=, GOOS=windows, PLATFORM_ID=windows-amd64] [2020-04-15T11:46:02.163Z] >> Building using: cmd='../../packetbeat/build/mage-linux-amd64 golangCrossBuild', env=[CC=x86_64-w64-mingw32-gcc, CXX=x86_64-w64-mingw32-g++, GOARCH=amd64, GOARM=, GOOS=windows, PLATFORM_ID=windows-amd64] [2020-04-15T11:48:24.606Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=o64-clang, CXX=o64-clang++, GOARCH=amd64, GOARM=, GOOS=darwin, PLATFORM_ID=darwin-amd64] [2020-04-15T11:48:24.606Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, PLATFORM_ID=linux-amd64] [2020-04-15T11:48:24.606Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=s390x-linux-gnu-gcc, CXX=s390x-linux-gnu-g++, GOARCH=s390x, GOARM=, GOOS=linux, PLATFORM_ID=linux-s390x] [2020-04-15T11:48:24.606Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=386, GOARM=, GOOS=linux, PLATFORM_ID=linux-386] [2020-04-15T11:52:41.540Z] >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=aarch64-linux-gnu-gcc, CXX=aarch64-linux-gnu-g++, GOARCH=arm64, GOARM=, GOOS=linux, PLATFORM_ID=linux-arm64] [2020-04-15T11:52:49.115Z] >> Building using: cmd='../../packetbeat/build/mage-linux-amd64 golangCrossBuild', env=[CC=s390x-linux-gnu-gcc, CXX=s390x-linux-gnu-g++, GOARCH=s390x, GOARM=, GOOS=linux, PLATFORM_ID=linux-s390x] [2020-04-15T11:52:57.588Z] >> Building using: cmd='../../packetbeat/build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=386, GOARM=, GOOS=linux, PLATFORM_ID=linux-386] [2020-04-15T11:53:02.372Z] >> Building using: cmd='../../packetbeat/build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, PLATFORM_ID=linux-amd64] [2020-04-15T11:57:14.075Z] >> Building using: cmd='../../packetbeat/build/mage-linux-amd64 golangCrossBuild', env=[CC=aarch64-linux-gnu-gcc, CXX=aarch64-linux-gnu-g++, GOARCH=arm64, GOARM=, GOOS=linux, PLATFORM_ID=linux-arm64] [2020-04-15T11:58:15.176Z] >> Building using: cmd='../../packetbeat/build/mage-linux-amd64 golangCrossBuild', env=[CC=o64-clang, CXX=o64-clang++, GOARCH=amd64, GOARM=, GOOS=darwin, PLATFORM_ID=darwin-amd64]

yep, checking the output of one of the old jobs, the PLATFORMS env var it is ignored always build all the packages

https://beats-ci.elastic.co/job/elastic+beats+7.x+multijob-package-linux/125/console

v1v

neat!

.ci/packaging.groovy

andrewkroh

To build one specific architecture you would use PLATFORMS='!defaults +linux/amd64'.

PLATFORMS is a set that implicitly contains defaults where defaults is (linux/amd64, linux/386, windows/amd64, windows/386, and darwin/amd64). When you use + that adds to the set. And when you write any other specificer (e.g. darwin) this acts as a filter to only allow platforms matching darwin. Special sets/keywords are defaults and all.

For example, these are all the same. They only build packages for linux/amd64.

+all linux/amd64 - add all possible platforms, but filter down to only linux/amd64
defaults linux/amd64 - add the defaults then filter down to only linux/amd64
linux/amd64 - defaults is always implicit so this just filters to only include linux/amd64
!defaults +linux/amd64 - remove the defaults (leaving nothing) then add in linux/amd64

Docs: https://godoc.org/github.com/elastic/beats/dev-tools/mage#NewPlatformList

.ci/packaging.groovy

kuisathaverat · 2020-04-16T12:26:17Z

@andrewkroh there is something weird with the windows packages, if I add the windows packages to the PLATFORMS env var they are not generated

export PLATFORMS="!defaults +linux/386 +linux/amd64 +linux/armv7 +linux/ppc64le +linux/s390x +linux/mips64 +windows/386 +windows/amd64"
cd /Users/inifc/src/beats/winlogbeat
mage clean package

>> update:moduleDocs: Collecting module documentation for security, sysmon.
>> Building winlogbeat.yml for windows/amd64
>> Building winlogbeat.reference.yml for windows/amd64
Generated fields.yml for winlogbeat to /Users/inifc/src/beats/winlogbeat/build/fields/fields.all.yml
Generated fields.yml for winlogbeat to /Users/inifc/src/beats/winlogbeat/build/fields/fields.common.yml
Generated fields.yml for winlogbeat to /Users/inifc/src/beats/winlogbeat/build/fields/fields.yml
>> package: Skipping because the platform list is empty
>> Testing package contents
package ran for 6.487589726s

If I do not set the PLATFORMS variable the windows packages are built.

cd /Users/inifc/src/beats/winlogbeat
mage clean package

>> update:moduleDocs: Collecting module documentation for security, sysmon.
>> Building winlogbeat.yml for windows/amd64
>> Building winlogbeat.reference.yml for windows/amd64
Generated fields.yml for winlogbeat to /Users/inifc/src/beats/winlogbeat/build/fields/fields.yml
Generated fields.yml for winlogbeat to /Users/inifc/src/beats/winlogbeat/build/fields/fields.all.yml
Generated fields.yml for winlogbeat to /Users/inifc/src/beats/winlogbeat/build/fields/fields.common.yml
>> golangCrossBuild: Building for windows/amd64
>> golangCrossBuild: Building for windows/386
>> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=i686-w64-mingw32-gcc, CXX=i686-w64-mingw32-g++, GOARCH=386, GOARM=, GOOS=windows, PLATFORM_ID=windows-386]
>> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=x86_64-w64-mingw32-gcc, CXX=x86_64-w64-mingw32-g++, GOARCH=amd64, GOARM=, GOOS=windows, PLATFORM_ID=windows-amd64]
>> package: Building winlogbeat-oss type=zip for platform=windows/386
>> package: Building winlogbeat-oss type=zip for platform=windows/amd64
>> Testing package contents
package ran for 4m38.144573113s

I've checked the filter applied and it is OK https://github.com/elastic/beats/blob/master/winlogbeat/scripts/mage/package.go#L40 BTW it happens with all beat.

I will keep the default and add only the needed

andrewkroh

This looks great. Left a few minor comments.

.ci/packaging.groovy

andrewkroh · 2020-04-16T12:53:34Z

@andrewkroh there is something weird with the windows packages, if I add the windows packages to the PLATFORMS env var they are not generated

I'll look into that. That's not the behavior I expected. One thing that is special about Winlogbeat is that it only builds packages for Windows so it has a built-in filter. Perhaps this is some side-effect of that.

beats/winlogbeat/scripts/mage/package.go

Line 40 in 98255f8

devtools.Platforms = devtools.Platforms.Filter("windows")

andrewkroh

The code change LGTM. I'm good with it if there's sucessful packaging build with it.

feat: build packages in parallel

2a47768

kuisathaverat added enhancement automation ci labels Apr 15, 2020

kuisathaverat requested review from jsoriano, urso, andrewkroh and a team April 15, 2020 10:26

kuisathaverat self-assigned this Apr 15, 2020

urso reviewed Apr 15, 2020

View reviewed changes

v1v approved these changes Apr 15, 2020

View reviewed changes

v1v reviewed Apr 15, 2020

View reviewed changes

.ci/packaging.groovy Outdated Show resolved Hide resolved

kuisathaverat added 3 commits April 15, 2020 14:11

fix: unstable on a stage ignored

866f166

fix: unstable on a stage ignored

eb1a613

fix: make beats in parallel by architecture

2dd22e9

andrewkroh reviewed Apr 15, 2020

View reviewed changes

kuisathaverat added 3 commits April 15, 2020 16:15

fix: use correct PLATFORMS format

009ae72

fix: synstax error

ab7f6c6

fix: method code too large! again

733080e

kuisathaverat force-pushed the parallel-package branch from 304ee6c to 733080e Compare April 15, 2020 16:15

andrewkroh reviewed Apr 15, 2020

View reviewed changes

.ci/packaging.groovy Outdated Show resolved Hide resolved

mdelapenya approved these changes Apr 15, 2020

View reviewed changes

kuisathaverat force-pushed the parallel-package branch from 6a5df4e to 733080e Compare April 15, 2020 18:16

kuisathaverat added 2 commits April 15, 2020 22:05

fix: use error step

1345540

fix: add goos method

d29bbca

cachedout approved these changes Apr 16, 2020

View reviewed changes

kuisathaverat added 3 commits April 16, 2020 10:25

fix: use only one axis

4727387

fix: path for scripts

a9c9863

fix: store artifects

d9902b7

kuisathaverat added 4 commits April 16, 2020 11:50

fix: add linux platforms

d6cd7a6

test: use checkout always

f75bd82

fix: search for packages also on the root build folder

efab4c4

fix: PLATFORMS variable

e6b1617

fix: remove defaults

67ac9c5

andrewkroh reviewed Apr 16, 2020

View reviewed changes

.ci/packaging.groovy Outdated Show resolved Hide resolved

.ci/packaging.groovy Outdated Show resolved Hide resolved

.ci/packaging.groovy Show resolved Hide resolved

kuisathaverat added 2 commits April 16, 2020 16:40

feta: push docker images for the CI use

3aca138

fix: typo

f8995aa

andrewkroh approved these changes Apr 16, 2020

View reviewed changes

kuisathaverat force-pushed the parallel-package branch from bc4342f to cdcd60c Compare April 17, 2020 09:49

fix: script worng

adfa160

kuisathaverat force-pushed the parallel-package branch from cdcd60c to adfa160 Compare April 17, 2020 10:20

kuisathaverat added 2 commits April 17, 2020 13:16

fix: the other way around

f122b1f

list Docker images

8f54245

kuisathaverat merged commit 914e6d3 into elastic:master Apr 17, 2020

kuisathaverat deleted the parallel-package branch April 17, 2020 15:53

jsoriano mentioned this pull request Apr 28, 2020

Backport some features in Jenkinsfile to 7.x #18040

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: build packages in parallel #17723

feat: build packages in parallel #17723

kuisathaverat commented Apr 15, 2020

kuisathaverat commented Apr 15, 2020 •

edited

Loading

urso Apr 15, 2020

kuisathaverat Apr 15, 2020

kuisathaverat Apr 15, 2020 •

edited

Loading

kuisathaverat Apr 15, 2020 •

edited

Loading

v1v left a comment

andrewkroh left a comment •

edited

Loading

kuisathaverat commented Apr 16, 2020

andrewkroh left a comment

andrewkroh commented Apr 16, 2020 •

edited

Loading

andrewkroh left a comment

feat: build packages in parallel #17723

feat: build packages in parallel #17723

Conversation

kuisathaverat commented Apr 15, 2020

What does this PR do?

Why is it important?

kuisathaverat commented Apr 15, 2020 • edited Loading

urso Apr 15, 2020

Choose a reason for hiding this comment

kuisathaverat Apr 15, 2020

Choose a reason for hiding this comment

kuisathaverat Apr 15, 2020 • edited Loading

Choose a reason for hiding this comment

kuisathaverat Apr 15, 2020 • edited Loading

Choose a reason for hiding this comment

v1v left a comment

Choose a reason for hiding this comment

andrewkroh left a comment • edited Loading

Choose a reason for hiding this comment

kuisathaverat commented Apr 16, 2020

andrewkroh left a comment

Choose a reason for hiding this comment

andrewkroh commented Apr 16, 2020 • edited Loading

andrewkroh left a comment

Choose a reason for hiding this comment

kuisathaverat commented Apr 15, 2020 •

edited

Loading

kuisathaverat Apr 15, 2020 •

edited

Loading

kuisathaverat Apr 15, 2020 •

edited

Loading

andrewkroh left a comment •

edited

Loading

andrewkroh commented Apr 16, 2020 •

edited

Loading