dockerBuild for native image doesn't work with remote Docker daemons

[rhuss]

The build fails, to try to create a native binary with dockerBuild true and accessing a remote Docker daemon which does not allow bind mounts via docker run -v as it is done in

quarkus/core/creator/src/main/java/io/quarkus/creator/phase/nativeimage/NativeImagePhase.java

Line 286 in fea6ba9

Collections.addAll(nativeImage, "docker", "run", "-v", outputDir.toAbsolutePath() + ":/project:z", "--rm", image);

This use is important when e.g. building against minikube/minishift's internal Docker daemon (eval $(minikube docker-env)) so that an image does not need to be pushed to a registry but could be directly used within minikube.

For this setup, in Syndesis we avoided bind mounts but used a combination of running the actual build during a Docker build and then copying out the generated binary from the created image by running a container with cat:

The actual build is like in

cd $operator_dir
docker build -t syndesis-operator-builder . -f Dockerfile-builder
docker run syndesis-operator-builder cat /syndesis-operator > syndesis-operator
chmod a+x syndesis-operator

with that Dockerfile

FROM golang:1.11.0
RUN go get -u github.com/golang/dep/cmd/dep
WORKDIR /go/src/github.com/syndesisio/syndesis/install/operator
COPY Gopkg.toml .
COPY Gopkg.lock .
RUN dep ensure -vendor-only -v
COPY . .
RUN CGO_ENABLED=0 go build -o /syndesis-operator ./cmd/syndesis-operator

This might not be useful in this context as it depends on the size of the source to copy over into the image when doing the build.

As an alternative, we used ssh to copy over the sources into the Minishift VM and then used a bind mount within the VM, but the current solution is (a) more generally applicable and (b) also more robust.

[rohanKanojia]

I was facing this problem too while using minikube's docker deamon for quarkus.

@cescoffier @geoand @Sanne : Polite ping, Could you guys please look into this issue whenever you get time? We're trying to integrate quarkus support into fabric8 magen plugin and facing issues due to this.

[geoand]

I could look into this on my own time (which probably means over the weekends).
@rohanKanojia @rhuss To get a little more context, could you explain a little more how you envision the fmp integration would work? Which fmp goals are we talking about?

Do you envision that some kind of flag that would enable the behavior you describe you use in Syndesis should control the Quarkus native image generation phase?

[rohanKanojia]

@geoand : Thanks for super fast response, (well I think it's a pre-requisite for being a quarkus guy ;-) )

we have added a quarkus generator and quarkus health check enricher in fabric8io/fabric8-maven-plugin#1577 , so I was just testing this feature but I faced issues when during native build as it requires local docker deamon to be exposed during native build.

If I build usually with minikube's docker deamon exposed. it fails with this error: https://pastebin.com/QYc3PViX

[geoand]

@geoand : Thanks for super fast response, (well I think it's a pre-requisite for being a quarkus guy ;-) )

LOL, although I'm a Spring Boot guy :P

we have added a quarkus generator and quarkus health check enricher in fabric8io/fabric8-maven-plugin#1577 , so I was just testing this feature but I faced issues when during native build as it requires local docker deamon to be exposed during native build.

If I build usually with minikube's docker deamon exposed. it fails with this error: https://pastebin.com/QYc3PViX

Ah yes, there is also a relevant Stack Overflow question here.

So the idea is to be able to seemlessly use the docker daemon of Minishift / Minikube to do the build... I would think it's definitely doable, but I would like more details for @rhuss :)

[geoand]

@rhuss I think the docker tarball context would be applicable here, WDYT?

[rhuss]

Well, not really a its about getting access to the created binary. I think, the most elegant way would be to use multistage Docker builds (i.e. running the native build + creating the final image with one 'docker build') is the way instead of storing the (linux) binary on your local FS as intermediate step.

Unfortunately Minshift's Docker daemon is too old to support multistage builds, but if we stick to minikube or more modern Docker daemon that's by far the best solution (not only for remote Docker daemons but in general).

[geoand]

Well, not really a its about getting access to the created binary. I think, the most elegant way would be to use multistage Docker builds (i.e. running the native build + creating the final image with one 'docker build') is the way instead of storing the (linux) binary on your local FS as intermediate step.

From the docs I thought that the tarball context could be used to "bundle" the created binary as well.

Unfortunately Minshift's Docker daemon is too old to support multistage builds, but if we stick to minikube or more modern Docker daemon that's by far the best solution (not only for remote Docker daemons but in general).

I'll hopefully have a look over the weekend. Thanks!

[geoand]

@rohanKanojia hey, wasn't able to look at this over the weekend unfortunately...

I saw however that FMP was released, did you work around this limitation or just ignore the issue for the time being (since it's not strictly necessary?

[rohanKanojia]

@geoand : No, no. Issue is still there. We just released with whatever we had.

[geoand]

@rohanKanojia thanks for the info.

[geoand]

I read through this again, but I still don't understand how building the native image inside a Minishift VM would help... The reason I say this is because even if we do accomplish that, there will be no usable docker image ready for consumption by users of the cluster (the image that the users would use is something like: https://github.com/quarkusio/quarkus/blob/master/devtools/common/src/main/resources/templates/dockerfile-native.ftl#L18).

Am I missing something here @rhuss @rohanKanojia ?

[rhuss]

The main issue with remote Docker daemons is, that you can't use volume mount in any case (the -v option to Docker run). So we need either another way to share the generated binary between the build process creating the native Linux binary (within a container) to the final application image (another container).

So the best answer for this is to combine the process of creating the application image and the actual compilation into one Dockerfile.

FROM graalvm as build
COPY src/* workdir
// create binary image
.....

FROM fedora
COPY --from build generated-binary image /target-dir/
CMD /target-dir/generated-binary

[geoand]

@rhuss For me your comment:

So the best answer for this is to combine the process of creating the application image and the actual compilation into one Dockerfile.

makes absolute sense :). We'll have to see how to best handle this use case.

Thanks for the details!

[geoand]

I looked into this and it should be rather easy to implement on the technical side of things using a multi-stage docker builds as @rhuss mentioned (basically we would trigger this "new" type of docker build instead of the regular build we do here).

The first stage of the docker build would just copy the runner jar and dependencies lib and invoke the native binary generation.
The second stage would just copy the native binary output from the first stage and build a docker image with only the native binary.

What I am struggling with is how to reconcile this new approach with what we have already - what should the UX be?
Any ideas @rhuss @cescoffier @rohanKanojia ?

[rsvoboda]

Quarkus Quickstarts have an example of combined approach - https://github.com/quarkusio/quarkus-quickstarts/blob/master/getting-started-knative/Dockerfile

building the app ++ building the native image ++ building the final Docker image

building the native image is done directly via native-image command
https://github.com/quarkusio/quarkus-images/tree/graalvm-1.0.0-rc16/centos-quarkus-maven could be used (as it has have both mvn and graalvm) for native build via mvn -Pnative verify