[WIP] Multi-stage (& multi-arch) dockerfile #1615

sapk · 2017-04-26T15:37:58Z

Implement multi-stage dockerfile. This will permit cleaner build with the ability to configure build params and maybe if FROM are multi-arch to have only one dockerfile.

After that someone could simply do docker build . to build the image locally on every platform. (Not yet ready)

sapk · 2017-04-28T16:50:47Z

This will need at least docker/moby@v17.05.0-ce-rc2 for multi-stage to work. So this PR is a little in advance. This will dependency is only for building multi-stage. The image resulting will be able to run even with older docker/moby version.

bkcsoft · 2017-04-28T18:30:25Z

Dockerfile

@@ -1,33 +1,38 @@
-FROM alpine:3.4
-MAINTAINER Thomas Boerger <thomas@webhippie.de>


Why remove the Maintainer? Add yourself instead

It's only needed for the published image (last FROM)

And I use the LABEL instruction since MAINTAINER is depraced.

bkcsoft · 2017-04-28T18:30:47Z

Dockerfile

+#FROM webhippie/golang:${GO_VERSION} AS build-env
+FROM sapk/alpine-multi AS build-env
+
+ARG TAGS


Maybe default to sqlite?

bkcsoft · 2017-04-28T18:32:07Z

Dockerfile

+ENV PATH "${PATH}:${GOPATH}/bin"
+
+RUN apk -U --no-cache add go git build-base \
+ && go get -u github.com/jteeuwen/go-bindata/...


No need to go get go-bindata since it will be fetched by make generate

I have re-tested and you are right. I remember that one time it ask for it but don't remember why.

I think that it was because git wasn't installed so it couldn't be installed and failed after.

bkcsoft · 2017-04-28T18:32:25Z

Dockerfile

+ADD . ${GOPATH}/src/code.gitea.io/gitea
+WORKDIR ${GOPATH}/src/code.gitea.io/gitea
+
+RUN pwd && ls -lah && make clean generate build


Why pwd && ls -lah ?

bkcsoft · 2017-04-28T18:33:11Z

.dockerignore

@@ -1,5 +1,16 @@
 *
-!gitea


this should def. be here if we're building it inside the container now

Heck, all there things you've added in here should still go into the first build...

At startup docker "clone" local folder to build folder. This limit what is copied into workspace.

After re-thinking if we add all the repo (remove .dockerignore) we could be able to ask to compile a specific version (tag) at build stage via git.

bkcsoft · 2017-04-28T18:37:27Z

Dockerfile

+    su-exec ca-certificates sqlite bash git linux-pam s6 curl openssh tzdata \
+ && addgroup -S -g ${GID} git \
+ && adduser -S -H -D -h /data/git -s /bin/bash -u ${UID} -G git git \
+ && echo "git:$(date +%s | sha256sum | base64 | head -c 32)" | chpasswd


echo "git:$(dd if=/dev/urandom bs=24 count=1 | base64)" | chpasswd (24 byte => base64 => 32 byte)

I haven't touch that.

right, it was more of a "since you're already screwing with it, might as well fix this"

bkcsoft · 2017-04-28T18:38:30Z

Dockerfile

-    openssh \
-    tzdata && \
-  rm -rf \
-    /var/cache/apk/* && \


We should still remove the cache...

I can but with --no-cache it shouldn't be needed

Erm no, completely different things... Don't remove this line...

https://github.com/gliderlabs/docker-alpine/blob/master/docs/usage.md#disabling-cache It seem, it doesn't even need -U also.

I have checked after apk --no-cache -> /var/cache/apk is empty.

ooh, apk --no-cache, I thought you meant docker build --no-cache 😂

Then it's fine 👍

bkcsoft · 2017-04-28T18:40:16Z

Makefile

@@ -99,11 +99,14 @@ build: $(EXECUTABLE)

 $(EXECUTABLE): $(SOURCES)
 	go build $(GOFLAGS) $(EXTRA_GOFLAGS) -tags '$(TAGS)' -ldflags '-s -w $(LDFLAGS)' -o $@
+
+.PHONY: docker-build
+docker-build:


Is this stage still required?

I have separate it but still keep it if someone want to build binaries via docker.

bkcsoft · 2017-04-28T18:45:34Z

I added status/blocked since this is blocked on build-server getting Docker 17.05 stable 😉

sapk · 2017-04-28T23:27:12Z

To test against docker version in master : http://blog.alexellis.io/mutli-stage-docker-builds/#howcanitryitout

bkcsoft · 2017-04-28T23:30:20Z

Makefile

@@ -105,8 +105,9 @@ docker-build:
 	docker run -ti --rm -v $(CURDIR):/srv/app/src/code.gitea.io/gitea -w /srv/app/src/code.gitea.io/gitea -e TAGS="bindata $(TAGS)" webhippie/golang:edge make clean generate build

 .PHONY: docker
+docker: TAGS=sqlite


no, ARG TAGS="sqlite" in the Dockerfile...

sapk · 2017-04-29T00:27:42Z

At this stage, Dockerfile should be buildable on platform amd64, arm, aarch64 simply by tapping docker build .

I am figuring out how to build them in drone and do multi-arch image.

sapk · 2017-04-29T01:33:01Z

Dockerfile

+    su-exec ca-certificates sqlite bash git linux-pam s6 curl openssh tzdata \
+ && addgroup -S -g ${GID} git \
+ && adduser -S -H -D -h /data/git -s /bin/bash -u ${UID} -G git git \
+ && echo "git:$(dd if=/dev/urandom bs=24 count=1 | base64)" | chpasswd

 ENV USER git


Reminder to myself : Check if this is usefull since this still run with root after.

bkcsoft · 2017-04-29T10:25:46Z

@sapk There's no way to build this on the drone server until it's upgraded to 17.05 🙂

sapk · 2017-04-29T11:42:37Z

@bkcsoft we could by using the method explain in http://blog.alexellis.io/mutli-stage-docker-builds/#howcanitryitout but this would make use build last version of docker and start a container with this version in it and build this Dockerfile in it. Not really efficient.

We could also do the same without docker building step and re-use release tar.gz

ulm0 · 2017-04-29T12:43:47Z

I am figuring out how to build them in drone and do multi-arch image.

@sapk I know it's not matter of this PR, but it'd be great if you make a HOW-TO on making a multi arch image, i'm interested in doing so as well.

sapk · 2017-04-29T13:17:32Z

@klud1 a multi-arch image is just a fake image linking to various image corresponding for arch (documented in files like https://github.com/sapk/dockerfiles/tree/master/alpine-multi) that you push with github.com/estesp/manifest-tool (you can see command-line example in https://github.com/sapk/dockerfiles/blob/master/.travis.yml that update the refs).

And docker pull (like https://hub.docker.com/r/sapk/alpine-multi/tags/) it will choose the corresponding hash/image to pull for his running arch.

So you build a image for each arch and reference them in a .yml

For the multi-arch part, I use https://github.com/portainer/docker-manifest as a reference and use multiarch/alpine for a good base to have image on most arch.

So with my alpine base multi-arch use as base, if you build this Dockerfile, it will/should result in a working on you current arch image.

ulm0 · 2017-04-29T13:41:11Z

@sapk amazing, thank you so much.

sapk · 2017-04-30T18:04:29Z

I tested to do a docker build on an armhf platform of the multi-stage image with multi-arch sapk/alpine-multi as base and this work seamlessly. \o/

I think that it would be more clean to push this change under Dockerfile.multi and keep the olds ones around until the needed version for docker is ready if is is ok with you.

bkcsoft

otherwise LG-TM

bkcsoft · 2017-05-01T16:32:20Z

docker/manifest/gitea-1-0-0.yml

@@ -0,0 +1,14 @@
+image: sapk/gitea:1.0.0


What are all these manifests for?

For generating multi-arch image that reference the image to corresponding arch.

The only part missing is generation of image on arch. This could be done via qemu like https://github.com/multiarch/alpine does.

appleboy · 2017-05-15T14:09:36Z

We need to waiting for this PR drone-plugins/drone-docker#124 for Drone to support multi-stage build.

sapk · 2017-05-16T09:40:09Z

Ok so for a little update.

I remove olds Dockerfile.

The task make docker will build gitea/gitea for the platform you are running on (compatible with amd64, arm, arm64). (For this to fully work the multi-arch base gitea/gitea-base need to be set on docker hub by someone with rights. DOCKER_MANIFEST=docker/manifest/base.yml make docker-multi-update-manifest )

The tasks make docker-multi-* permit to cross-build and do multi-arch image. I don't think this is possible yet with drone-plugins/drone-docker.

sapk · 2017-05-16T09:55:42Z

It's also possible to choose the version gitea we want in the container via variable GITEA_VERSION. It accept any ref that is accepted by git checkout

sapk · 2017-06-16T13:11:42Z

I will break this into multiple PR to ease the process of validation.

Unify dockerfile with one multi-arch as base. (permit to build the same dockerfile on every platform)
Multi-stage (permit to have a stage build to limit image size)
Multi-arch final image (permit to download from docker hub the corresponding image of your running arch with the same name for every platform)

lunny added this to the 1.3.0 milestone Apr 27, 2017

lunny added the type/enhancement An improvement of existing functionality label Apr 27, 2017

cez81 mentioned this pull request Apr 28, 2017

Refactor Dockerfile #1632

Merged

sapk changed the title ~~[WIP] Multi-stage dockerfile~~ [WIP] Multi-stage (& multi-arch) dockerfile Apr 28, 2017

tboerger added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Apr 28, 2017

bkcsoft suggested changes Apr 28, 2017

View reviewed changes

sapk commented Apr 29, 2017

View reviewed changes

bkcsoft approved these changes Apr 29, 2017

View reviewed changes

bkcsoft reviewed May 1, 2017

View reviewed changes

sapk force-pushed the multi-stage-docker branch 5 times, most recently from ae0f493 to 599056b Compare May 2, 2017 17:31

sapk added 3 commits May 16, 2017 01:21

Add docker-multi-update-all and use GITEA_VERSION

09da0ea

Allow more flexible config

276ab45

fix and improve

a242fb5

sapk force-pushed the multi-stage-docker branch from 599056b to a242fb5 Compare May 15, 2017 23:22

sapk added 9 commits May 16, 2017 01:35

De-duplicate

617234d

Add DOCKER_PUSHIMAGE config env to docker-multi-update-manifest

942da85

Use docker login folder

e4a7fae

REmove useless "

c8de80a

Use vX.X.X like git tags

f374195

set docker-multi-push under env params

6299ecd

some fixes

da400ba

Use gitea/gitea- to have allways format gitea/gitea*

26a669b

Use gitea/gitea- to have allways format gitea/gitea* as base

42f6b25

appleboy mentioned this pull request May 16, 2017

feat: upgrade drone docker image to support multi-stage build. #1732

Merged

sapk added 6 commits May 16, 2017 11:30

Delete Dockerfile.aarch64

3d726d7

Delete Dockerfile

3401ee0

Delete Dockerfile.rpi

6e672a8

Rename Dockerfile.multi to Dockerfile

1b98be6

Update Makefile

f1738a7

Update Makefile

57f68e9

Update Makefile

d2f9ae8

Enforce netgo at build stage

3fc4708

sapk mentioned this pull request Jun 16, 2017

Docker multi-arch base #1985

Merged

sapk closed this Jun 16, 2017

lunny removed this from the 1.3.0 milestone Jun 17, 2017

sapk mentioned this pull request Mar 12, 2018

Enable Multiarchitecture build for Docker Container #3633

Closed

go-gitea locked and limited conversation to collaborators Nov 23, 2020

		@@ -1,33 +1,38 @@
		FROM alpine:3.4
		MAINTAINER Thomas Boerger <thomas@webhippie.de>

[WIP] Multi-stage (& multi-arch) dockerfile #1615

[WIP] Multi-stage (& multi-arch) dockerfile #1615

Conversation

sapk commented Apr 26, 2017 • edited Loading

sapk commented Apr 28, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sapk Apr 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sapk Apr 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bkcsoft Apr 29, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bkcsoft commented Apr 28, 2017

sapk commented Apr 28, 2017

Choose a reason for hiding this comment

sapk commented Apr 29, 2017

Choose a reason for hiding this comment

bkcsoft commented Apr 29, 2017

sapk commented Apr 29, 2017

ulm0 commented Apr 29, 2017

sapk commented Apr 29, 2017 • edited Loading

ulm0 commented Apr 29, 2017

sapk commented Apr 30, 2017 • edited Loading

bkcsoft left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

appleboy commented May 15, 2017

sapk commented May 16, 2017

sapk commented May 16, 2017

sapk commented Jun 16, 2017

sapk commented Apr 26, 2017 •

edited

Loading

sapk Apr 28, 2017 •

edited

Loading

sapk Apr 28, 2017 •

edited

Loading

bkcsoft Apr 29, 2017 •

edited

Loading

sapk commented Apr 29, 2017 •

edited

Loading

sapk commented Apr 30, 2017 •

edited

Loading