Cross build image is not working

[marthydavid]

I've tried to deploy the fresh new grafana/loki:v0.2.0 and grafana/promtail:v0.2.0 images via the loki-stack helm chart. Also tested the master tagged image. But it's not able to run, as far as i know because the binary is not arm32/armv7/armhf architecture

I'm running it on a 2 node Raspberry Pi 4 "cluster".

pi@master01:~ $ uname -r
4.19.64-v7l+

pi@master01:~ $ docker version
Client: Docker Engine - Community
 Version:           19.03.1
 API version:       1.40
 Go version:        go1.12.5
 Git commit:        74b1e89
 Built:             Thu Jul 25 21:33:17 2019
 OS/Arch:           linux/arm
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.1
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.5
  Git commit:       74b1e89
  Built:            Thu Jul 25 21:27:09 2019
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

pi@master01:~ $ kubectl get nodes -owide
NAME                           STATUS   ROLES    AGE    VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION   CONTAINER-RUNTIME
master01.bla.xyz   Ready    master   3d2h   v1.15.2   1.1.1.1   <none>        Raspbian GNU/Linux 10 (buster)   4.19.64-v7l+     docker://19.3.1
node01.bla.xyz     Ready    <none>   3d2h   v1.15.2   1.1.1.2   <none>        Raspbian GNU/Linux 10 (buster)   4.19.64-v7l+     docker://19.3.1
pi@master01:~ $

Hostname and ip-s are mocked.

pi@master01:~ $ kubectl version
vClient Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:23:26Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/arm"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/arm"}
pi@master01:~ $ helm version
Client: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
pi@master01:~ $

pi@master01:~ $ helm list
NAME               	REVISION	UPDATED                 	STATUS  	CHART                     	APP VERSION	NAMESPACE
cert-chart         	2       	Fri Aug  9 18:57:10 2019	DEPLOYED	cert-manager-v0.9.0       	v0.9.0     	cert-manager
loki               	2       	Mon Aug 12 19:31:20 2019	DEPLOYED	loki-stack-0.15.0         	0.0.1      	monitoring
prometheus-operator	3       	Sat Aug 10 13:24:55 2019	DEPLOYED	prometheus-operator-5.15.0	0.31.1     	monitoring
pi@master01:~ $

Last try:

pi@master01:~ $ helm get values loki
loki:
  enabled: true
  image:
    tag: master
  serviceMonitor:
    enabled: true
promtail:
  enabled: true
  image:
    tag: master
  serviceMonitor:
    enabled: true

Status of these pods:

pi@master01:~ $ kubectl get pods  -n monitoring -l release=loki
NAME                  READY   STATUS             RESTARTS   AGE
loki-0                0/1     CrashLoopBackOff   22         91m
loki-promtail-77fc9   0/1     CrashLoopBackOff   22         92m
loki-promtail-x6mjf   0/1     CrashLoopBackOff   22         92m

Logs from pods:

pi@master01:~ $ kubectl -n monitoring logs -l app=loki
standard_init_linux.go:211: exec user process caused "exec format error"

pi@master01:~ $ kubectl -n monitoring logs -l app=promtail
standard_init_linux.go:211: exec user process caused "exec format error"
standard_init_linux.go:211: exec user process caused "exec format error"

[sh0rez]

Hi, I was able to reproduce this and yes, while the images are built for the correct architecture, the included binary is indeed wrong.

This was due to a tiny ordering mistake in the Dockerfile, which should be resolved by #886.

Please try master once this is merged.

[marthydavid]

@sh0rez Build is failed for promtail arm64
https://circleci.com/gh/grafana/loki/9388

[marthydavid]

@sh0rez.
I've managed to build the image on my mac with the following commands from 600b6c

docker buildx build --platform linux/amd64,linux/arm64,linux/arm/v7  --build-arg grafana/loki-build-image:0.5.0  -t marthydavid/promtail:master-600b6cf -f cmd/promtail/Dockerfile  . --push

pi@master01:~ $ helm get values loki
loki:
  enabled: true
  image:
    tag: master-e4b99f27
  serviceMonitor:
    enabled: true
promtail:
  enabled: true
  image:
    repository: marthydavid/promtail
    tag: master-600b6cf
  serviceMonitor:
    enabled: true

pi@master01:~ $ kubectl get pods -n monitoring -l release=loki -o jsonpath="{..image}" |tr -s '[[:space:]]' '\n' |sort |uniq -c
      2 grafana/loki:master-e4b99f27
      4 marthydavid/promtail:master-600b6cf