Golang 1.14 panics on some Linux kernel versions, causing Kaniko to crash

[dani29]

Actual behavior
I ran into this issue while debugging #1162, trying to build a docker image on my machine.
Kaniko crashed with a panic coming directly from Golang's runtime, emitting the following error message and a stacktrace:

runtime: mlock of signal stack failed: 12
runtime: increase the mlock limit (ulimit -l) or
runtime: update your kernel to 5.3.15+, 5.4.2+, or 5.5+
fatal error: mlock failed
...

Digging a bit further, the short story is that some Linux kernel v5.2 had a memory corruption bug; Golang 1.13 was susceptible to silently suffering from it, and in Golang 1.14 the runtime crashes once it detects it runs on a buggy kernel.

This issue, was fixed Golang 1.14.1, which was released to Docker Hub on March 19, 2020. We have built the latest version of Kaniko on March 18th, so we currently run with the runtime before the fix.

The resolution is pretty simple - rebuilding Kaniko should fix it, as the current image we're using will pick up the latest Golang version (1.14.2).

The next release of Kaniko (v0.20.0) should automatically fix the issue, so this can stay here as a marker in case anyone else runs into the problem. If some users are actually blocked and we are not ready to release, perhaps it's worth issuing v0.19.1 patch.

To Reproduce
Steps to reproduce the behavior:
Build with kaniko on a Linux running specific kernel versions: 5.2, 5.3<15, 5.4.1
Run $ uname -a to find out which version you run.

Additional Information
Dockerfile:

FROM python:3.7.4-slim AS service-base
RUN apt-get update \
  && apt-get --assume-yes install curl gpg  libcurl4-openssl-dev libssl-dev gcc git build-essential netcat libsnappy-dev

RUN pip install poetry=="1.0.2"
COPY ./_packages /api/_packages
COPY ./config /config

WORKDIR /service/cat
COPY . /service/cat
RUN rm -rf _packages
RUN rm -rf config

RUN \
  poetry config virtualenvs.create false && \
  poetry install

Context (only the contents of the pyproject.toml matter)

/_packages
    foo.txt
/config
    foo.txt
pyproject.toml

pyproject.toml

[tool.poetry]
name = "poetry-demo"
version = "0.1.0"
description = ""
authors = ["Sébastien Eustace <sebastien@eustace.io>"]

[tool.poetry.dependencies]
python = "*"

[tool.poetry.dev-dependencies]
pytest = "^3.4"

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag	- []
Please check if your dockerfile is a multistage dockerfile

[iosifnicolae2]

We are getting the same error:

INFO[0044] Unpacking rootfs as cmd RUN apk add --no-cache --update bash git && rm -rf /var/cache/apk/* requires it.
--
Sat, May 14 2022 1:20:30 pm | runtime: mlock of signal stack failed: 12
Sat, May 14 2022 1:20:30 pm | runtime: increase the mlock limit (ulimit -l) or
Sat, May 14 2022 1:20:30 pm | runtime: update your kernel to 5.3.15+, 5.4.2+, or 5.5+
Sat, May 14 2022 1:20:30 pm | fatal error: mlock failed

• error log

Any solution on how to solve this?

Thank you!

---

k3s v1.21.5+k3s2

uname -a
Linux dedicated-worker-1 5.4.0-107-generic #121-Ubuntu SMP Thu Mar 24 16:04:27 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

on host

ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 514705
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 514705
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

in container (in the meantime i've increased the number of open files on host):

ulimit -a
core file size (blocks)         (-c) unlimited
data seg size (kb)              (-d) unlimited
scheduling priority             (-e) 0
file size (blocks)              (-f) unlimited
pending signals                 (-i) 514705
max locked memory (kb)          (-l) 64
max memory size (kb)            (-m) unlimited
open files                      (-n) 1048576
POSIX message queues (bytes)    (-q) 819200
real-time priority              (-r) 0
stack size (kb)                 (-s) 8192
cpu time (seconds)              (-t) unlimited
max user processes              (-u) unlimited
virtual memory (kb)             (-v) unlimited
file locks                      (-x) unlimited

host baseline kernel version:

cat /proc/version_signature
Ubuntu 5.4.0-110.124-generic 
baseline kernel version: 5.4.181

Most likely there is a link with this issue: https://github.com/golang/go/wiki/LinuxKernelSignalVectorBug