Graylog fails to run after updating docker image from 4.3.3 to 4.3.4

[MahdiGhiasi]

We've updated our Graylog docker image to latest version pushed on graylog/graylog:4.3 (which is 4.3.4), and since then the graylog container fails to start.

Inspecting docker logs shows that this seems to be the issue:

graylog_1        | adding environment opts
graylog_1        | mkdir: cannot create directory ‘/data’: Permission denied

Our elastic and mongo are running fine, but graylog container fails to start with the error message above.

We have two separate instances of Graylog (not linked, used separately by separate teams), and both have encountered this issue after upgrading. They were both working fine on v4.3.3.

---

We use a folder-mounted journal folder in our docker-compose file:

 graylog:
    image: graylog/graylog:4.3
    restart: always
    volumes:
      - /root/docker-data/graylog/graylog_journal:/usr/share/graylog/data/journal

Update (by @mpfz0r):

This only happens on docker versions older than 20.10.10 https://docs.docker.com/engine/release-notes/#201010
Because their default seccomp policy does not support clone3 syscalls.

[mpfz0r]

@MahdiGhiasi
could you run your graylog in docker-compose with a debugging entry point for us and give us the output?
entrypoint: "/bin/bash -c 'find /usr/share/graylog -type d -ls'"

[mpfz0r]

This seems to be where this happens:
But I don't see why there is a regression between 4.3.3 and 4.3.4

https://github.com/Graylog2/graylog-docker/blob/4.3/docker-entrypoint.sh#L91-L94

  # Create data directories
  for d in journal log plugin config contentpacks
  do
    dir=${GRAYLOG_HOME}/data/${d}
    [[ -d "${dir}" ]] || mkdir -p "${dir}"

    if [[ "$(stat --format='%U:%G' $dir)" != 'graylog:graylog' ]] && [[ -w "$dir" ]]; then
      chown -R graylog:graylog "$dir" || echo "Warning can not change owner to graylog:graylog"
    fi
  done
}

[coffee-squirrel]

We're currently running 2 environments on the 4.3.4 images (1 OSS, 1 Enterprise) and haven't ran into this yet.

mkdir: cannot create directory ‘/data’ almost makes it seem like ${GRAYLOG_HOME} isn't set. So far I've only seen that type of message with something like mkdir -p /data/foo (or /data, of course).

[MahdiGhiasi]

@mpfz0r Here's the output for the debugging entry point you requested:

graylog_1        |   1317781      4 drwxr-xr-x   8 graylog  graylog      4096 Aug  3 19:02 /usr/share/graylog
graylog_1        |   1317784      4 drwxr-xr-x   2 graylog  graylog      4096 Aug  3 19:02 /usr/share/graylog/bin
graylog_1        |   1317797      4 drwxr-xr-x   2 graylog  graylog      4096 Aug  3 19:02 /usr/share/graylog/plugin
graylog_1        |   1317805      4 drwxr-xr-x   2 graylog  graylog      4096 Aug  3 19:02 /usr/share/graylog/plugins-merged
graylog_1        |    527453      4 drwxr-x---   7 graylog  graylog      4096 Aug  4 17:26 /usr/share/graylog/data
graylog_1        |    527460      4 drwxr-x---   2 graylog  graylog      4096 Aug  3 19:02 /usr/share/graylog/data/plugin
graylog_1        |   1703937      4 drwxrwxrwx   3 graylog  graylog      4096 Aug  4 04:20 /usr/share/graylog/data/journal
graylog_1        |   1703938      4 drwxr-xr-x   2 graylog  graylog      4096 Aug  4 04:01 /usr/share/graylog/data/journal/messagejournal-0
graylog_1        |    527457      4 drwxr-x---   2 graylog  graylog      4096 Aug  3 19:02 /usr/share/graylog/data/data
graylog_1        |    527459      4 drwxr-x---   2 graylog  graylog      4096 Aug  3 19:02 /usr/share/graylog/data/log
graylog_1        |    527454      4 drwxr-x---   2 graylog  graylog      4096 Aug  4 17:26 /usr/share/graylog/data/config
graylog_1        |   1317796      4 drwxr-xr-x   2 graylog  graylog      4096 Aug  3 18:01 /usr/share/graylog/log
graylog_1        |   1317798      4 drwxr-xr-x   2 graylog  graylog      4096 Aug  3 19:02 /usr/share/graylog/plugins-default

[MahdiGhiasi]

I've also confirmed again that docker image graylog/graylog:4.3.3 can start properly but graylog/graylog:4.3.4 gives the error I described, in our environment (nothing is changed in our environment other than the docker image tag).

I've rolled back our production Graylog instance to 4.3.3 for now and it runs properly; but I'm happy to do any tests or give any details that might help finding the root cause of this in 4.3.4 on our environment.

[mpfz0r]

@MahdiGhiasi Thanks.

Could you run entrypoint: "/bin/bash -xv /docker-entrypoint.sh"
for me?
I think @coffee-squirrel might be right. the /data error output is odd

[MahdiGhiasi]

@mpfz0r There you go:

graylog_1        | + grep -q UseConcMarkSweepGC
graylog_1        | + /opt/java/openjdk/bin/java -XX:+PrintFlagsFinal
graylog_1        | adding environment opts
graylog_1        |
graylog_1        | # and add the previous saved settings to our defaults
graylog_1        | if [[ ! -z ${__GRAYLOG_SERVER_JAVA_OPTS} ]]
graylog_1        | then
graylog_1        |   echo "adding environment opts"
graylog_1        |   GRAYLOG_SERVER_JAVA_OPTS="${GRAYLOG_SERVER_JAVA_OPTS} ${__GRAYLOG_SERVER_JAVA_OPTS}"
graylog_1        |   export GRAYLOG_SERVER_JAVA_OPTS
graylog_1        | fi
graylog_1        | + [[ ! -z -Xms256m -Xmx256m ]]
graylog_1        | + echo 'adding environment opts'
graylog_1        | + GRAYLOG_SERVER_JAVA_OPTS='-Xms256m -Xmx256m -Xms256m -Xmx256m'
graylog_1        | + export GRAYLOG_SERVER_JAVA_OPTS
graylog_1        |
graylog_1        | # Convert all environment variables with names ending in __FILE into the content of
graylog_1        | # the file that they point at and use the name without the trailing __FILE.
graylog_1        | # This can be used to carry in Docker secrets.
graylog_1        | for VAR_NAME in $(env | grep '^GRAYLOG_[^=]\+__FILE=.\+' | sed -r 's/^(GRAYLOG_[^=]*)__FILE=.*/\1/g'); do
graylog_1        |   VAR_NAME_FILE="${VAR_NAME}__FILE"
graylog_1        |   if [ "${!VAR_NAME}" ]; then
graylog_1        |     echo >&2 "ERROR: Both ${VAR_NAME} and ${VAR_NAME_FILE} are set but are exclusive"
graylog_1        |     exit 1
graylog_1        |   fi
graylog_1        |   VAR_FILENAME="${!VAR_NAME_FILE}"
graylog_1        |   echo "Getting secret ${VAR_NAME} from ${VAR_FILENAME}"
graylog_1        |   if [ ! -r "${VAR_FILENAME}" ]; then
graylog_1        |     echo >&2 "ERROR: ${VAR_FILENAME} does not exist or is not readable"
graylog_1        |     exit 1
graylog_1        |   fi
graylog_1        |   export "${VAR_NAME}"="$(< "${VAR_FILENAME}")"
graylog_1        |   unset "${VAR_NAME_FILE}"
graylog_1        | done
graylog_1        | ++ env
graylog_1        | ++ sed -r 's/^(GRAYLOG_[^=]*)__FILE=.*/\1/g'
graylog_1        | ++ grep '^GRAYLOG_[^=]\+__FILE=.\+'
graylog_1        |
graylog_1        |
graylog_1        | # Delete outdated PID file
graylog_1        | [[ -e /tmp/graylog.pid ]] && rm --force /tmp/graylog.pid
graylog_1        | + [[ -e /tmp/graylog.pid ]]
graylog_1        |
graylog_1        | # check if we are inside kubernetes, Graylog should be run as statefulset and $POD_NAME env var should be defined like this
graylog_1        | #          env:
graylog_1        | #          - name: POD_NAME
graylog_1        | #            valueFrom:
graylog_1        | #              fieldRef:
graylog_1        | #                fieldPath: metadata.name
graylog_1        | # First stateful member is having pod name ended with -0, so
graylog_1        | if [[ ! -z "${POD_NAME}" ]]
graylog_1        | then
graylog_1        |  if echo "${POD_NAME}" | grep "\\-0$" >/dev/null
graylog_1        |  then
graylog_1        |    export GRAYLOG_IS_MASTER="true"
graylog_1        |  else
graylog_1        |    export GRAYLOG_IS_MASTER="false"
graylog_1        |  fi
graylog_1        | fi
graylog_1        | + [[ ! -z '' ]]
graylog_1        |
graylog_1        | # check if we are inside a nomad cluster
graylog_1        | # First member is having alloc-index 0, so
graylog_1        | if [[ ! -z "${NOMAD_ALLOC_INDEX}" ]]; then
graylog_1        |   if [ ${NOMAD_ALLOC_INDEX} == 0 ]; then
graylog_1        |     export GRAYLOG_IS_MASTER="true"
graylog_1        |   else
graylog_1        |     export GRAYLOG_IS_MASTER="false"
graylog_1        |   fi
graylog_1        | fi
graylog_1        | + [[ ! -z '' ]]
graylog_1        |
graylog_1        | # Merge plugin dirs to allow mounting of /plugin as a volume
graylog_1        | export GRAYLOG_PLUGIN_DIR=/usr/share/graylog/plugins-merged
graylog_1        | + export GRAYLOG_PLUGIN_DIR=/usr/share/graylog/plugins-merged
graylog_1        | + GRAYLOG_PLUGIN_DIR=/usr/share/graylog/plugins-merged
graylog_1        | rm -f /usr/share/graylog/plugins-merged/*
graylog_1        | + rm -f /usr/share/graylog/plugins-merged/graylog-plugin-aws-4.3.4.jar /usr/share/graylog/plugins-merged/graylog-plugin-collector-4.3.4.jar /usr/share/graylog/plugins-merged/graylog-plugin-integrations-4.3.4.jar /usr/share/graylog/plugins-merged/graylog-plugin-threatintel-4.3.4.jar /usr/share/graylog/plugins-merged/graylog-storage-elasticsearch6-4.3.4.jar /usr/share/graylog/plugins-merged/graylog-storage-elasticsearch7-4.3.4.jar
graylog_1        | find /usr/share/graylog/plugins-default/ -type f -exec cp {} /usr/share/graylog/plugins-merged/ \;
graylog_1        | + find /usr/share/graylog/plugins-default/ -type f -exec cp '{}' /usr/share/graylog/plugins-merged/ ';'
graylog_1        | find /usr/share/graylog/plugin/ -type f -exec cp {} /usr/share/graylog/plugins-merged/ \;
graylog_1        | + find /usr/share/graylog/plugin/ -type f -exec cp '{}' /usr/share/graylog/plugins-merged/ ';'
graylog_1        |
graylog_1        |
graylog_1        | setup() {
graylog_1        |   # Create data directories
graylog_1        |   for d in journal log plugin config contentpacks
graylog_1        |   do
graylog_1        |     dir=${GRAYLOG_HOME}/data/${d}
graylog_1        |     [[ -d "${dir}" ]] || mkdir -p "${dir}"
graylog_1        |
graylog_1        |     if [[ "$(stat --format='%U:%G' $dir)" != 'graylog:graylog' ]] && [[ -w "$dir" ]]; then
graylog_1        |       chown -R graylog:graylog "$dir" || echo "Warning can not change owner to graylog:graylog"
graylog_1        |     fi
graylog_1        |   done
graylog_1        | }
graylog_1        |
graylog_1        | graylog() {
graylog_1        |
graylog_1        |   exec "${JAVA_HOME}/bin/java" \
graylog_1        |     ${GRAYLOG_SERVER_JAVA_OPTS} \
graylog_1        |     -jar \
graylog_1        |     -Dlog4j.configurationFile="${GRAYLOG_HOME}/data/config/log4j2.xml" \
graylog_1        |     -Djava.library.path="${GRAYLOG_HOME}/lib/sigar/" \
graylog_1        |     -Dgraylog2.installation_source=docker \
graylog_1        |     "${GRAYLOG_HOME}/graylog.jar" \
graylog_1        |     "$@" \
graylog_1        |     -f "${GRAYLOG_HOME}/data/config/graylog.conf"
graylog_1        | }
graylog_1        |
graylog_1        | run() {
graylog_1        |   setup
graylog_1        |
graylog_1        |   # if being called without an argument assume "server" for backwards compatibility
graylog_1        |   if [ $# = 0 ]; then
graylog_1        |     graylog server "$@"
graylog_1        |   fi
graylog_1        |
graylog_1        |   graylog "$@"
graylog_1        | }
graylog_1        |
graylog_1        | run "$@"
graylog_1        | + run
graylog_1        | + setup
graylog_1        | + for d in journal log plugin config contentpacks
graylog_1        | + dir=/data/journal
graylog_1        | + [[ -d /data/journal ]]
graylog_1        | + mkdir -p /data/journal
graylog_1        | mkdir: cannot create directory ‘/data’: Permission denied
graylog_graylog_1 exited with code 1

It seems that @coffee-squirrel is right, the last few lines seem to indicate that ${GRAYLOG_HOME} is not being set properly.

[mpfz0r]

@MahdiGhiasi Yeah, looks like it. But your output is truncated. It misses the part where /etc/profile is sourced, that's where GRAYLOG_HOME should be set

[MahdiGhiasi]

@mpfz0r Oh, sorry. Here's the complete output:

#!/bin/bash

set -e
+ set -e

# save the settings over the docker(-compose) environment
__GRAYLOG_SERVER_JAVA_OPTS=${GRAYLOG_SERVER_JAVA_OPTS}
+ __GRAYLOG_SERVER_JAVA_OPTS='-Xms256m -Xmx256m'

# shellcheck disable=SC1091
source /etc/profile
+ source /etc/profile
# /etc/profile: system-wide .profile file for the Bourne shell (sh(1))
# and Bourne compatible shells (bash(1), ksh(1), ash(1), ...).

if [ "${PS1-}" ]; then
  if [ "${BASH-}" ] && [ "$BASH" != "/bin/sh" ]; then
    # The file bash.bashrc already sets the default PS1.
    # PS1='\h:\w\$ '
    if [ -f /etc/bash.bashrc ]; then
      . /etc/bash.bashrc
    fi
  else
    if [ "$(id -u)" -eq 0 ]; then
      PS1='# '
    else
      PS1='$ '
    fi
  fi
fi
++ '[' '' ']'

if [ -d /etc/profile.d ]; then
  for i in /etc/profile.d/*.sh; do
    if [ -r $i ]; then
      . $i
    fi
  done
  unset i
fi
++ '[' -d /etc/profile.d ']'
++ for i in /etc/profile.d/*.sh
++ '[' -r /etc/profile.d/01-locale-fix.sh ']'
++ for i in /etc/profile.d/*.sh
++ '[' -r /etc/profile.d/graylog.sh ']'
++ unset i

#Set default GC
if [[ -z ${GRAYLOG_DOCKER_DISABLE_CMS_GC} ]]
then
  if "${JAVA_HOME}/bin/java" -XX:+PrintFlagsFinal 2>&1 |grep -q UseParNewGC; then
    GRAYLOG_SERVER_JAVA_OPTS="${GRAYLOG_SERVER_JAVA_OPTS} -XX:+UseParNewGC"
    export GRAYLOG_SERVER_JAVA_OPTS
  fi
  if "${JAVA_HOME}/bin/java" -XX:+PrintFlagsFinal 2>&1 |grep -q UseConcMarkSweepGC; then
    GRAYLOG_SERVER_JAVA_OPTS="${GRAYLOG_SERVER_JAVA_OPTS} -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled"
    export GRAYLOG_SERVER_JAVA_OPTS
  fi
fi
+ [[ -z '' ]]
+ /opt/java/openjdk/bin/java -XX:+PrintFlagsFinal
+ grep -q UseParNewGC
+ /opt/java/openjdk/bin/java -XX:+PrintFlagsFinal
+ grep -q UseConcMarkSweepGC

# and add the previous saved settings to our defaults
if [[ ! -z ${__GRAYLOG_SERVER_JAVA_OPTS} ]]
then
  echo "adding environment opts"
  GRAYLOG_SERVER_JAVA_OPTS="${GRAYLOG_SERVER_JAVA_OPTS} ${__GRAYLOG_SERVER_JAVA_OPTS}"
  export GRAYLOG_SERVER_JAVA_OPTS
fi
+ [[ ! -z -Xms256m -Xmx256m ]]
+ echo 'adding environment opts'
adding environment opts
+ GRAYLOG_SERVER_JAVA_OPTS='-Xms256m -Xmx256m -Xms256m -Xmx256m'
+ export GRAYLOG_SERVER_JAVA_OPTS

# Convert all environment variables with names ending in __FILE into the content of
# the file that they point at and use the name without the trailing __FILE.
# This can be used to carry in Docker secrets.
for VAR_NAME in $(env | grep '^GRAYLOG_[^=]\+__FILE=.\+' | sed -r 's/^(GRAYLOG_[^=]*)__FILE=.*/\1/g'); do
  VAR_NAME_FILE="${VAR_NAME}__FILE"
  if [ "${!VAR_NAME}" ]; then
    echo >&2 "ERROR: Both ${VAR_NAME} and ${VAR_NAME_FILE} are set but are exclusive"
    exit 1
  fi
  VAR_FILENAME="${!VAR_NAME_FILE}"
  echo "Getting secret ${VAR_NAME} from ${VAR_FILENAME}"
  if [ ! -r "${VAR_FILENAME}" ]; then
    echo >&2 "ERROR: ${VAR_FILENAME} does not exist or is not readable"
    exit 1
  fi
  export "${VAR_NAME}"="$(< "${VAR_FILENAME}")"
  unset "${VAR_NAME_FILE}"
done
++ sed -r 's/^(GRAYLOG_[^=]*)__FILE=.*/\1/g'
++ grep '^GRAYLOG_[^=]\+__FILE=.\+'
++ env


# Delete outdated PID file
[[ -e /tmp/graylog.pid ]] && rm --force /tmp/graylog.pid
+ [[ -e /tmp/graylog.pid ]]

# check if we are inside kubernetes, Graylog should be run as statefulset and $POD_NAME env var should be defined like this
#          env:
#          - name: POD_NAME
#            valueFrom:
#              fieldRef:
#                fieldPath: metadata.name
# First stateful member is having pod name ended with -0, so
if [[ ! -z "${POD_NAME}" ]]
then
 if echo "${POD_NAME}" | grep "\\-0$" >/dev/null
 then
   export GRAYLOG_IS_MASTER="true"
 else
   export GRAYLOG_IS_MASTER="false"
 fi
fi
+ [[ ! -z '' ]]

# check if we are inside a nomad cluster
# First member is having alloc-index 0, so
if [[ ! -z "${NOMAD_ALLOC_INDEX}" ]]; then
  if [ ${NOMAD_ALLOC_INDEX} == 0 ]; then
    export GRAYLOG_IS_MASTER="true"
  else
    export GRAYLOG_IS_MASTER="false"
  fi
fi
+ [[ ! -z '' ]]

# Merge plugin dirs to allow mounting of /plugin as a volume
export GRAYLOG_PLUGIN_DIR=/usr/share/graylog/plugins-merged
+ export GRAYLOG_PLUGIN_DIR=/usr/share/graylog/plugins-merged
+ GRAYLOG_PLUGIN_DIR=/usr/share/graylog/plugins-merged
rm -f /usr/share/graylog/plugins-merged/*
+ rm -f '/usr/share/graylog/plugins-merged/*'
find /usr/share/graylog/plugins-default/ -type f -exec cp {} /usr/share/graylog/plugins-merged/ \;
+ find /usr/share/graylog/plugins-default/ -type f -exec cp '{}' /usr/share/graylog/plugins-merged/ ';'
find /usr/share/graylog/plugin/ -type f -exec cp {} /usr/share/graylog/plugins-merged/ \;
+ find /usr/share/graylog/plugin/ -type f -exec cp '{}' /usr/share/graylog/plugins-merged/ ';'


setup() {
  # Create data directories
  for d in journal log plugin config contentpacks
  do
    dir=${GRAYLOG_HOME}/data/${d}
    [[ -d "${dir}" ]] || mkdir -p "${dir}"

    if [[ "$(stat --format='%U:%G' $dir)" != 'graylog:graylog' ]] && [[ -w "$dir" ]]; then
      chown -R graylog:graylog "$dir" || echo "Warning can not change owner to graylog:graylog"
    fi
  done
}

graylog() {

  exec "${JAVA_HOME}/bin/java" \
    ${GRAYLOG_SERVER_JAVA_OPTS} \
    -jar \
    -Dlog4j.configurationFile="${GRAYLOG_HOME}/data/config/log4j2.xml" \
    -Djava.library.path="${GRAYLOG_HOME}/lib/sigar/" \
    -Dgraylog2.installation_source=docker \
    "${GRAYLOG_HOME}/graylog.jar" \
    "$@" \
    -f "${GRAYLOG_HOME}/data/config/graylog.conf"
}

run() {
  setup

  # if being called without an argument assume "server" for backwards compatibility
  if [ $# = 0 ]; then
    graylog server "$@"
  fi

  graylog "$@"
}

run "$@"
+ run
+ setup
+ for d in journal log plugin config contentpacks
+ dir=/data/journal
+ [[ -d /data/journal ]]
+ mkdir -p /data/journal
mkdir: cannot create directory ‘/data’: Permission denied

[mpfz0r]

@MahdiGhiasi
looks like /etc/profile.d/graylog.sh is not readable for some reason.

what is the output of entrypoint: "/bin/bash -c 'ls -l /etc/profile.d/'"

[MahdiGhiasi]

@mpfz0r

graylog_1        | total 8
graylog_1        | -rw-r--r-- 1 root root  96 Oct 15  2021 01-locale-fix.sh
graylog_1        | -rw-r--r-- 1 root root 564 Aug  3 19:02 graylog.sh

[mpfz0r]

@MahdiGhiasi entrypoint: "/bin/bash -c 'ls -ld /etc/profile.d/; ls -ld /etc; ls -ld /'"

[MahdiGhiasi]

@mpfz0r

graylog_1        | drwxr-xr-x 1 root root 4096 Aug  3 19:02 /etc/profile.d/
graylog_1        | drwxr-xr-x 1 root root 4096 Aug  4 18:24 /etc
graylog_1        | drwxr-xr-x 1 root root 4096 Aug  4 18:24 /

[mpfz0r]

@MahdiGhiasi entrypoint: "/bin/bash -c '[ -r /etc/profile.d/graylog.sh ] && echo YEAH'"

[MahdiGhiasi]

@mpfz0r This does not print YEAH.