Nginx 1.22.0 doesn't start with OpenTelemetry enabled

[jmadureira]

Describe your environment
The issue was detected when using:

• nginx:1.22.0-alpine
• opentelemetry-cpp v.1.2.0
• latest version of the nginx instrumentation

Steps to reproduce
It is possible to reproduce using the following Dockerfile and nginx.conf sample:

FROM nginx:1.22.0-alpine as builder

ENV OPENTELEMETRY_VERSION v1.2.0

RUN apk update \
  && apk add --update \
      alpine-sdk build-base cmake linux-headers libressl-dev pcre-dev zlib-dev \
      grpc grpc-dev curl-dev protobuf-dev c-ares-dev \
      re2-dev abseil-cpp

RUN git clone --shallow-submodules --depth 1 --recurse-submodules -b ${OPENTELEMETRY_VERSION} \
  https://github.com/open-telemetry/opentelemetry-cpp.git \
  && cd opentelemetry-cpp \
  && mkdir build \
  && cd build \
  && cmake -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_PREFIX=/install \
    -DCMAKE_PREFIX_PATH=/install \
    -DWITH_OTLP=ON \
    -DWITH_OTLP_GRPC=ON \
    -DWITH_OTLP_HTTP=OFF \
    -DBUILD_TESTING=OFF \
    -DWITH_EXAMPLES=OFF \
    -DWITH_ABSEIL=ON \
    -DCMAKE_POSITION_INDEPENDENT_CODE=ON \
    .. \
  && make -j2 \
  && make install

RUN git clone https://github.com/open-telemetry/opentelemetry-cpp-contrib.git \
  && cd opentelemetry-cpp-contrib/instrumentation/nginx \
  && mkdir build \
  && cd build \
  && cmake -DCMAKE_BUILD_TYPE=Release \
    -DNGINX_BIN=/usr/sbin/nginx \
    -DCMAKE_PREFIX_PATH=/install \
    -DCMAKE_INSTALL_PREFIX=/usr/lib/nginx/modules \
    -DCURL_LIBRARY=/usr/lib/libcurl.so.4 \
    .. \
  && make -j2 \
  && make install

FROM nginx:1.22.0-alpine

RUN apk update \
  && apk add --update \
      grpc protobuf c-ares
      # gdb # uncomment this if you need to debug nginx

COPY --from=builder /usr/lib/nginx/modules/otel_ngx_module.so /usr/lib/nginx/modules/

COPY nginx/conf/nginx.conf /etc/nginx/nginx.conf

# CMD ["/usr/sbin/nginx-debug", "-g", "daemon off;"]

worker_processes  auto;
error_log  /dev/stderr;
pid        /tmp/nginx.pid;
worker_rlimit_nofile 8192;
worker_rlimit_core  500M;
working_directory /tmp;
debug_points abort;
load_module /usr/lib/nginx/modules/otel_ngx_module.so;

events {
  worker_connections  4096;  ## Default: 1024
}

http {
  server {
    listen 8080;

    location /status {
      # opentelemetry off;
      stub_status on;
      access_log  off;
      allow 127.0.0.1;
      deny all;
    }

    location / {
      root   html;
      index  index.html index.htm;
    }
  }
}

What is the expected behavior?
Nginx should start normally

What is the actual behavior?
Nginx logs the following messages and fails:

/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2022/07/28 09:16:02 [emerg] 1#1: dlopen() "/usr/lib/nginx/modules/otel_ngx_module.so" failed (Error relocating /usr/lib/nginx/modules/otel_ngx_module.so: _ZN4absl12lts_2021110216variant_internal21ThrowBadVariantAccessEv: symbol not found) in /etc/nginx/nginx.conf:8
nginx: [emerg] dlopen() "/usr/lib/nginx/modules/otel_ngx_module.so" failed (Error relocating /usr/lib/nginx/modules/otel_ngx_module.so: _ZN4absl12lts_2021110216variant_internal21ThrowBadVariantAccessEv: symbol not found) in /etc/nginx/nginx.conf:8

Additional context

[jmadureira]

Seems to be addressed in PR #205.

[DebajitDas]

@jmadureira #205 takes care of "instrumentation/otel-webserver-module" directory which also has opentelemetry module for nginx.
This issue is referred to "intrumentation/nginx" directory, #205 does not take care of that.

[rcjsuen]

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.3.0
  Build:         2b7b74854d90ad9b4b96a5011b9e8b67d20bfb8f
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.10

-------------------------------------------------------------------------------

We have the same problem with k8s.gcr.io/ingress-nginx/controller:v1.3.0 which looks to be on 1.19.10 so I don't think this problem is unique to Nginx 1.22.0.

[esigo]

@rcjsuen, @jmadureira you may check my PR on nginx repo kubernetes/ingress-nginx#9062.

[rcjsuen]

@esigo Yeah, I also came across kubernetes/ingress-nginx#8585 and kubernetes/ingress-nginx#9017 while trying to figure out this symbol not found error.

At this rate it might be better to just wait for the official release of OpenTelemetry support in the Nginx controller in my case.

[rcjsuen]

Compiling gRPC manually seems like a workaround...? 🤔

FROM nginx:1.22.0-alpine as builder

RUN apk update \
  && apk add --update \
      alpine-sdk build-base cmake linux-headers libressl-dev pcre-dev zlib-dev \
      curl-dev protobuf-dev c-ares-dev \
      re2-dev abseil-cpp

ENV GRPC_VERSION v1.43.2
RUN git clone --shallow-submodules --depth 1 --recurse-submodules -b ${GRPC_VERSION} \
  https://github.com/grpc/grpc \
  && cd grpc \
  && mkdir -p cmake/build \
  && cd cmake/build \
  && cmake \
    -DgRPC_INSTALL=ON \
    -DgRPC_BUILD_TESTS=OFF \
    -DCMAKE_INSTALL_PREFIX=/install \
    -DCMAKE_BUILD_TYPE=Release \
    -DgRPC_BUILD_GRPC_NODE_PLUGIN=OFF \
    -DgRPC_BUILD_GRPC_OBJECTIVE_C_PLUGIN=OFF \
    -DgRPC_BUILD_GRPC_PHP_PLUGIN=OFF \
    -DgRPC_BUILD_GRPC_PHP_PLUGIN=OFF \
    -DgRPC_BUILD_GRPC_PYTHON_PLUGIN=OFF \
    -DgRPC_BUILD_GRPC_RUBY_PLUGIN=OFF \
    ../.. \
  && make -j2 \
    && make install

ENV OPENTELEMETRY_VERSION v1.3.0
RUN git clone --shallow-submodules --depth 1 --recurse-submodules -b ${OPENTELEMETRY_VERSION} \
  https://github.com/open-telemetry/opentelemetry-cpp.git \
  && cd opentelemetry-cpp \
  && mkdir build \
  && cd build \
  && cmake -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_PREFIX=/install \
    -DCMAKE_PREFIX_PATH=/install \
    -DWITH_ZIPKIN=OFF \
    -DWITH_JAEGER=OFF \
    -DWITH_OTLP=ON \
    -DWITH_OTLP_GRPC=ON \
    -DWITH_OTLP_HTTP=OFF \
    -DBUILD_TESTING=OFF \
    -DWITH_EXAMPLES=OFF \
    -DWITH_ABSEIL=ON \
    -DCMAKE_POSITION_INDEPENDENT_CODE=ON \
    .. \
  && make -j2 \
  && make install

RUN git clone https://github.com/open-telemetry/opentelemetry-cpp-contrib.git \
  && cd opentelemetry-cpp-contrib/instrumentation/nginx \
  && mkdir build \
  && cd build \
  && cmake -DCMAKE_BUILD_TYPE=Release \
    -DNGINX_BIN=/usr/sbin/nginx \
    -DCMAKE_PREFIX_PATH=/install \
    -DCMAKE_INSTALL_PREFIX=/usr/lib/nginx/modules \
    -DCURL_LIBRARY=/usr/lib/libcurl.so.4 \
    .. \
  && make -j2 \
  && make install

FROM nginx:1.22.0-alpine

RUN apk update \
  && apk add --update \
      grpc protobuf c-ares
      # gdb # uncomment this if you need to debug nginx

COPY --from=builder /usr/lib/nginx/modules/otel_ngx_module.so /usr/lib/nginx/modules/

COPY nginx.conf /etc/nginx/nginx.conf

# CMD ["/usr/sbin/nginx-debug", "-g", "daemon off;"]

[esigo]

The symbols were not found for k8s-nginx mainly because the shared libraries were not copied over to the controller container from the init container:
https://github.com/kubernetes/ingress-nginx/blob/50be2bf95fd1ef480420e2aa1d6c5c7c138c95ea/images/opentelemetry/rootfs/Dockerfile#L38-L45

[rcjsuen]

Yeah, I went down the path of manual compiling gRPC because I saw that in the ingress controller project.

[jmadureira]

Also tested with manual gRPC compilation and including the shared libraries and it works as expected:

FROM nginx:1.22.0-alpine as builder

ENV OPENTELEMETRY_VERSION v1.2.0

RUN apk update \
  && apk add --update \
      alpine-sdk build-base cmake linux-headers libressl-dev pcre-dev zlib-dev \
      curl-dev protobuf-dev c-ares-dev \
      re2-dev abseil-cpp

ENV GRPC_VERSION v1.43.2
RUN git clone --shallow-submodules --depth 1 --recurse-submodules -b ${GRPC_VERSION} \
  https://github.com/grpc/grpc \
  && cd grpc \
  && mkdir -p cmake/build \
  && cd cmake/build \
  && cmake \
    -DgRPC_INSTALL=ON \
    -DgRPC_BUILD_TESTS=OFF \
    -DCMAKE_INSTALL_PREFIX=/install \
    -DCMAKE_BUILD_TYPE=Release \
    -DgRPC_BUILD_GRPC_NODE_PLUGIN=OFF \
    -DgRPC_BUILD_GRPC_OBJECTIVE_C_PLUGIN=OFF \
    -DgRPC_BUILD_GRPC_PHP_PLUGIN=OFF \
    -DgRPC_BUILD_GRPC_PHP_PLUGIN=OFF \
    -DgRPC_BUILD_GRPC_PYTHON_PLUGIN=OFF \
    -DgRPC_BUILD_GRPC_RUBY_PLUGIN=OFF \
    ../.. \
  && make -j2 \
    && make install

RUN git clone --shallow-submodules --depth 1 --recurse-submodules -b ${OPENTELEMETRY_VERSION} \
  https://github.com/open-telemetry/opentelemetry-cpp.git \
  && cd opentelemetry-cpp \
  && mkdir build \
  && cd build \
  && cmake -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_PREFIX=/install \
    -DCMAKE_PREFIX_PATH=/install \
    -DWITH_OTLP=ON \
    -DWITH_OTLP_GRPC=ON \
    -DWITH_OTLP_HTTP=OFF \
    -DBUILD_TESTING=OFF \
    -DWITH_EXAMPLES=OFF \
    -DWITH_ABSEIL=ON \
    -DCMAKE_POSITION_INDEPENDENT_CODE=ON \
    .. \
  && make -j2 \
  && make install

RUN git clone https://github.com/open-telemetry/opentelemetry-cpp-contrib.git \
  && cd opentelemetry-cpp-contrib/instrumentation/nginx \
  && mkdir build \
  && cd build \
  && cmake -DCMAKE_BUILD_TYPE=Release \
    -DNGINX_BIN=/usr/sbin/nginx \
    -DCMAKE_PREFIX_PATH=/install \
    -DCMAKE_INSTALL_PREFIX=/usr/lib/nginx/modules \
    -DCURL_LIBRARY=/usr/lib/libcurl.so.4 \
    .. \
  && make -j2 \
  && make install

FROM nginx:1.22.0-alpine

RUN apk update \
  && apk add --update \
      grpc protobuf c-ares
      # gdb # uncomment this if you need to debug nginx

COPY --from=builder /usr/lib/nginx/modules/otel_ngx_module.so /usr/lib/nginx/modules/
COPY --from=builder install/lib/lib* /etc/nginx/modules/

COPY nginx/conf/nginx.conf /etc/nginx/nginx.conf

# CMD ["/usr/sbin/nginx-debug", "-g", "daemon off;"]

It would be nice if the > 15 minutes of gRPC compilation could be avoided.

[rcjsuen]

Also tested with manual gRPC compilation and including the shared libraries and it works as expected

Thanks for verifying, @jmadureira!

It would be nice if the > 15 minutes of gRPC compilation could be avoided.

Agreed. This step burns through a lot of compute resources in the CI pipeline. Not sure if this is an issue with Alpine itself or what. There may also be additional cmake flags that could be configured to reduce the compilation time some more... 🤔

[esigo]

Hi,
here is the solution with static link:
kubernetes/ingress-nginx#9286