nginx does not include all configured modules

[ledroide]

I was surprised to get errors like "missing devel_kit (ndk) module" when trying to use environment variables with set_by_lua directive.

Checking here below using upstream image tsuru/nginx-tsuru:1.16.1 - also tried with locally built image, with 1.16.1, 1.17.3 and 1.17.4 nginx : same result.

First of all, there is no module "nginx devel kit" (ndk) in the running nginx instance, while it's cleary built when choosing tsuru flavor :
$ export TEST_IMAGE=docker.io/tsuru/nginx-tsuru:1.16.1
$ docker run -t ${TEST_IMAGE} nginx -V 2>&1 | tr -- - '\n' | grep module

modules
path=/usr/lib/nginx/modules
http_addition_module
http_auth_request_module
http_dav_module
http_flv_module
http_gunzip_module
http_gzip_static_module
http_mp4_module
http_random_index_module
http_realip_module
http_secure_link_module
http_slice_module
http_ssl_module
http_stub_status_module
http_sub_module
http_v2_module
mail_ssl_module
stream_realip_module
stream_ssl_module
stream_ssl_preread_module

At build level and image contents, it looks good :

$ docker run -t ${TEST_IMAGE} ls /etc/nginx/modules/ndk_http_module.so
/etc/nginx/modules/ndk_http_module.so
$
$ docker run -t ${TEST_IMAGE} grep ndk_http_module.so /etc/nginx/modules/all.conf
load_module /etc/nginx/modules/ndk_http_module.so;

But nginx.conf does not include /etc/nginx/modules/all.conf :

docker run -t ${TEST_IMAGE} cat /etc/nginx/nginx.conf | egrep -v '^[[:space:]]*$'
user  nginx;
worker_processes  1;
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;
events {
    worker_connections  1024;
}
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    access_log  /var/log/nginx/access.log  main;
    sendfile        on;
    #tcp_nopush     on;
    keepalive_timeout  65;
    #gzip  on;
    include /etc/nginx/conf.d/*.conf;
}

But this does not explain how some modules are loaded and some are not.

So I first built a new image based on nginx 1.17.4 and including a nginx.conf with directive include /etc/nginx/modules/all.conf; at first line.

This nginx.conf is added to the Dockerfile like this COPY nginx.conf /etc/nginx/

I also added the configure option --with-http_perl_module in order to try env vars usage through perl module.

configure_args="$configure_args --with-http_perl_module=dynamic"; \

After build with command make image flavor=tsuru nginx_version=1.17.4 and a custom docker tag :

$ export TEST_IMAGE=local/nginx-tsuru:1.17.4-0.0.2
$ docker run -t ${TEST_IMAGE} nginx -V 2>&1 | tr -- - '\n' | grep module
modules
path=/usr/lib/nginx/modules
http_addition_module
http_auth_request_module
http_dav_module
http_flv_module
http_gunzip_module
http_gzip_static_module
http_mp4_module
http_random_index_module
http_realip_module
http_secure_link_module
http_slice_module
http_ssl_module
http_stub_status_module
http_sub_module
http_v2_module
mail_ssl_module
stream_realip_module
stream_ssl_module
stream_ssl_preread_module
$
$ docker run -t ${TEST_IMAGE} ls /etc/nginx/modules/ndk_http_module.so /etc/nginx/modules/ngx_http_perl_module.so
/etc/nginx/modules/ndk_http_module.so
/etc/nginx/modules/ngx_http_perl_module.so
$
$ docker run -t ${TEST_IMAGE} egrep 'ndk_http_module.so|ngx_http_perl_module.so' /etc/nginx/modules/all.conf
load_module /etc/nginx/modules/ndk_http_module.so;
load_module /etc/nginx/modules/ngx_http_perl_module.so;
$
$ docker run -t ${TEST_IMAGE} cat /etc/nginx/nginx.conf | egrep -v '^[[:space:]]*$' | head -4
include       /etc/nginx/modules/all.conf;
worker_processes  1;
error_log  /var/log/nginx/error.log warn;
pid        /tmp/nginx.pid;

That means : even when adding include /etc/nginx/modules/all.conf in nginx.conf, it looks like modules are not loaded by nginx.
Also tried with tag v0.3.1 instead of v0.3.0 for simplresty/ngx_devel_kit : no difference.
Any idea why modules are missing ?

[ledroide]

Maybe I have found something interesting.
When I add include /etc/nginx/modules/all.conf in nginx.conf, then nginx reads all.conf when starting, and displays this message :

2019/10/15 09:07:50 [emerg] 1#1: dlopen() "/etc/nginx/modules/ngx_http_modsecurity_module.so" failed (libmaxminddb.so.0: cannot open shared object file: No such file or directory) in /etc/nginx/modules/all.conf:11
nginx: [emerg] dlopen() "/etc/nginx/modules/ngx_http_modsecurity_module.so" failed (libmaxminddb.so.0: cannot open shared object file: No such file or directory) in /etc/nginx/modules/all.conf:11

Line 11 of all.yaml is load_module /etc/nginx/modules/ngx_http_modsecurity_module.so;
So I check my own-built version and upstream image : libmaxminddb.so.0 isn't anywhere.

$ export TEST_IMAGE=docker.io/tsuru/nginx-tsuru:1.16.1
$ docker run -t ${TEST_IMAGE} ldconfig -p | wc -l
206
$ docker run -t ${TEST_IMAGE} ldconfig -p | grep libmaxminddb

Let's dig :

$ echo $TEST_IMAGE
docker.io/tsuru/nginx-tsuru:1.16.1
$ docker run -u 0:0 -ti ${TEST_IMAGE} /bin/bash
root@31b7a89d65d2:/etc/nginx# apt update && apt install -y mlocate && updatedb
root@31b7a89d65d2:/etc/nginx# locate libmodsecurity
/usr/local/lib/libmodsecurity.a
/usr/local/lib/libmodsecurity.la
/usr/local/lib/libmodsecurity.so
/usr/local/lib/libmodsecurity.so.3
/usr/local/lib/libmodsecurity.so.3.0.3
root@31b7a89d65d2:/etc/nginx# locate libmaxminddb
root@31b7a89d65d2:/etc/nginx# 
root@31b7a89d65d2:/etc/nginx# apt install -y apt-file && apt-file update && apt-file list libmaxminddb0
libmaxminddb0: /usr/lib/x86_64-linux-gnu/libmaxminddb.so.0
libmaxminddb0: /usr/lib/x86_64-linux-gnu/libmaxminddb.so.0.0.7
(...)

That clearly means the library is not copied from the build step to the final image.

[ledroide]

I have fixed the issue on my own fork at ledroide/docker-nginx-with-modules.
There are too many changes for a pull-request.
My fork does what I need :

• it allows using environment variables through perl module.
• all other modules are now loaded in nginx, including nginx_devel_kit, geoip, modsecurity, lua, etc.

Feel free to cherry-pick whatever you need on my fork to fix or update your own work.
Serge

[nettoclaudio]

Hi @ledroide! First of all, thank you so much for your detailed diagnostic of this issue. I'm so glade you are using our project. Please feel free to bring more issues or push a PR.

As you have noticed, since all modules defined in your flavor (described into flavors.json file, e.g tsuru) are dynamic, we don't need to rebuild the whole nginx to link and use them. We just need to load the desired module(s) in nginx.conf file and run. For convenience, we always create a modules/all.conf file with all flavor's modules.

About the reported error: "missing devel_kit (ndk) module". I can't reproduce that on my machine. Below I show what I did and works fine!

$ cat my-nginx.conf
error_log /dev/stderr warn;

include /etc/nginx/modules/all.conf;

events {}

env MY_MOTD_ENV;

http {
    access_log  /dev/stdout;

    server {
       listen 8080;

       location /hello {
           set $name '';
           set_by_lua $name 'return ngx.var.arg_name';
           return 200 "Hello world, $name!\n";
       }

       location /motd {
           set_by_lua $message_of_the_day 'return os.getenv("MY_MOTD_ENV")';
           return 200 "$message_of_the_day\n";
       }
    }
}

$ docker run -d --name my-test -e MY_MOTD_ENV="Just more one coffee" -v $(pwd)/my-nginx.conf:/etc/nginx/nginx.conf -p 8080:8080 tsuru/nginx-tsuru:
1.16.1

And finally hitting the webserver it returns the expected responses.

$ curl localhost:8080/hello?name=nettoclaudio
Hello world, nettoclaudio!
$ curl localhost:8080/motd
Just more one coffee
$ docker logs my-test
172.17.0.1 - - [17/Oct/2019:23:49:48 +0000] "GET /hello?name=nettoclaudio HTTP/1.1" 200 27 "-" "curl/7.66.0"
172.17.0.1 - - [17/Oct/2019:23:49:51 +0000] "GET /motd HTTP/1.1" 200 21 "-" "curl/7.66.0"

Do am I doing something wrong? Can you show the steps to reproduce your error like above?

[ledroide]

@nettoclaudio : Do you know if modules list from command nginx -V is reliable ? Maybe it's not, and there is another way to check all static and dynamic modules ?
I was asserting something like "http_devel_kit_module" in the output, maybe it's a mistake ?
Serge

[nettoclaudio]

The nginx -V remains reliable to list static modules embed into nginx binary. On other hand, to list the dynamic modules you can list all shared objects in /etc/nginx/modules directory.

• Listing the static modules (take a look in parameters with _module suffix):

$ docker run -t --rm tsuru/nginx-tsuru:1.16.1 nginx -V 2>1 | grep 'configure arguments:'
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fdebug-prefix-map=/data/builder/debuild/nginx-1.16.1/debian/debuild-base/nginx-1.16.1=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'

• Listing dynamic modules

$ docker run -t --rm tsuru/nginx-tsuru:1.16.1 bash -c 'ls /etc/nginx/modules/*.so'
/etc/nginx/modules/ndk_http_module.so
/etc/nginx/modules/ngx_http_cache_purge_module.so
/etc/nginx/modules/ngx_http_dav_ext_module.so
/etc/nginx/modules/ngx_http_echo_module.so
/etc/nginx/modules/ngx_http_fancyindex_module.so
/etc/nginx/modules/ngx_http_geoip_module-debug.so
/etc/nginx/modules/ngx_http_geoip_module.so
/etc/nginx/modules/ngx_http_headers_more_filter_module.so
/etc/nginx/modules/ngx_http_image_filter_module-debug.so
/etc/nginx/modules/ngx_http_image_filter_module.so
/etc/nginx/modules/ngx_http_js_module-debug.so
/etc/nginx/modules/ngx_http_js_module.so
/etc/nginx/modules/ngx_http_lua_module.so
/etc/nginx/modules/ngx_http_modsecurity_module.so
/etc/nginx/modules/ngx_http_push_stream_module.so
/etc/nginx/modules/ngx_http_subs_filter_module.so
/etc/nginx/modules/ngx_http_uploadprogress_module.so
/etc/nginx/modules/ngx_http_vhost_traffic_status_module.so
/etc/nginx/modules/ngx_http_xslt_filter_module-debug.so
/etc/nginx/modules/ngx_http_xslt_filter_module.so
/etc/nginx/modules/ngx_stream_geoip_module-debug.so
/etc/nginx/modules/ngx_stream_geoip_module.so
/etc/nginx/modules/ngx_stream_js_module-debug.so
/etc/nginx/modules/ngx_stream_js_module.so

I'm unsure but http_devel_kit_module it's the ndk_http_module dynamic module.

[ledroide]

• Listing dynamic modules
  $ docker run -t --rm tsuru/nginx-tsuru:1.16.1 bash -c 'ls /etc/nginx/modules/*.so'

Listing a directory is not the same as checking that dynamic modules are actually loaded.
Modules files in /etc/nginx/modules/ are possibly usable by nginx, it does not mean they are in a given nginx running instance.
I did not find a reliable way to check this.

I'm unsure but http_devel_kit_module it's the ndk_http_module dynamic module.

Yes, this is the same module (nginx devel kit, alias ndk).

[nettoclaudio]

Listing a directory is not the same as checking that dynamic modules are actually loaded.
Modules files in /etc/nginx/modules/ are possibly usable by nginx, it does not mean they are in a given nginx running instance.
I did not find a reliable way to check this.

Yeah, absolutely... I just mean that those are the compiled dynamic modules. They can be loaded firing the load_module directive into nginx.conf file. Look all load_module directive, is it not enough for you?

[nn-rego]

I am trying copy all the required files from host to nginx:alpine image. My idea is to keep the image size very small.
This is my Dockerfile

FROM nginx:alpine

LABEL maintainer="noel"

RUN mkdir /etc/nginx/modsec &&
mkdir /usr/local/owasp-modsecurity-crs-3.0.0

RUN chown -R nginx:nginx /usr/share/nginx /etc/nginx

RUN echo "" && echo "Copying Host files. . . "

COPY ./modsec/nginx.conf /etc/nginx/nginx.conf
COPY ./modsec/modules /etc/nginx/modules
COPY --chown=nginx:nginx /modsec/modules/ngx_http_modsecurity_module.so /etc/nginx/modules/ngx_http_modsecurity_module.so
RUN chmod 777 /etc/nginx/modules/ngx_http_modsecurity_module.so

COPY ./modsec/modsecurity.conf /etc/nginx/modsec/modsecurity.conf
COPY ./modsec/unicode.mapping /etc/nginx/modsec/unicode.mapping
COPY ./modsec/owasp-modsecurity-crs-3.0.0 /usr/local/owasp-modsecurity-crs-3.0.0
COPY ./app/index.html /usr/share/nginx/html/index.html

RUN echo "" && echo "Copy complete. . . "

When i run this image I am getting this error.

docker build -t modsec_nginx:alpine1.0 .
docker run --rm -it -p 80:80/tcp modsec_nginx:alpine1.0
2020/03/13 04:47:43 [emerg] 1#1: dlopen() "/etc/nginx/modules/ngx_http_modsecurity_module.so" failed (Error loading shared library libmodsecurity.so.3: No such file or directory (needed by /etc/nginx/modules/ngx_http_modsecurity_module.so)) in /etc/nginx/nginx.conf:1
nginx: [emerg] dlopen() "/etc/nginx/modules/ngx_http_modsecurity_module.so" failed (Error loading shared library libmodsecurity.so.3: No such file or directory (needed by /etc/nginx/modules/ngx_http_modsecurity_module.so)) in /etc/nginx/nginx.conf:1

How can I keep Modsecurity nginx image size small,by using alpine niginx?