nginx segfault in libmodsecurity.so.3.0.0

[brianp9906]

When attempting to remove a false positive from CRS Rules, Nginx stops responding and generates segmentation faults with libmodsecurity.

file: RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf
add: SecRuleUpdateTargetById 941120 "!REQUEST_HEADERS:Referer"

systemctl reload nginx

Check /var/log/messages:

kernel: nginx[58950]: segfault at 28 ip 00007f59e985af4a sp 00007ffd0657cd80 error 4 in libmodsecurity.so.3.0.0[7f59e9740000+1ed000]

libmodsecurity config output from compiling:

`ModSecurity - v3.0.0-48-ga66aceb for Linux

Mandatory dependencies

• libInjection ....v3.0.0-48-ga66aceb
• SecLang tests ....a66aceb

Optional dependencies

• GeoIP ....found v1.5.0
  -lGeoIP , -I/usr/include/
• LibCURL ....found v7.29.0
  -lcurl , -DWITH_CURL
• YAJL ....found v2.0.4
  -lyajl , -DWITH_YAJL
• LMDB ....not found
• LibXML2 ....found v2.9.1
  -lxml2 -lz -lm -ldl, -I/usr/include/libxml2 -DWITH_LIBXML2
• SSDEEP ....not found
• LUA ....not found

Other Options

• Test Utilities ....enabled
• SecDebugLog ....enabled
• afl fuzzer ....disabled
• library examples ....enabled
• Building parser ....disabled
• Treating pm operations as critical section ....disabled`

[victorhora]

@brianp9906 Are you using any other 3rd party modules or Kubernetes? See ModSecurity-nginx/#74

[brianp9906]

Nope. Just nginx official repo for easy yum installation. Then used the libmodsecurity compile recipe for CentOS 7. I’m using CRS 3.1 and simply putting an exclusion in the file I mentioned.

[zimmerle]

Hi @brianp9906,

Can you share the output of nginx -V on gists?

[brianp9906]

Sure. Nothing special here. Just used Nginx centos7 repo:
[nginx]
name=nginx_repo
baseurl=http://nginx.org/packages/mainline/centos/7/$basearch/
gpgcheck=0
enabled=1

nginx -V output:

nginx version: nginx/1.13.9
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC)
built with OpenSSL 1.0.2k-fips 26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'

[victorhora]

@brianp9906 Thanks for sharing that. Do you know if you have any "conditionals" like "if"/"else"/"then" inside your Nginx.conf? Would mind sharing the conf as well?

Thanks.

[zimmerle]

Running a very simple config with the same environment, I was not able to reproduce the issue.

[LeSwiss]

I guess I have exactly the same issue.

Running nginx plus:

[nginx@stwnetvm01v nginx]# nginx -V
nginx version: nginx/1.13.7 (nginx-plus-r14-p1)
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC)
built with OpenSSL 1.0.1e-fips 11 Feb 2013
TLS SNI support enabled
configure arguments: --build=nginx-plus-r14-p1 --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_jwt_module --with-http_auth_request_module --with-http_dav_module --with-http_f4f_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_hls_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_session_log_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'

As soon as I activate just one SecRuleRemoveByTag, even an example in RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf like "SecRuleUpdateTargetByTag "attack-sqli" "!ARGS:foo"", I receive a ERR_EMPTY_RESPONSE on browsing every domain behind this nginx / WAF instance and some of theses messages:

Apr  7 01:32:33 stwnetvm01v kernel: nginx[108260]: segfault at 28 ip 00007f0c00d2a95a sp 00007ffc145323f0 error 4 in libmodsecurity.so.3.0.0[7f0c00c12000+1f4000]
Apr  7 01:32:33 stwnetvm01v kernel: nginx[108255]: segfault at 28 ip 00007f0c00d2a95a sp 00007ffc145323f0 error 4 in libmodsecurity.so.3.0.0[7f0c00c12000+1f4000]
Apr  7 01:32:33 stwnetvm01v kernel: nginx[108257]: segfault at 28 ip 00007f0c00d2a95a sp 00007ffc145323f0 error 4 in libmodsecurity.so.3.0.0[7f0c00c12000+1f4000]
Apr  7 01:32:33 stwnetvm01v kernel: nginx[108378]: segfault at 28 ip 00007f0c00d2a95a sp 00007ffc14532470 error 4 in libmodsecurity.so.3.0.0[7f0c00c12000+1f4000]
Apr  7 01:32:33 stwnetvm01v kernel: nginx[108262]: segfault at 28 ip 00007f0c00d2a95a sp 00007ffc145323f0 error 4 in libmodsecurity.so.3.0.0[7f0c00c12000+1f4000]
Apr  7 01:32:33 stwnetvm01v kernel: nginx[108379]: segfault at 28 ip 00007f0c00d2a95a sp 00007ffc14532470 error 4 in libmodsecurity.so.3.0.0[7f0c00c12000+1f4000]
Apr  7 01:32:33 stwnetvm01v kernel: nginx[112407]: segfault at 28 ip 00007f0c00d2a95a sp 00007ffc14532470 error 4 in libmodsecurity.so.3.0.0[7f0c00c12000+1f4000]
Apr  7 01:32:39 stwnetvm01v kernel: nginx[112409]: segfault at 28 ip 00007f0c00d2a95a sp 00007ffc14532470 error 4 in libmodsecurity.so.3.0.0[7f0c00c12000+1f4000]
Apr  7 01:33:09 stwnetvm01v kernel: nginx[108253]: segfault at 28 ip 00007f0c00d2a95a sp 00007ffc145323f0 error 4 in libmodsecurity.so.3.0.0[7f0c00c12000+1f4000]

SecRuleRemoveById on the other hand is working as expected.

No if / else used anywhere in my config.

Any help appreciated.
Kind regards.

[LeSwiss]

Nobody can help here?

[victorhora]

@LeSwiss

Can you confirm your version of libModSecurity and nginx-connector?

Maybe ctl:ruleRemovebyTag works for you? It was recently implemented here.

[zimmerle]

@defanator as it is an nginx plus can you help us to try to reproduce the issue? Not able to reproduce in a regular one.

[defanator]

@zimmerle sure thing, I'll take a look. Which "regular" versions of nginx have you tried? I suppose, you've been trying with current v3/master?