segfaults with socket connections and daemon off

[istr]

I get reproducible segfaults in ngx_postgres_keepalive_get_peer_single (and sometimes in ngx_postgres_keepalive_cleanup). Postgres connections are ok (functional) but they segfault after the requests have finished.

These are the versions used:

• openresty 1.7.0.1
• nginx-1.7.1.tar.gz (not packaged w/ openresty, but want syslog facility)
• ngx_postgres-1.0rc3-8f0eae297fc9e88e6ab08e6c9d18628568befba9
  (not packaged, but want unix socket facility)

It runs in a chroot jail with "daemon off" under supervision of sv.
Any ideas? Should I provide the binary and the coredumps?

nginx/sbin/nginx -V
nginx version: nginx/1.7.1
TLS SNI support enabled
configure arguments: --prefix=/opt/foo/sys/opt/ngx_openresty-1.7.0.1.tar.gz-b7f7b683e0d0748db1ec2d66c23959bf5f13cb1471f8e33631d132d0-neo-20140615200651/nginx --with-debug --add-module=../ngx_devel_kit-0.2.19 --add-module=../echo-nginx-module-0.53 --add-module=../xss-nginx-module-0.04 --add-module=../ngx_coolkit-0.2rc1 --add-module=../set-misc-nginx-module-0.24 --add-module=../form-input-nginx-module-0.08 --add-module=../encrypted-session-nginx-module-0.03 --add-module=../ngx_postgres-1.0rc3-8f0eae297fc9e88e6ab08e6c9d18628568befba9 --add-module=../srcache-nginx-module-0.27 --add-module=../ngx_lua-0.9.8 --add-module=../ngx_lua_upstream-0.02 --add-module=../headers-more-nginx-module-0.25 --add-module=../array-var-nginx-module-0.03 --add-module=../memc-nginx-module-0.14 --add-module=../redis2-nginx-module-0.11 --add-module=../redis-nginx-module-0.3.7 --add-module=../rds-json-nginx-module-0.13 --add-module=../rds-csv-nginx-module-0.05 --with-ld-opt=-Wl,-rpath,/opt/foo/sys/opt/ngx_openresty-1.7.0.1.tar.gz-b7f7b683e0d0748db1ec2d66c23959bf5f13cb1471f8e33631d132d0-neo-20140615200651/luajit/lib --pid-path=/var/run/orsty.pid --error-log-path=/var/log/orsty/error.log --http-fastcgi-temp-path=/var/tmp/orsty/fcgi --http-uwsgi-temp-path=/var/tmp/orsty/uwsgi --http-scgi-temp-path=/var/tmp/orsty/scgi --http-client-body-temp-path=/var/tmp/orsty/cbody --http-proxy-temp-path=/var/tmp/orsty/proxy --with-http_ssl_module --with-http_spdy_module --with-http_gzip_static_module --with-http_stub_status_module --with-file-aio --without-http_autoindex_module --without-http_browser_module --without-http_map_module --without-http_ssi_module --without-http_memcached_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --without-http_userid_module --without-http_uwsgi_module --add-module=/opt/foo/pkg/patch/nginx/naxsi-0.53-2/naxsi_src

[agentzh]

@istr Several suggestions:

1. The latest openresty 1.7.2.1 RC1 release contains nginx 1.7.2 which has the syslog thing. You can try that out instead: https://groups.google.com/d/msg/openresty-en/iH3DhYjjf8E/h5o2a9ayEmkJ
2. Please provide the gdb command "bt full" output from for the core dumps. See http://wiki.nginx.org/Debugging#Core_dump
3. It'll be great if you can provide a minimal and self-contained example that can reproduce the issue on others' boxes (on my box, for example).
4. Please check if you can reproduce the issue without the unix domain socket patch in ngx_postgres.

[istr]

Thanks for the hints.
Some additional info: the issue disappears with "postgres_keepalive off" (as you could expect).

1. I will try, thanks a lot.
2. github does not allow for text file attachments, so please have a look here:
   http://www.ingo-struck.de/10622-1404240193.bt-full.txt
   http://www.ingo-struck.de/13055-1404240224.bt-full.txt
   http://www.ingo-struck.de/13076-1404240227.bt-full.txt
   http://www.ingo-struck.de/12935-1404240222.bt-full.txt
   http://www.ingo-struck.de/13069-1404240225.bt-full.txt
   http://www.ingo-struck.de/14792-1404240555.bt-full.txt
3. a little bit difficult to extract, I have a rather sophisticated build script, it is perfectly reproducible, but far away from being "minimal"
4. ok, will check and report the results

[agentzh]

@istr The backtraces you provided are not really useful due to the lack of debug symbols (see all those "No symbol table info available" complaints from gdb?).

Try not to strip your nginx executable (the debug info is enabled by default in both nginx and openresty). Also, better use the -O0 option to compile everything, that is, using --with-cc-opt="-O0" for ./configure.

[istr]

Yes, sorry... The build was a production build, I hoped that the trace would be of some use anyway. I will recompile and produce some useful (complete) backtraces...
FYI: the version is not stripped, but maybe gdb has a problem to find the symbols due to relocation within the chroot-jail:

file $SYS/orsty/nginx/sbin/nginx
/opt/foo/sys/orsty/nginx/sbin/nginx: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=0x2b0ed83a6f27bb3b8c39a0020345795e59e0196c, not stripped

[istr]

Ok, I created backtraces using a binary compiled --with-cc-opt="-O0 -g" (same source tree).
Please re-visit the links above, they contain updated versions with symbols now.

[agentzh]

@istr The connection pool's queue data structure has been corrupted it seems. You can try using valgrind memcheck to identify the culprit (which could be far away from the crashing site). Also, for valgrind memcheck, you need to specify the --with-no-pool-patch option for openresty's ./configure script to disable the nginx memory pool (to make valgrind's life easier).

[agentzh]

@istr I've just fixed a memory issue in commit 088ab19. Could you please try the latest git master branch out on your side? Thank you!

[istr]

• checked out latest version of https://github.com/buddy-ekb/ngx_postgres (w/ unix sockets)
• replaced src/ngx_postgres_keepalive.c with your commit (088ab19)
• segfaults still occur. I am just back from holidays now -- will investigate further next Monday

[agentzh]

@istr Can you reproduce the issue without the (pending) unix domain socket patch? Also, will you try to run your tests with valgrind memcheck + no-pool patch for nginx? Thanks!

[buddy-ekb]

Obviously my unix domain patch is the cause of the issue. The problem can be easily reproduced under the same stress test if connections were made via unix socket.
Unfortunately I can't figure out the reason for it. The only difference seems meaningful to me is a call to function ngx_parse_unix_domain_url inside function ngx_parse_url in ngx_inet.c: if it is done then nginx process will crash after a few requests. On the contrary, if a connection string doesn't contain unix: prefix then everything works like a charm.
Of course, I can provide core dumps and every required piece of information.

[istr]

Ok, it seems that this boils down to a problem specific to:

• unix: domain sockets

in conjunction with

• postgres keepalive connections

The problem disappears if

• using vanilla TCP connections (keepalive works and rotates properly)
• using unix: sockets in conjunction with "postgres_keepalive off"

I will try to run the check with valgrind + no-pool patch to get grips to this.

[buddy-ekb]

@istr right you are!
Finally I managed to track down the root of the issue and commited quick'n'dirty fix into test branch of my fork of this project buddy-ekb/ngx_postgres@9f550ee .
I'd like to hear from @PiotrSikora or @agentzh their opinion about a better way to fix the problem.

[istr]

I would do as nginx does... See my commit comment: buddy-ekb@9f550ee

[buddy-ekb]

@istr, will you consider reopening this issue? I was able to make nginx crash by connection via IPv6 with keep-aliving and in the absence of my patch. To me these crashes are expected because sizeof(struct sockaddr) < sizeof(struct sockaddr_in6) < sizeof(struct sockaddr_un).