opensips-2.3.0 load tls_mgm.so failed (tls_mgm:mod_init: unable to set the memory allocation functions)

[mach1997]

1 runtime

os version
uname -a
centos6.5 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

openssl version
OpenSSL 1.0.1e-fips 11 Feb 2013

opensips version
 ./opensips -V
version: opensips 2.3.0 (x86_64/linux)
flags: STATS: On, EXTRA_DEBUG, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
git revision: 377b7c7a1
main.c compiled on 02:06:33 May  3 2017 with gcc 4.4.7

opensips start cmdline
./opensips -M 8 -f ../custom.cfg

load related module:

loadmodule "signaling.so"
loadmodule "sl.so"
loadmodule "tm.so"
loadmodule "rr.so"
loadmodule "maxfwd.so"
loadmodule "sipmsgops.so"
loadmodule "mi_fifo.so"
loadmodule "textops.so"
loadmodule "mathops.so"
loadmodule "json.so"
loadmodule "statistics.so"
loadmodule "tls_mgm.so"
loadmodule "proto_udp.so"
loadmodule "proto_tcp.so"
loadmodule "proto_ws.so"
loadmodule "proto_wss.so"
loadmodule "proto_hep.so"
loadmodule "db_mysql.so"
loadmodule "avpops.so"
loadmodule "uri.so"
loadmodule "usrloc.so"
loadmodule "registrar.so"
loadmodule "presence.so"
loadmodule "acc.so"
loadmodule "rest_client.so"
loadmodule "auth.so"
loadmodule "uac_auth.so"
loadmodule "auth_db.so"
loadmodule "alias_db.so"
loadmodule "cachedb_local.so"
loadmodule "cachedb_redis.so"
loadmodule "cachedb_mongodb.so"
loadmodule "msilo.so"

2.tls_mgm init error

INFO:tls_mgm:mod_init: initializing TLS protocol
May  3 11:26:39 [2164] ERROR:tls_mgm:mod_init: unable to set the memory allocation functions
May  3 11:26:39 [2164] ERROR:tls_mgm:mod_init: NOTE: check if you are using openssl 1.0.1e-fips, (or other FIPS version of openssl, as this is known to be broken; if so, you need to upgrade or downgrade to a different openssl version!

3. replace openssl version

3.1
openssl Compile Options
./config -g3 shared zlib-dynamic --prefix=/
try openssl-1.0.1t，openssl-1.0.2h，openssl-1.0.1q and so on
replace openssl verion (rm -rf xxx & ln -s xxxx)
recompile openssl-2.3.0 (make clean&make&make install)
./opensips -M 8 -f ../custom.cfg
init tls_mgm error

INFO:tls_mgm:mod_init: initializing TLS protocol
May  3 11:26:39 [2164] ERROR:tls_mgm:mod_init: unable to set the memory allocation functions
May  3 11:26:39 [2164] ERROR:tls_mgm:mod_init: NOTE: check if you are using openssl 1.0.1e-fips, (or other FIPS version of openssl, as this is known to be broken; if so, you need to upgrade or downgrade to a different openssl version!
May  3 11:26:39 [2164] ERROR:tls_mgm:mod_init: current version: OpenSSL 1.0.1t  3 May 2016
May  3 11:26:39 [2164] ERROR:core:init_mod: failed to initialize module tls_mgm

3.2
try openssl Compile Options(fips)
./config -g3 fips shared zlib-dynamic --prefix=/
try openssl-1.0.1t，openssl-1.0.2h，openssl-1.0.1q and so on
replace openssl verion (rm -rf xxx & ln -s xxxx)
recompile openssl-2.3.0 (make clean&make&make install)
./opensips -M 8 -f ../custom.cfg
init tls_mgm error

INFO:tls_mgm:mod_init: initializing TLS protocol
May  3 11:26:39 [2164] ERROR:tls_mgm:mod_init: unable to set the memory allocation functions
May  3 11:26:39 [2164] ERROR:tls_mgm:mod_init: NOTE: check if you are using openssl 1.0.1e-fips, (or other FIPS version of openssl, as this is known to be broken; if so, you need to upgrade or downgrade to a different openssl version!
May  3 11:26:39 [2164] ERROR:tls_mgm:mod_init: current version: OpenSSL 1.0.1t  3 May 2016
May  3 11:26:39 [2164] ERROR:core:init_mod: failed to initialize module tls_mgm

4 gdb opensips

 gdb ./opensips
 (gdb) set args -M 8 -f -f ../custom.cfg
 (gdb) b CRYPTO_set_mem_functions
 (gdb) r
 ...............................................................................
 (gdb) p allow_customize
 (gdb) $1=0
 (gdb) watch allow_customize
 (gdb) r
 restart gdb-opensips living-step
 
 _gdb-opensips log below:_
May  3 02:14:24 [30845] DBG:core:set_mod_param_regex: found <query_timeout> in module cachedb_redis [/xxxxxxx/opensips-2.3.0//lib64/opensips/modules/]
May  3 02:14:24 [30845] DBG:core:load_module: loading module /xxxxxxxx/opensips-2.3.0//lib64/opensips/modules/cachedb_mongodb.so
Hardware watchpoint 2: allow_customize

Old value = 1
New value = 0
0x00007ffff3aebab2 in CRYPTO_malloc (num=176, file=0x7ffff3bff58a "lhash.c", line=120) at mem.c:336
336	        allow_customize = 0;
(gdb) bt
#0  0x00007ffff3aebab2 in CRYPTO_malloc (num=176, file=0x7ffff3bff58a "lhash.c", line=120) at mem.c:336
#1  0x00007ffff3b71140 in lh_new (h=0x7ffff3aeec40 <obj_name_LHASH_HASH>, c=0x7ffff3aeebb0 <obj_name_LHASH_COMP>) at lhash.c:120
#2  0x00007ffff3aeee3d in OBJ_NAME_init () at o_names.c:61
#3  0x00007ffff3aeef15 in OBJ_NAME_add (name=0x7ffff3be6991 "DES-CBC", type=2, data=0x7ffff3e4de00 "\037") at o_names.c:185
#4  0x00007ffff3b7d447 in EVP_add_cipher (c=0x7ffff3e4de00) at names.c:74
#5  0x00007ffff3eb0891 in SSL_library_init () at ssl_algs.c:68
#6  0x00007ffff061354f in _mongoc_ssl_init () from /usr/local/lib/libmongoc-1.0.so.0
#7  0x00007ffff0605ad9 in _mongoc_do_init () from /usr/local/lib/libmongoc-1.0.so.0
#8  0x0000003f20c0ce03 in pthread_once () from /lib64/libpthread.so.0
#9  0x00007ffff06136b6 in __do_global_ctors_aux () from /usr/local/lib/libmongoc-1.0.so.0
#10 0x00007ffff05f7e63 in _init () from /usr/local/lib/libmongoc-1.0.so.0
#11 0x0000007c0000005b in ?? ()
#12 0x0000003f2000e705 in _dl_init_internal () from /lib64/ld-linux-x86-64.so.2
#13 0x0000003f20012f75 in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#14 0x0000003f2000e366 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#15 0x0000003f2001271a in _dl_open () from /lib64/ld-linux-x86-64.so.2
#16 0x0000003f20800f66 in dlopen_doit () from /lib64/libdl.so.2
#17 0x0000003f2000e366 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#18 0x0000003f2080129c in _dlerror_run () from /lib64/libdl.so.2
#19 0x0000003f20800ee1 in dlopen@@GLIBC_2.2.5 () from /lib64/libdl.so.2
#20 0x00000000004caf35 in sr_load_module (path=0x87a6c0 "/aigongzuo/opensips-2.3.0//lib64/opensips/modules/cachedb_mongodb.so") at sr_module.c:231
#21 0x00000000004cb6fd in load_module (name=<value optimized out>) at sr_module.c:377
#22 0x00000000005be494 in yyparse () at cfg.y:1101
#23 0x0000000000442e1c in main (argc=<value optimized out>, argv=0x7fffffffe4b8) at main.c:1018

4 remove cachedb_mongodb.so

remove cachedb_mongodb.so&mongodb related route logic

#6  0x00007ffff061354f in _mongoc_ssl_init () from /usr/local/lib/libmongoc-1.0.so.0
#7  0x00007ffff0605ad9 in _mongoc_do_init () from /usr/local/lib/libmongoc-1.0.so.0

mongoc-driver-libs related ssl maybe corrupt tls_mgm.so

version:mongo-c-driver-1.1.0
libmongoc-ssl-1.0.pc
prefix=/usr/local
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${exec_prefix}/include

Name: libmongoc-1.0
Description: SSL support for the libmongoc-1.0 library.
Version: 1.1.0
Requires: libmongoc-1.0
Libs:
Cflags:

libmongoc-1.0.pc 
prefix=/usr/local
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${exec_prefix}/include
Name: libmongoc
Description: The libmongoc MongoDB client library.
Version: 1.1.0
Requires: libbson-1.0
Libs:  -lssl -lcrypto   -lrt -L${libdir} -lmongoc-1.0
Cflags: -I${includedir}/libmongoc-1.0

after remove cachedb_mongodb.so,opensips-2.3.0 successfully start (include tls_mgm.so) above

openssl 1.0.1e and below openssl-1.0.2i
openssl compile option not include fips flags 
Compile Options
./config -g3 shared zlib-dynamic --prefix=/ 

5 bug?

at present adjust cache-store(redis), don't use mongo;
the tls_mgm.c error-hint look like some mistake;
cache_mongo&tls_mgm modules corrupts related on ssl funnction?

[razvancrainea]

Indeed, there seems to be a problem with the tls_mgm module and the mongo library. Basically the mongo library initializes the SSL library before the tls_mgm gets to set custom allocation modules. This breaks openssl init for tls_mgm.

Any chance you could use mongodb without SSL support (I think it is possible if you compile using ./configure --enable-ssl=no)?

[Jeffrey2019]

Dear razvancrainea
We met the same issue, we use tls_mgm module with postgres database.
5 months passed, will there a fix, or should we try to find a workaround.
Is there a workaround?

Thanks.

[razvancrainea]

This is not something we can fix in OpenSIPS, at least not in an easy way. The only thing I can think of right now is to link opensips binary with TLS, but that seems completely broken.

A workaround is to compile your postgres library without SSL support.

[Jeffrey2019]

Dear razvancrainea
Thanks for the reply.
After Oracle acquired MySQL, lots of companies migrated database from MySQL to Postgres, including us.
Unfortunately, I believe it is very hard to influence postgres community to fix the issue.

FreeSWITCH has native Postgres database support, meanwhile it has SIP TLS support, they work fine together.
But FreeSWITCH is a single process application using multiple threads, OpenSIPS is multiple processes application.
Can you kindly have a look of FreeSWITCH source code? Maybe there is something helpful.

BTW, is it possible to compile postgres library without SSL if postgres and OPENSIPS installed on the same server?
Is there instructions to compile postgres library without SSL?

Thank you very much!

Best regards.
Jeffrey.

[razvancrainea]

Hi, Jeffrey!

Thanks for your suggestion; unfortunately OpenSIPS uses an OpenSSL feature that FreeSWITCH doesn't, which is exactly what breaks this module (setting the allocation functions using CRYPTO_set_mem_functions).
Can you tell me what version of postgres are you using?

You can use these steps to compile libpq from sources, but make sure you are not using the --with-openssl parameter.

[Jeffrey2019]

@razvancrainea Does the above commit fixed the issue? If yes, which LTS version include the fix?
Thank you very much for the effort.

[razvancrainea]

@Jeffrey2019 yes, this has been committed in 2.4 and upper. 2.4.6 is LTS, so if you are using 2.4, you should upgrade to the latest release to get this fix. Cheers!

[jalung]

Hi, loading tls_mgm as the first module on startup may fix one issue, but the same message appears if the version of openssl on the system is incompatible with tls_mgm. For those in the latter situation, upgrading to a newer opensips isn't the solution. The likely solution is to resolve the openssl library issues.

After upgrading the underlying linux system, opensips(2.3.6) failed to start with the following message:

:tls_mgm:mod_init: unable to set the memory allocation functions
ERROR:tls_mgm:mod_init: NOTE: check if you are using openssl 1.0.1e-fips

The information below contains some information that may be applicable to other users in their quest to solve this problem.

The error messsage indicates openssl library incompatibity after upgrade. Module tls_mgm requires a compatible openssl 1.0.1x library and the distro now had openssl 1.1.0x. Included in the distro is a package "compat-openssl10" which is automatically installed. It is supposed to allow openssl 1.0.1x compatible programs to still function. However, it was not working for tls_mgm, and after recompiling opensips, tls_mgm was including the system's openssl 1.1 headers and libs and not the 1.0 compat.

To solve this, because downgrading anything was not an option, "compat-openssl110" was uninstalled and it dragged mongodb and a few other packages along with it. Mongodb was a required package, so instead of using mongodb from the official repository, disabling the mongodb repo and installing the distro version of mongodb fixed that issue. The package maintainer's version of mongo used openssl 1.1

For tls_mgm the solution was to maintain a copy of openssl 1.0.1o on the server.

#openssl source
$ cd openssl-1.0.2o

#local openssl options: prefix is important (must not be found by ldconfig)
$ ./config --prefix=/opt/local --openssldir=/opt/local/openssl \
    enable-ec_nistp_64_gcc_128 \
    zlib sctp enable-camellia enable-seed enable-tlsext enable-rfc3779 \
    enable-cms enable-md2 enable-rc5 \
    no-mdc2 no-ec2m no-gost no-srp no-krb5 \
    shared

$ make depend
$ make test
$ make install

#opensips source
$ export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig"
$ cd opensips-000
$ make menuconfig
$ make
$ make install

# check the module for openssl version 1.0
$ ldd /lib64/opensips/modules/tls_mgm.so
	linux-vdso.so.1 (0x00007ffc0859f000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f905d265000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f905d04e000)
	libsctp.so.1 => /lib64/libsctp.so.1 (0x00007f905ce4b000)
	libssl.so.10 => not found
	libcrypto.so.10 => not found
	libc.so.6 => /lib64/libc.so.6 (0x00007f905ca8d000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f905d692000)

# tell opensips where to find it on startup
# for systemd contents of /etc/sysconfig/opensips

	LD_LIBRARY_PATH=/opt/local/lib

	# Any additional OpenSIPS options
	OPTIONS=""

# systemctl start opensips