opensips is crashing

[AnjanIpc]

OpenSIPS version you are running

Describe the bug

The below logs are priniting during crash
 CRITICAL:core:sig_usr: segfault in process pid: 11929, id: 4
 CRITICAL:core:handle_worker: dead child 4 (EOF received), pid 11929.
when i BT the core dump, it is carshing at
#0  0x00000000005a0d0f in _tcp_write_on_socket (send_sock=0x7f0e6b536bc8,
    buf=0x7f0e6b549858 "SIP/2.0 200 OK\r\nTo: <sip:10.170.129.10:5061>;tag=5e843462\r\nVia: SIP/2.0/TCP 127.0.0.1:5061;received=10.67.17.2;rport=43438\r\nRecord-Route: <sip:10.170.129.10:5066;r2=on;lr>\r\nRecord-Route: <sip:10.170.1"..., len=388, to=<value optimized out>, id=<value optimized out>)
    at net/proto_tcp/proto_tcp.c:736

To Reproduce

Expected behavior

Relevant System Logs

OS/environment information

• Operating System:
• OpenSIPS installation:
• other relevant information:

Additional context

2 instances of opensips is running. one instance is internal opensips and other instance is external .
opensips. Here the crash has happened on External opensips.
core.11929.gz.

Please help ASAP.

[AnjanIpc]

Can someone please help me here ??

[razvancrainea]

Please provide a full backtrace of the core dump, as well as all the other information this template requires, such as OpenSIPS version, etc. Otherwise we are unable to help you.

[AnjanIpc]

OpenSIPS version: opensips 2.4.4 (x86_64/linux)

BACKTRACE

(gdb) bt
#0  0x00000000005a0d0f in _tcp_write_on_socket (send_sock=0x7f0e6b536bc8,
    buf=0x7f0e6b549858 "SIP/2.0 200 OK\r\nTo: <sip:10.170.129.10:5061>;tag=5e843462\r\nVia: SIP/2.0/TCP 127.0.0.1:5061;received=10.67.17.2;rport=43438\r\nRecord-Route: <sip:10.170.129.10:5066;r2=on;lr>\r\nRecord-Route: <sip:10.170.1"..., len=388, to=<value optimized out>, id=<value optimized out>)
    at net/proto_tcp/proto_tcp.c:736
#1  proto_tcp_send (send_sock=0x7f0e6b536bc8,
    buf=0x7f0e6b549858 "SIP/2.0 200 OK\r\nTo: <sip:10.170.129.10:5061>;tag=5e843462\r\nVia: SIP/2.0/TCP 127.0.0.1:5061;received=10.67.17.2;rport=43438\r\nRecord-Route: <sip:10.170.129.10:5066;r2=on;lr>\r\nRecord-Route: <sip:10.170.1"..., len=388, to=<value optimized out>, id=<value optimized out>)
    at net/proto_tcp/proto_tcp.c:933
#2  0x00000000004375ff in msg_send (send_sock=0x7f0e6b536bc8, proto=2, to=0x7f0e6b549530, id=1127612764,
    buf=0x7f0e6b549858 "SIP/2.0 200 OK\r\nTo: <sip:10.170.129.10:5061>;tag=5e843462\r\nVia: SIP/2.0/TCP 127.0.0.1:5061;received=10.67.17.2;rport=43438\r\nRecord-Route: <sip:10.170.129.10:5066;r2=on;lr>\r\nRecord-Route: <sip:10.170.1"..., len=<value optimized out>, msg=0x7f0e6b5482e0) at forward.h:123
#3  0x0000000000439fd7 in forward_reply (msg=0x7f0e6b5482e0) at forward.c:539
#4  0x0000000000496725 in receive_msg (
    buf=0x85d480 "SIP/2.0 200 OK\r\nTo: <sip:10.170.129.10:5061>;tag=5e843462\r\nVia: SIP/2.0/UDP 10.170.129.10:5066;branch=z9hG4bK8211.d0c8ec670653c9643ca96622ef658bbb.0;i=c5106334\r\nVia: SIP/2.0/TCP 127.0.0.1:5061;receive"..., len=<value optimized out>, rcv_info=<value optimized out>,
    existing_context=<value optimized out>, flags=<value optimized out>) at receive.c:258
#5  0x00000000005a82c6 in udp_read_req (si=<value optimized out>, bytes_read=<value optimized out>) at net/proto_udp/proto_udp.c:181
#6  0x000000000059424b in handle_io (repeat=<value optimized out>, t=<value optimized out>, h=<value optimized out>) at net/net_udp.c:261
#7  io_wait_loop_epoll (repeat=<value optimized out>, t=<value optimized out>, h=<value optimized out>) at net/../io_wait_loop.h:280
#8  0x0000000000596eb5 in udp_start_processes (chd_rank=<value optimized out>, startup_done=0x0) at net/net_udp.c:389
#9  0x0000000000444a55 in main_loop (argc=<value optimized out>, argv=<value optimized out>) at main.c:782
#10 main (argc=<value optimized out>, argv=<value optimized out>) at main.c:1439

Please let me know if u need any other info.

[razvancrainea]

Can you please open the core file in gdb and run the following commands:

(gdb) f 0
(gdb) p tcp_async
(gdb) p c
(gdb) p c->proto_data

[AnjanIpc]

(gdb) f 0
#0 0x00000000005a0d0f in _tcp_write_on_socket (send_sock=0x7f0e6b536bc8,
buf=0x7f0e6b549858 "SIP/2.0 200 OK\r\nTo: sip:10.170.129.10:5061;tag=5e843462\r\nVia: SIP/2.0/TCP 127.0.0.1:5061;received=10.67.17.2;rport=43438\r\nRecord-Route: sip:10.170.129.10:5066;r2=on;lr\r\nRecord-Route: <sip:10.170.1"..., len=388, to=, id=)
at net/proto_tcp/proto_tcp.c:736
736 in net/proto_tcp/proto_tcp.c

(gdb) p tcp_async
$1 = 1
(gdb)

(gdb) p c
$2 = (struct tcp_connection *) 0x7f0e5b888e00
(gdb)

(gdb) p c->proto_data
$3 = (void *) 0x0
(gdb)

[razvancrainea]

can you also print in gdb p *(struct fm_block *)shm_block

[AnjanIpc]

(gdb) p *(struct fm_block *)shm_block
$4 = {name = 0x5d5e11 "shm", size = 268435456, large_space = 264855552, large_limit = 13421770, fragments = 4132, used = 3420448, real_used = 3554768,
max_real_used = 3596080, first_frag = 0x7f0e5b527380, last_frag = 0x7f0e6b51efe8, free_hash = {{first = 0x0, no = 0}, {first = 0x7f0e5b888058, no = 1}, {
first = 0x0, no = 0}, {first = 0x7f0e5b888210, no = 1}, {first = 0x7f0e5b8880a0, no = 16}, {first = 0x7f0e5b883910, no = 6}, {first = 0x0, no = 0}, {
first = 0x7f0e5b871018, no = 1}, {first = 0x0, no = 0}, {first = 0x0, no = 0}, {first = 0x7f0e5b8888e8, no = 2}, {first = 0x0, no = 0}, {first = 0x0,
no = 0}, {first = 0x7f0e5b885748, no = 1}, {first = 0x7f0e5b885980, no = 1}, {first = 0x7f0e5b885810, no = 10}, {first = 0x7f0e5b888d50, no = 2}, {
first = 0x7f0e5b88a668, no = 3}, {first = 0x0, no = 0} <repeats 14 times>, {first = 0x7f0e5b88af70, no = 1}, {first = 0x7f0e5b88b088, no = 1}, {
first = 0x0, no = 0}, {first = 0x0, no = 0}, {first = 0x0, no = 0}, {first = 0x0, no = 0}, {first = 0x0, no = 0}, {first = 0x0, no = 0}, {
first = 0x0, no = 0}, {first = 0x7f0e5b888bf0, no = 1}, {first = 0x0, no = 0}, {first = 0x0, no = 0}, {first = 0x0, no = 0}, {first = 0x0, no = 0}, {
first = 0x7f0e5b88b238, no = 1}, {first = 0x0, no = 0} <repeats 17 times>, {first = 0x7f0e5b88c278, no = 2}, {first = 0x7f0e5b88ac28, no = 1}, {
first = 0x7f0e5b8899f8, no = 1}, {first = 0x7f0e5b88bbf0, no = 1}, {first = 0x7f0e5b88c040, no = 1}, {first = 0x0, no = 0} <repeats 81 times>, {
first = 0x7f0e5b884aa8, no = 1}, {first = 0x0, no = 0} <repeats 334 times>, {first = 0x7f0e5b88c490, no = 1}, {first = 0x0,
no = 0} <repeats 26 times>, {first = 0x7f0e5b883950, no = 2}, {first = 0x0, no = 0} <repeats 39 times>, {first = 0x7f0e5b88d3d0, no = 1}, {
first = 0x0, no = 0} <repeats 1496 times>, {first = 0x7f0e5b871948, no = 1}, {first = 0x0, no = 0} <repeats 12 times>, {first = 0x7f0e5b88e528,
no = 1}, {first = 0x0, no = 0} <repeats 36 times>}}
(gdb)

[razvancrainea]

Unfortunately the logs you have sent me does not really indicate where the problem is.
In order to make sure this hasn't been fixed yet in a newer version, please update your sources to the latest 2.4 release, and let us know if the crash is still happening.

[AnjanIpc]

HI,

Please let me know which all logs you required. I will try to get them.
Actually our product is in Production already and they have seen this issue twice. so they need a RCA and The upgrade might take some time.

so if any logs can help , please let me know,i will try to provide.
Thanks

[razvancrainea]

Well, an upgrade would be mandatory anyway, as any fix would be done, it will be done against the latest 2.4 version. Unless you are maintaining your own source code and will be able to cherry-pick the fix into your 2.4.4 sources, with additional amendments if needed.
That's why I believe an upgrade would be the first start here, to double check this hasn't been sorted out already.

[AnjanIpc]

HI,

What is the latest version of opensips currently available. ?

[AnjanIpc]

can u please let me know , What is the latest version of opensips currently available. ?

[razvancrainea]

Going on the project's website, you will see on the top right side the available versions of OpenSIPS. I would recommend you use OpenSIPS 2.4.7.

[stale]

Any updates here? No progress has been made in the last 15 days, marking as stale. Will close this issue if no further updates are made in the next 30 days.