[CRASH] Wolfssl segfault

[volga629-1]

Opensips with wolfssl segfaulting in Init.
This is explanation of dev ( not chat GPT :-) true real one )

movaps requires the destination to be 16-byte aligned. If %rbx + 0x480 isn't aligned to 16 bytes, instant SIGSEGV. Since 0x480 is already 16-byte aligned, %rbx itself must be misaligned.

OpenSIPS uses custom memory allocators (pkg_malloc/shm_malloc) that likely return 8-byte aligned memory. wolfSSL 5.8.x is compiled with SSE optimizations that require 16-byte alignment for its internal structures.

OPensips 3.4.11
Wolfssl 5.8.4

Feb  9 21:41:16 cavprx01 opensips[562820]: [OSIP-SBC-1] [562820]: CRITICAL:core:sig_usr: segfault in process pid: 562820, id: 17

Feb  9 21:41:17 cavprx01 opensips[562803]: [OSIP-SBC-1] [562803]: INFO:core:handle_sigs: child process 562820 exited by a signal 11

Feb  9 21:41:17 cavprx01 opensips[562803]: [OSIP-SBC-1] [562803]: INFO:core:handle_sigs: core was generated

Feb  9 21:41:17 cavprx01 opensips[562803]: [OSIP-SBC-1] [562803]: INFO:core:handle_sigs: terminating due to SIGCHLD

Feb  9 21:41:17 cavprx01 opensips[562810]: [OSIP-SBC-1] [562810]: INFO:core:sig_usr: signal 15 received

Feb  9 21:41:17 cavprx01 opensips[562805]: [OSIP-SBC-1] [562805]: INFO:core:sig_usr: signal 15 received

Feb  9 21:41:17 cavprx01 opensips[562808]: [OSIP-SBC-1] [562808]: INFO:core:sig_usr: signal 15 received

Feb  9 21:41:17 cavprx01 opensips[562803]: [OSIP-SBC-1] [562803]: INFO:core:shutdown_opensips: process 2(562805) [RTPEngine notification receiver] terminated, still waiting for 21 more

Feb  9 21:41:17 cavprx01 opensips[562803]: [OSIP-SBC-1] [562803]: INFO:core:shutdown_opensips: process 7(562810) [time_keeper] terminated, still waiting for 20 more

Feb  9 21:41:17 cavprx01 opensips[562803]: [OSIP-SBC-1] [562803]: INFO:core:shutdown_opensips: process 21(562824) [Timer handler] terminated, still waiting for 16 more

Feb  9 21:41:17 cavprx01 opensips[562803]: [OSIP-SBC-1] [562803]: INFO:core:shutdown_opensips: process 22(562825) [Timer handler] terminated, still waiting for 15 more


Feb  9 21:41:17 cavprx01 opensips[562803]: [OSIP-SBC-1] [562803]: INFO:core:shutdown_opensips: process 23(562826) [TCP main] terminated, still waiting for 13 more

Feb  9 21:41:17 cavprx01 opensips[562803]: [OSIP-SBC-1] [562803]: INFO:core:shutdown_opensips: process 4(562807) [MI FIFO] terminated, still waiting for 11 more

Feb  9 21:41:17 cavprx01 opensips[562803]: [OSIP-SBC-1] [562803]: INFO:core:shutdown_opensips: process 20(562823) [TCP receiver] terminated, still waiting for 10 more

Feb  9 21:41:17 cavprx01 opensips[562811]: [OSIP-SBC-1] [562811]: INFO:core:sig_usr: signal 15 received

Feb  9 21:41:17 cavprx01 opensips[562803]: [OSIP-SBC-1] [562803]: INFO:core:shutdown_opensips: process 18(562821) [TCP receiver] terminated, still waiting for 0 more

Feb  9 21:41:17 cavprx01 opensips[562803]: [OSIP-SBC-1] [562803]: INFO:core:cleanup: cleanup

Feb  9 21:41:17 cavprx01 opensips[562803]: [OSIP-SBC-1] [562803]: INFO:tls_wolfssl:mod_destroy: destroying tls_wolfssl module

Feb  9 21:41:17 cavprx01 opensips[562803]: [OSIP-SBC-1] [562803]: NOTICE:event_stream:destroy: destroy module ...


  This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.fedoraproject.org/>
Enable debuginfod for this session? (y or [n]) y
Debuginfod has been enabled.
To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/opensips -P /run/opensips/opensips.pid -f /etc/opensips/opensips.cfg -m 512 -M 128'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f091cd6c395 in InitSSL () from /lib64/libwolfssl.so.44
Missing rpms, try: dnf --enablerepo='*debug*' install opensips-tls-wolfssl-debuginfo-3.4.11-17.fc41.x86_64 wolfssl-debuginfo-5.8.4-11.fc41.x86_64
(gdb) bt
#0  0x00007f091cd6c395 in InitSSL () from /lib64/libwolfssl.so.44
#1  0x00007f091cdaf2cd in wolfSSL_new () from /lib64/libwolfssl.so.44
#2  0x00007f094529030e in _wolfssl_tls_conn_init () from /usr/lib64/opensips/modules/tls_wolfssl.so
#3  0x0000560645611063 in handle_io (fm=<optimized out>, idx=idx@entry=2, event_type=event_type@entry=1) at net/net_tcp_proc.c:253
#4  0x0000560645611aa7 in io_wait_loop_epoll (h=<optimized out>, t=<optimized out>, repeat=<optimized out>) at net/../io_wait_loop.h:305
#5  tcp_worker_proc_loop () at net/net_tcp_proc.c:450
#6  0x0000560645613a92 in tcp_start_processes (chd_rank=0x560645729eb4 <chd_rank>, startup_done=0x7f091d7d16f8) at net/net_tcp.c:2138
#7  0x00005606454736b8 in main_loop () at /usr/src/debug/opensips-3.4.11-17.fc41.x86_64/main.c:243
#8  main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/opensips-3.4.11-17.fc41.x86_64/main.c:966
(gdb)
(gdb)  x/i $rip
=> 0x7f091cd6c395 <InitSSL+773>:        movaps %xmm0,0x480(%rbx)
(gdb) info registers rdi rsi rdx
rdi            0x7f091d7ddda0      139677126221216
rsi            0x7f091d356ca4      139677121473700
rdx            0x0                 0
(gdb) frame 1
#1  0x00007f091cdaf2cd in wolfSSL_new () from /lib64/libwolfssl.so.44
(gdb)  info args
No symbol table info available.
(gdb)
(gdb) info registers rbx
rbx            0x7f091d7dd2d8      139677126218456

Potential fix wolssl.c

# Before wolfSSL_Init() in tls_wolfssl mod_init:
wolfSSL_SetAllocators(malloc, free, realloc);

[razvancrainea]

Please update this report with the information required in the template - that would help us understand your system particularities and help you accordingly.

[github-actions]

Any updates here? No progress has been made in the last 15 days, marking as stale. Will close this issue if no further updates are made in the next 30 days.

[NormB]

tls_wolfssl: fix movaps segfault from shm alignment mismatch

We ran into the same crash described in #3814 and spent some time investigating.
The write-up below documents what we found and a patch that resolved the issue
in our environment (OpenSIPS 4.0.0-dev, wolfSSL 5.7.2, x86_64/linux). We've
tested it fairly extensively -- including 20,000 TLS connections with memory
leak checks -- and it's been solid for us.

That said, your build flags, wolfSSL version, and platform may differ, so this
may or may not apply directly to your situation. We're sharing it in the hope
that it's useful, whether as a direct fix or as a starting point for the
maintainers. Happy to answer questions or adjust anything.

Single file change: modules/tls_wolfssl/wolfssl.c.

---

The problem

As described in #3814, wolfSSL segfaults in InitSSL() when its custom
allocators return memory that isn't 16-byte aligned:

(gdb) x/i $rip
=> 0x7f091c86c6a3:  movaps %xmm0,0x480(%rbx)

(gdb) p/x $rbx
$1 = 0x7f091d7dd2d8       # 8-byte aligned, NOT 16-byte aligned

The movaps instruction needs 16-byte alignment. 0x480 is already aligned,
so %rbx itself is the problem -- shm_malloc returned an address that's only
8-byte aligned.

Why this happens

The module Makefile passes --enable-all to wolfSSL's configure, which turns on
AES-NI on capable x86_64 CPUs. When WOLFSSL_AESNI is defined, wolfSSL sets
WOLFSSL_GENERAL_ALIGNMENT to 16 in wolfssl/wolfcrypt/settings.h:

#ifdef WOLFSSL_AESNI
    #define WOLFSSL_GENERAL_ALIGNMENT 16

This applies __attribute__((aligned(16))) to internal structures via
XGEN_ALIGN, and the compiler emits movaps (aligned move) instead of movups
(unaligned move). Without AES-NI the alignment attribute isn't applied and
there's no crash -- which is why this doesn't reproduce on every build.

OpenSIPS's shm_malloc only guarantees 8-byte alignment (ROUNDTO is
sizeof(void *) on x86_64). There's no shm_memalign() or similar, and we
can't use posix_memalign() because that allocates from the process-private
heap, not shared memory.

The Makefile does have -disable-aligndata but that's both misspelled (missing
a leading dash) and irrelevant -- --disable-aligndata controls padding in
wolfSSL_SetAllocators, not the XGEN_ALIGN attribute that drives movaps
codegen.

Full alignment path in wolfSSL settings.h

#ifndef WOLFSSL_GENERAL_ALIGNMENT
    #ifdef WOLFSSL_AESNI
        #define WOLFSSL_GENERAL_ALIGNMENT 16
    #elif defined(XSTREAM_ALIGN)
        #define WOLFSSL_GENERAL_ALIGNMENT  4
    #else
        #define WOLFSSL_GENERAL_ALIGNMENT  0
    #endif
#endif

When WOLFSSL_GENERAL_ALIGNMENT is 16, wolfSSL marks internal structures with
__attribute__((aligned(16))) via XGEN_ALIGN, and the compiler emits movaps
instructions to zero them. When it's 0, no alignment attribute is applied, and
the compiler uses movups (unaligned) or byte-wise stores instead.

Alternatives considered

Approach	Why we didn't go this route
Fix the --disable-aligndata typo in the Makefile	Even when spelled correctly, this flag controls padding behavior in wolfSSL_SetAllocators, not the XGEN_ALIGN attribute that causes movaps codegen.
Build wolfSSL without AES-NI	Disables hardware AES acceleration -- not a reasonable tradeoff for a TLS library.
Use memalign() / posix_memalign()	These allocate from the process-private heap, not shared memory. The allocations wouldn't be visible across forked OpenSIPS workers.
Add an aligned allocator to OpenSIPS's shm	None of the four allocators (FM, QM, HP, F_PARALLEL) have one. Adding shm_memalign() would be a much larger change across the memory subsystem.
Increase ROUNDTO globally	Would affect every allocation in OpenSIPS and waste memory globally for a wolfSSL-specific need.

OpenSIPS allocator alignment reference

All four OpenSIPS memory allocators were checked:

Allocator	Header	ROUNDTO (x86_64)	Aligned API?
FM (F_MALLOC)	mem/f_malloc.h	sizeof(void *) = 8	No
QM (Q_MALLOC)	mem/q_malloc.h	sizeof(void *) = 8	No
HP (HP_MALLOC)	mem/hp_malloc.h	8UL (hardcoded)	No
F_PARALLEL	mem/f_parallel_malloc.h	8UL (hardcoded)	No

Alignment is enforced by ROUNDUP() macros and __attribute__((aligned(ROUNDTO)))
on fragment header structs. The test suite (mem/test/test_malloc.c) explicitly
asserts 8-byte alignment for every pkg_malloc() and shm_malloc() return.
There is no shm_memalign(), pkg_memalign(), or any function accepting an
alignment parameter anywhere in the codebase.

memalign() / posix_memalign() / aligned_alloc() are not usable here
because they allocate from the process-private heap (brk/mmap MAP_PRIVATE),
not from OpenSIPS's shared memory region (mmap MAP_SHARED or SysV shm).
Memory allocated this way would be copy-on-write after fork() -- each worker
would get its own copy, invisible to the others.

---

The fix

Over-allocate by sizeof(void*) + 15 bytes in the existing oss_malloc /
oss_free / oss_realloc callbacks, align up to the next 16-byte boundary,
and stash the original shm pointer just before the returned address:

[pad 0..15] [void *orig] [aligned user data ...]
                          ^ returned to wolfSSL (16-byte aligned)

Actual per-allocation waste is 8 or 16 bytes (given 8-byte base alignment and
16-byte target). The main wolfSSL structure (WOLFSSL) is ~5 KB so this is
negligible.

Two shared static inline helpers keep the logic in one place across the
debug and non-debug #ifdef variants:

• oss_aligned(raw) -- aligns a fresh allocation, stores the back-pointer.
• oss_realigned(new_raw, off, size) -- re-aligns after shm_realloc.

The tricky part is oss_realigned: after shm_realloc the block may land at
an address with different alignment properties. If the raw-to-aligned offset
changes, the data needs to be memmove'd to the new position. The back-pointer
has to be written after the memmove because the back-pointer slot can overlap
where the data currently sits. Quick example:

• Old: raw=0x108, aligned=0x110, data at 0x110.
• Realloc moves block to 0x100. Data preserved at old offset: 0x108.
• New alignment: aligned=0x110, back-pointer slot = 0x108.
• If we write the back-pointer first, we clobber the data at 0x108.
• So: memmove 0x108 -> 0x110 first, then write back-pointer.

memmove is the standard C library version. OpenSIPS doesn't wrap it -- the
codebase uses memmove() directly throughout.

oss_free also needs a NULL check since we dereference ((void**)ptr)[-1] to
recover the raw pointer. The original shm_free handles NULL, but our wrapper
would crash on it. wolfSSL does pass NULL to free in cleanup paths.

Detailed correctness notes

Three things that are easy to get wrong in this kind of wrapper:

1. oss_free must NULL-check before dereferencing. The original
   shm_free(ptr) handles NULL internally, but our wrapper dereferences
   ((void**)ptr)[-1] to recover the raw pointer. Without the guard,
   free(NULL) segfaults. wolfSSL calls free(NULL) in several cleanup paths
   (wolfSSL_free(), FreeSSL(), error recovery in wolfSSL_new()).

2. oss_realigned must handle alignment shift. After shm_realloc, the
   block may move to an address with different 16-byte alignment properties.
   shm_realloc preserves user data at the old offset within the block, but if
   the new aligned position differs, the data must be memmove'd. Skipping
   this silently corrupts data whenever realloc changes the block's base
   address modulo 16.

3. Back-pointer must be written after memmove, not before. The
   back-pointer slot (aligned[-1]) can overlap the old data position when
   alignment changes direction. Writing it first would corrupt user data before
   memmove copies it to safety.

   Concrete scenario:

   • Old state: raw=0x108, aligned=0x110, offset=8. User data at 0x110.
   • shm_realloc moves the block to 0x100. Data is now at 0x108
     (preserved at the old offset by realloc).
   • New alignment: aligned=0x110, back-pointer slot at 0x108.
   • Writing the back-pointer at 0x108 before memmove destroys the user
     data sitting there.
   • Correct order: memmove 0x108 -> 0x110 first, then write back-pointer
     at 0x108.

---

Patch

--- a/modules/tls_wolfssl/wolfssl.c
+++ b/modules/tls_wolfssl/wolfssl.c
@@ -24,6 +24,8 @@
 #include <errno.h>
 #include <unistd.h>
 #include <netinet/tcp.h>
+#include <stdint.h>
+#include <string.h>

 #include "wolfssl_mem.h"

@@ -139,35 +141,94 @@
 	}
 }

+/*
+ * wolfSSL uses aligned SSE instructions (movaps) requiring 16-byte aligned
+ * memory; shm_malloc() only guarantees 8-byte.  Over-allocate and return an
+ * aligned pointer, stashing the original shm pointer in the slot before it.
+ *
+ *   [pad 0..15] [void *orig] [aligned user data ...]
+ *                             ^ returned to caller
+ */
+#define OSS_ALIGN 16
+#define OSS_PAD   (sizeof(void *) + OSS_ALIGN - 1)
+
+/* Return an OSS_ALIGN-aligned pointer, storing raw just before it */
+static inline void *oss_aligned(void *raw)
+{
+	uintptr_t a = ((uintptr_t)raw + OSS_PAD) & ~(uintptr_t)(OSS_ALIGN - 1);
+	((void **)a)[-1] = raw;
+	return (void *)a;
+}
+
+/* Re-align after shm_realloc.  off is the old raw-to-aligned offset.
+ * memmove before writing back-pointer: the slot may overlap old data. */
+static inline void *oss_realigned(void *new_raw, size_t off, size_t size)
+{
+	uintptr_t a;
+
+	if (!new_raw)
+		return NULL;
+
+	a = ((uintptr_t)new_raw + OSS_PAD) & ~(uintptr_t)(OSS_ALIGN - 1);
+
+	if (a != (uintptr_t)new_raw + off)
+		memmove((void *)a, (char *)new_raw + off, size);
+
+	((void **)a)[-1] = new_raw;
+	return (void *)a;
+}
+
 #ifndef WOLFSSL_DEBUG_MEMORY
 static void *oss_malloc(size_t size)
 {
-	return shm_malloc(size);
+	void *raw = shm_malloc(size + OSS_PAD);
+	return raw ? oss_aligned(raw) : NULL;
 }

 static void oss_free(void *ptr)
 {
-	return shm_free(ptr);
+	if (ptr)
+		shm_free(((void **)ptr)[-1]);
 }

 static void *oss_realloc(void *ptr, size_t size)
 {
-	return shm_realloc(ptr, size);
+	void *raw;
+	size_t off;
+
+	if (!ptr)
+		return oss_malloc(size);
+
+	raw = ((void **)ptr)[-1];
+	off = (char *)ptr - (char *)raw;
+	return oss_realigned(shm_realloc(raw, size + OSS_PAD), off, size);
 }
 #else
 static void *oss_malloc(size_t size, const char* func, unsigned int line)
 {
-	return shm_malloc_func(size, "wolfssl.lib", func, line);
+	void *raw = shm_malloc_func(size + OSS_PAD, "wolfssl.lib", func, line);
+	return raw ? oss_aligned(raw) : NULL;
 }

 static void oss_free(void *ptr, const char* func, unsigned int line)
 {
-	return shm_free_func(ptr, "wolfssl.lib", func, line);
+	if (ptr)
+		shm_free_func(((void **)ptr)[-1], "wolfssl.lib", func, line);
 }

 static void *oss_realloc(void *ptr, size_t size, const char* func, unsigned int line)
 {
-	return shm_realloc_func(ptr, size, "wolfssl.lib", func, line);
+	void *raw;
+	size_t off;
+
+	if (!ptr)
+		return oss_malloc(size, func, line);
+
+	raw = ((void **)ptr)[-1];
+	off = (char *)ptr - (char *)raw;
+	return oss_realigned(
+		shm_realloc_func(raw, size + OSS_PAD, "wolfssl.lib", func, line),
+		off, size);
 }
 #endif

Only wolfssl.c is changed. wolfssl_conn_ops.c, wolfssl_mem.h, and the
rest of the module are untouched. The wolfSSL_SetAllocators() call in
mod_init() stays the same -- the callback signatures haven't changed.

Files not changed and why

• wolfssl_conn_ops.c -- its shm_malloc(sizeof(struct _WOLFSSL)) allocates
  OpenSIPS's own wrapper struct (two pointers), not wolfSSL internals. This
  struct doesn't have alignment attributes and doesn't trigger movaps.
• wolfssl_mem.h -- declares the memory function prototypes. No changes
  needed since the callback signatures are unchanged.
• wolfssl_config.c, wolfssl.h -- unrelated to the allocator path.
• mod_init() -- the wolfSSL_SetAllocators(oss_malloc, oss_free, oss_realloc)
  call is unchanged. The function pointer types match wolfSSL's callback typedefs
  in both the debug and non-debug cases.

---

Testing

Tested on Debian trixie (x86_64), OpenSIPS 4.0.0-dev, wolfSSL 5.7.2 built
with --enable-all --enable-aesni --enable-intelasm.

Crash reproduction (unpatched)

Removed the patch, rebuilt, started OpenSIPS (8 workers). First TLS connection
crashed immediately:

CRITICAL:core:sig_usr: segfault in process pid: 51233, id: 4
INFO:core:handle_sigs: child process 51233 exited by a signal 11

All workers died. Every connection after the first got "Connection refused".

Fix verification (patched)

Applied the patch, rebuilt, started OpenSIPS (9 processes). Ran 20,000
sequential TLS handshakes in two batches of 10,000. All 20,000 succeeded.
Zero crash signals. All processes stable. Clean shutdown.

Memory

Shared memory stats via MI (get_statistics shmem:) at each stage:

Snapshot	used_size	real_used_size	free_size	fragments
Unpatched idle	1,416,984	1,680,656	31,873,776	470
Patched idle	1,417,224	1,680,896	31,873,536	470
After 10,000 TLS	1,417,592	1,681,456	31,872,976	471
After 20,000 TLS	1,417,592	1,681,456	31,872,976	471

The patch adds 240 bytes at startup from the per-allocation padding across
wolfSSL's init-time allocations. No leak -- the 10k and 20k snapshots are
identical. wolfSSL cleans up its per-connection allocations and the wrappers
correctly recover the original shm pointers.

Standalone alignment test (isolates the issue outside OpenSIPS)

A standalone test program was used to confirm the crash is purely an alignment
issue, independent of OpenSIPS's routing or TLS handling:

========================================
  TEST A: MISALIGNED (8-byte, no patch)
========================================
Test: wolfSSL_new() with misaligned allocator (AES-NI: enabled)
  Allocator: 8-byte aligned (NOT 16-byte)
  wolfSSL_Init()...
  wolfSSL_CTX_new()...
  wolfSSL_new() -- triggers InitSSL() with AES-NI movaps...
RESULT: SIGSEGV -- crashed on misaligned memory
Exit code: 1

========================================
  TEST B: ALIGNED (16-byte, with patch)
========================================
Test: wolfSSL_new() with aligned allocator (AES-NI: enabled)
  Allocator: 16-byte aligned
  wolfSSL_Init()...
  wolfSSL_CTX_new()...
  wolfSSL_new() -- triggers InitSSL() with AES-NI movaps...
RESULT: OK -- no crash, ssl=0x55eaf265a770
Exit code: 0

The test registers custom allocators with wolfSSL_SetAllocators() and calls
wolfSSL_new() to trigger the InitSSL() code path that emits movaps. With
8-byte alignment: SIGSEGV. With 16-byte alignment: no crash.

GDB analysis of the crash

Program received signal SIGSEGV, Segmentation fault.
0x00007f091c86c6a3 in InitSSL () from .../tls_wolfssl.so

(gdb) x/i $rip
=> 0x7f091c86c6a3:  movaps %xmm0,0x480(%rbx)

(gdb) p/x $rbx
$1 = 0x7f091d7dd2d8       # 8-byte aligned, NOT 16-byte aligned

0x7f091d7dd2d8 + 0x480 = 0x7f091d7dd758 -- the low nibble 0x8 confirms
misalignment. movaps requires the effective address to end in 0x...0.

Since 0x480 is already 16-byte aligned (ends in 0x0), the fault is entirely
due to %rbx being 8-byte aligned but not 16-byte aligned. %rbx holds the
pointer returned by the custom allocator (shm_malloc via oss_malloc), which
wolfSSL uses as the base of the WOLFSSL structure.