Segmentation fault on Linux (cli arguments and exiting with s / x commands)

[mtwango]

Some arguments work without problem: -C, -D, -Nname, -P1234, -Otest.txt
Some ends up in Segmentation fault (core dumped) error: -?, -H, -Gsaves/, -Q

When starting with any params, for example like ./SphereSvrX64_nightly -? or ./SphereSvrX64_nightly -Gsaves/ it loads scripts, encryption keys and then dies with Segmentation fault (core dumped) error.

Tested on dev branch (build 3950 / GIT hash 14f74bd)

Debug output from debug build with ASAN an UBSAN:

argument -?, -H and -Q

Loaded 312 client encryption keys.
=================================================================
==2734==ERROR: AddressSanitizer: heap-use-after-free on address 0x60f000049ba0 at pc 0x55db284436b5 bp 0x7ffc43cc2a10 sp 0x7ffc43cc2a08
READ of size 8 at 0x60f000049ba0 thread T0 (T_SphereStartup)
    #0 0x55db284436b4 in CResourceRef::~CResourceRef() src/common/resource/CResourceRef.cpp:34
    #1 0x55db2723ac06 in void std::destroy_at<CResourceRef>(CResourceRef*) /usr/include/c++/12/bits/stl_construct.h:88
    #2 0x55db2723a1c2 in void std::_Destroy<CResourceRef>(CResourceRef*) /usr/include/c++/12/bits/stl_construct.h:149
    #3 0x55db27238e2c in void std::_Destroy_aux<false>::__destroy<CResourceRef*>(CResourceRef*, CResourceRef*) /usr/include/c++/12/bits/stl_construct.h:163
    #4 0x55db272382eb in void std::_Destroy<CResourceRef*>(CResourceRef*, CResourceRef*) /usr/include/c++/12/bits/stl_construct.h:196
    #5 0x55db27236a5f in void std::_Destroy<CResourceRef*, CResourceRef>(CResourceRef*, CResourceRef*, std::allocator<CResourceRef>&) /usr/include/c++/12/bits/alloc_traits.h:850
    #6 0x55db27234ddc in std::vector<CResourceRef, std::allocator<CResourceRef> >::~vector() /usr/include/c++/12/bits/stl_vector.h:730
    #7 0x55db272345fb in CResourceRefArray::~CResourceRefArray() src/common/resource/CResourceRef.h:46
    #8 0x55db2733f5d4 in CRegion::~CRegion() src/game/CRegion.cpp:34
    #9 0x55db2733fa03 in CRegion::~CRegion() src/game/CRegion.cpp:34
    #10 0x55db274e554f in std::default_delete<CResourceDef>::operator()(CResourceDef*) const /usr/include/c++/12/bits/unique_ptr.h:95
    #11 0x55db274ccaae in std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >::~unique_ptr() /usr/include/c++/12/bits/unique_ptr.h:396
    #12 0x55db2750181c in void std::destroy_at<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> > >(std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*) /usr/include/c++/12/bits/stl_construct.h:88
    #13 0x55db274f677f in void std::_Destroy<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> > >(std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*) (/home/mulambo/sphereX/SphereSvrX64_debug+0x283a77f)
    #14 0x55db274eb212 in void std::_Destroy_aux<false>::__destroy<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*>(std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*, std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*) (/home/mulambo/sphereX/SphereSvrX64_debug+0x282f212)
    #15 0x55db274d7b73 in void std::_Destroy<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*>(std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*, std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*) (/home/mulambo/sphereX/SphereSvrX64_debug+0x281bb73)
    #16 0x55db274c2285 in void std::_Destroy<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*, std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> > >(std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*, std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*, std::allocator<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> > >&) (/home/mulambo/sphereX/SphereSvrX64_debug+0x2806285)
    #17 0x55db274b1070 in std::vector<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >, std::allocator<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> > > >::~vector() /usr/include/c++/12/bits/stl_vector.h:730
    #18 0x55db274ac3b5 in sl::sorted_vector<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >, CResourceHashArraySorter>::~sorted_vector() src/common/sphere_library/ssorted_vector.h:41
    #19 0x55db274ac4b9 in sl::_ptr_sorted_vector_base<CResourceDef, std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >, CResourceHashArraySorter>::~_ptr_sorted_vector_base() src/common/sphere_library/sptr_containers.h:65
    #20 0x55db274ac5bd in sl::unique_ptr_sorted_vector<CResourceDef, CResourceHashArraySorter>::~unique_ptr_sorted_vector() src/common/sphere_library/sptr_containers.h:162
    #21 0x55db274ac6c1 in CResourceHashArray::~CResourceHashArray() src/common/resource/CResourceHash.h:19
    #22 0x55db274ac90a in CResourceHash::~CResourceHash() src/common/resource/CResourceHash.h:35
    #23 0x55db274ad058 in CResourceHolder::~CResourceHolder() src/common/resource/CResourceHolder.h:68
    #24 0x55db274404ff in CServerConfig::~CServerConfig() src/game/CServerConfig.cpp:362
    #25 0x7f480a0e555c in __run_exit_handlers stdlib/exit.c:116
    #26 0x7f480a0e5699 in __GI_exit stdlib/exit.c:146
    #27 0x7f480a0ce250 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:74
    #28 0x7f480a0ce304 in __libc_start_main_impl ../csu/libc-start.c:360
    #29 0x55db272234a0 in _start (/home/mulambo/sphereX/SphereSvrX64_debug+0x25674a0)

0x60f000049ba0 is located 0 bytes inside of 168-byte region [0x60f000049ba0,0x60f000049c48)
freed by thread T0 (T_SphereStartup) here:
    #0 0x7f480acea3c8 in operator delete(void*, unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:164
    #1 0x55db28488322 in CRandGroupDef::~CRandGroupDef() src/common/resource/sections/CRandGroupDef.h:36
    #2 0x55db274e554f in std::default_delete<CResourceDef>::operator()(CResourceDef*) const /usr/include/c++/12/bits/unique_ptr.h:95
    #3 0x55db274ccaae in std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >::~unique_ptr() /usr/include/c++/12/bits/unique_ptr.h:396
    #4 0x55db2750181c in void std::destroy_at<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> > >(std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*) /usr/include/c++/12/bits/stl_construct.h:88
    #5 0x55db274f677f in void std::_Destroy<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> > >(std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*) (/home/mulambo/sphereX/SphereSvrX64_debug+0x283a77f)
    #6 0x55db274eb212 in void std::_Destroy_aux<false>::__destroy<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*>(std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*, std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*) (/home/mulambo/sphereX/SphereSvrX64_debug+0x282f212)
    #7 0x55db274d7b73 in void std::_Destroy<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*>(std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*, std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*) (/home/mulambo/sphereX/SphereSvrX64_debug+0x281bb73)
    #8 0x55db274c2285 in void std::_Destroy<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*, std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> > >(std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*, std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*, std::allocator<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> > >&) (/home/mulambo/sphereX/SphereSvrX64_debug+0x2806285)
    #9 0x55db274b1070 in std::vector<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >, std::allocator<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> > > >::~vector() /usr/include/c++/12/bits/stl_vector.h:730
    #10 0x55db274ac3b5 in sl::sorted_vector<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >, CResourceHashArraySorter>::~sorted_vector() src/common/sphere_library/ssorted_vector.h:41
    #11 0x55db274ac4b9 in sl::_ptr_sorted_vector_base<CResourceDef, std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >, CResourceHashArraySorter>::~_ptr_sorted_vector_base() src/common/sphere_library/sptr_containers.h:65
    #12 0x55db274ac5bd in sl::unique_ptr_sorted_vector<CResourceDef, CResourceHashArraySorter>::~unique_ptr_sorted_vector() src/common/sphere_library/sptr_containers.h:162
    #13 0x55db274ac6c1 in CResourceHashArray::~CResourceHashArray() src/common/resource/CResourceHash.h:19
    #14 0x55db274ac90a in CResourceHash::~CResourceHash() src/common/resource/CResourceHash.h:35
    #15 0x55db274ad058 in CResourceHolder::~CResourceHolder() src/common/resource/CResourceHolder.h:68
    #16 0x55db274404ff in CServerConfig::~CServerConfig() src/game/CServerConfig.cpp:362
    #17 0x7f480a0e555c in __run_exit_handlers stdlib/exit.c:116

previously allocated by thread T0 (T_SphereStartup) here:
    #0 0x7f480ace94c8 in operator new(unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:95
    #1 0x55db2747ff0c in CServerConfig::LoadResourceSection(CScript*) src/game/CServerConfig.cpp:3576
    #2 0x55db2842b81a in CResourceHolder::LoadResourcesOpen(CScript*) src/common/resource/CResourceHolder.cpp:202
    #3 0x55db2842c674 in CResourceHolder::LoadResources(CResourceScript*) src/common/resource/CResourceHolder.cpp:225
    #4 0x55db274a033c in CServerConfig::Load(bool) src/game/CServerConfig.cpp:4738
    #5 0x55db2741cad5 in CServer::Load() src/game/CServer.cpp:2690
    #6 0x55db27671158 in Sphere_InitServer(int, char**) src/game/spheresvr.cpp:250
    #7 0x55db276788ea in main src/game/spheresvr.cpp:507
    #8 0x7f480a0ce249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58

SUMMARY: AddressSanitizer: heap-use-after-free src/common/resource/CResourceRef.cpp:34 in CResourceRef::~CResourceRef()
Shadow bytes around the buggy address:
  0x0c1e80001320: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fa
  0x0c1e80001330: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c1e80001340: 00 00 00 00 00 00 00 00 00 00 00 00 00 fa fa fa
  0x0c1e80001350: fa fa fa fa fa fa 00 00 00 00 00 00 00 00 00 00
  0x0c1e80001360: 00 00 00 00 00 00 00 00 00 00 00 fa fa fa fa fa
=>0x0c1e80001370: fa fa fa fa[fd]fd fd fd fd fd fd fd fd fd fd fd
  0x0c1e80001380: fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa
  0x0c1e80001390: fa fa 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c1e800013a0: 00 00 00 00 00 00 00 fa fa fa fa fa fa fa fa fa
  0x0c1e800013b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c1e800013c0: fd fd fd fd fd fd fa fa fa fa fa fa fa fa 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==2734==ABORTING

argument -Gsaves/

Loaded 312 client encryption keys.
Defragmentation (UID alteration) of SphereServer saves.
Use it on your risk and if you know what you are doing since it can possibly harm your server.
The process can take up to several hours depending on the CPU you have.
After finished, you will have your 'sphere*.scp' files converted and saved as 'sphere*.scp.new'.
Reading current UIDs: save/spherestatics.scp
Reading current UIDs: save/sphereworld.scp
Reading current UIDs: save/spherechars.scp
AddressSanitizer:DEADLYSIGNAL
=================================================================
==2729==ERROR: AddressSanitizer: SEGV on unknown address 0x7f7f0689c7fc (pc 0x559ebdbd5eb2 bp 0x7ffda4c40780 sp 0x7ffda4c3fe30 T0)
==2729==The signal is caused by a READ memory access.
    #0 0x559ebdbd5eb2 in defragSphere src/game/CServer.cpp:139
    #1 0x559ebdc07b40 in CServer::CommandLine(int, char**) src/game/CServer.cpp:2374
    #2 0x559ebde66265 in Sphere_InitServer(int, char**) src/game/spheresvr.cpp:256
    #3 0x559ebde6d8ea in main src/game/spheresvr.cpp:507
    #4 0x7f7b17749249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #5 0x7f7b17749304 in __libc_start_main_impl ../csu/libc-start.c:360
    #6 0x559ebda184a0 in _start (/home/mulambo/sphereX/SphereSvrX64_debug+0x25674a0)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV src/game/CServer.cpp:139 in defragSphere
==2729==ABORTING

After exiting sphere with commands s and x:

s
Signal handlers UNinstalled.
16:49:Secure mode disabled.
x
16:50:FATAL:Immediate Shutdown initialized!
=================================================================
==2749==ERROR: AddressSanitizer: heap-use-after-free on address 0x60f0000499c0 at pc 0x55d6a11296b5 bp 0x7ffc7c364c30 sp 0x7ffc7c364c28
READ of size 8 at 0x60f0000499c0 thread T0 (T_Monitor)
    #0 0x55d6a11296b4 in CResourceRef::~CResourceRef() src/common/resource/CResourceRef.cpp:34
    #1 0x55d69ff20c06 in void std::destroy_at<CResourceRef>(CResourceRef*) /usr/include/c++/12/bits/stl_construct.h:88
    #2 0x55d69ff201c2 in void std::_Destroy<CResourceRef>(CResourceRef*) /usr/include/c++/12/bits/stl_construct.h:149
    #3 0x55d69ff1ee2c in void std::_Destroy_aux<false>::__destroy<CResourceRef*>(CResourceRef*, CResourceRef*) /usr/include/c++/12/bits/stl_construct.h:163
    #4 0x55d69ff1e2eb in void std::_Destroy<CResourceRef*>(CResourceRef*, CResourceRef*) /usr/include/c++/12/bits/stl_construct.h:196
    #5 0x55d69ff1ca5f in void std::_Destroy<CResourceRef*, CResourceRef>(CResourceRef*, CResourceRef*, std::allocator<CResourceRef>&) /usr/include/c++/12/bits/alloc_traits.h:850
    #6 0x55d69ff1addc in std::vector<CResourceRef, std::allocator<CResourceRef> >::~vector() /usr/include/c++/12/bits/stl_vector.h:730
    #7 0x55d69ff1a5fb in CResourceRefArray::~CResourceRefArray() src/common/resource/CResourceRef.h:46
    #8 0x55d6a00255d4 in CRegion::~CRegion() src/game/CRegion.cpp:34
    #9 0x55d6a0025a03 in CRegion::~CRegion() src/game/CRegion.cpp:34
    #10 0x55d6a01cb54f in std::default_delete<CResourceDef>::operator()(CResourceDef*) const /usr/include/c++/12/bits/unique_ptr.h:95
    #11 0x55d6a01b2aae in std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >::~unique_ptr() /usr/include/c++/12/bits/unique_ptr.h:396
    #12 0x55d6a01e781c in void std::destroy_at<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> > >(std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*) /usr/include/c++/12/bits/stl_construct.h:88
    #13 0x55d6a01dc77f in void std::_Destroy<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> > >(std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*) (/home/mulambo/sphereX/SphereSvrX64_debug+0x283a77f)
    #14 0x55d6a01d1212 in void std::_Destroy_aux<false>::__destroy<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*>(std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*, std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*) (/home/mulambo/sphereX/SphereSvrX64_debug+0x282f212)
    #15 0x55d6a01bdb73 in void std::_Destroy<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*>(std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*, std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*) (/home/mulambo/sphereX/SphereSvrX64_debug+0x281bb73)
    #16 0x55d6a01a8285 in void std::_Destroy<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*, std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> > >(std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*, std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*, std::allocator<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> > >&) (/home/mulambo/sphereX/SphereSvrX64_debug+0x2806285)
    #17 0x55d6a0197070 in std::vector<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >, std::allocator<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> > > >::~vector() /usr/include/c++/12/bits/stl_vector.h:730
    #18 0x55d6a01923b5 in sl::sorted_vector<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >, CResourceHashArraySorter>::~sorted_vector() src/common/sphere_library/ssorted_vector.h:41
    #19 0x55d6a01924b9 in sl::_ptr_sorted_vector_base<CResourceDef, std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >, CResourceHashArraySorter>::~_ptr_sorted_vector_base() src/common/sphere_library/sptr_containers.h:65
    #20 0x55d6a01925bd in sl::unique_ptr_sorted_vector<CResourceDef, CResourceHashArraySorter>::~unique_ptr_sorted_vector() src/common/sphere_library/sptr_containers.h:162
    #21 0x55d6a01926c1 in CResourceHashArray::~CResourceHashArray() src/common/resource/CResourceHash.h:19
    #22 0x55d6a019290a in CResourceHash::~CResourceHash() src/common/resource/CResourceHash.h:35
    #23 0x55d6a0193058 in CResourceHolder::~CResourceHolder() src/common/resource/CResourceHolder.h:68
    #24 0x55d6a01264ff in CServerConfig::~CServerConfig() src/game/CServerConfig.cpp:362
    #25 0x7f440c94c55c in __run_exit_handlers stdlib/exit.c:116
    #26 0x7f440c94c699 in __GI_exit stdlib/exit.c:146
    #27 0x7f440c935250 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:74
    #28 0x7f440c935304 in __libc_start_main_impl ../csu/libc-start.c:360
    #29 0x55d69ff094a0 in _start (/home/mulambo/sphereX/SphereSvrX64_debug+0x25674a0)

0x60f0000499c0 is located 0 bytes inside of 168-byte region [0x60f0000499c0,0x60f000049a68)
freed by thread T0 (T_Monitor) here:
    #0 0x7f440d5513c8 in operator delete(void*, unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:164
    #1 0x55d6a116e322 in CRandGroupDef::~CRandGroupDef() src/common/resource/sections/CRandGroupDef.h:36
    #2 0x55d6a01cb54f in std::default_delete<CResourceDef>::operator()(CResourceDef*) const /usr/include/c++/12/bits/unique_ptr.h:95
    #3 0x55d6a01b2aae in std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >::~unique_ptr() /usr/include/c++/12/bits/unique_ptr.h:396
    #4 0x55d6a01e781c in void std::destroy_at<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> > >(std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*) /usr/include/c++/12/bits/stl_construct.h:88
    #5 0x55d6a01dc77f in void std::_Destroy<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> > >(std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*) (/home/mulambo/sphereX/SphereSvrX64_debug+0x283a77f)
    #6 0x55d6a01d1212 in void std::_Destroy_aux<false>::__destroy<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*>(std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*, std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*) (/home/mulambo/sphereX/SphereSvrX64_debug+0x282f212)
    #7 0x55d6a01bdb73 in void std::_Destroy<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*>(std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*, std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*) (/home/mulambo/sphereX/SphereSvrX64_debug+0x281bb73)
    #8 0x55d6a01a8285 in void std::_Destroy<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*, std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> > >(std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*, std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >*, std::allocator<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> > >&) (/home/mulambo/sphereX/SphereSvrX64_debug+0x2806285)
    #9 0x55d6a0197070 in std::vector<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >, std::allocator<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> > > >::~vector() /usr/include/c++/12/bits/stl_vector.h:730
    #10 0x55d6a01923b5 in sl::sorted_vector<std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >, CResourceHashArraySorter>::~sorted_vector() src/common/sphere_library/ssorted_vector.h:41
    #11 0x55d6a01924b9 in sl::_ptr_sorted_vector_base<CResourceDef, std::unique_ptr<CResourceDef, std::default_delete<CResourceDef> >, CResourceHashArraySorter>::~_ptr_sorted_vector_base() src/common/sphere_library/sptr_containers.h:65
    #12 0x55d6a01925bd in sl::unique_ptr_sorted_vector<CResourceDef, CResourceHashArraySorter>::~unique_ptr_sorted_vector() src/common/sphere_library/sptr_containers.h:162
    #13 0x55d6a01926c1 in CResourceHashArray::~CResourceHashArray() src/common/resource/CResourceHash.h:19
    #14 0x55d6a019290a in CResourceHash::~CResourceHash() src/common/resource/CResourceHash.h:35
    #15 0x55d6a0193058 in CResourceHolder::~CResourceHolder() src/common/resource/CResourceHolder.h:68
    #16 0x55d6a01264ff in CServerConfig::~CServerConfig() src/game/CServerConfig.cpp:362
    #17 0x7f440c94c55c in __run_exit_handlers stdlib/exit.c:116

previously allocated by thread T0 (T_Monitor) here:
    #0 0x7f440d5504c8 in operator new(unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:95
    #1 0x55d6a0165f0c in CServerConfig::LoadResourceSection(CScript*) src/game/CServerConfig.cpp:3576
    #2 0x55d6a111181a in CResourceHolder::LoadResourcesOpen(CScript*) src/common/resource/CResourceHolder.cpp:202
    #3 0x55d6a1112674 in CResourceHolder::LoadResources(CResourceScript*) src/common/resource/CResourceHolder.cpp:225
    #4 0x55d6a018633c in CServerConfig::Load(bool) src/game/CServerConfig.cpp:4738
    #5 0x55d6a0102ad5 in CServer::Load() src/game/CServer.cpp:2690
    #6 0x55d6a0357158 in Sphere_InitServer(int, char**) src/game/spheresvr.cpp:250
    #7 0x55d6a035e8ea in main src/game/spheresvr.cpp:507
    #8 0x7f440c935249 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58

SUMMARY: AddressSanitizer: heap-use-after-free src/common/resource/CResourceRef.cpp:34 in CResourceRef::~CResourceRef()
Shadow bytes around the buggy address:
  0x0c1e800012e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c1e800012f0: 00 00 00 fa fa fa fa fa fa fa fa fa 00 00 00 00
  0x0c1e80001300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c1e80001310: 00 fa fa fa fa fa fa fa fa fa 00 00 00 00 00 00
  0x0c1e80001320: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fa
=>0x0c1e80001330: fa fa fa fa fa fa fa fa[fd]fd fd fd fd fd fd fd
  0x0c1e80001340: fd fd fd fd fd fd fd fd fd fd fd fd fd fa fa fa
  0x0c1e80001350: fa fa fa fa fa fa 00 00 00 00 00 00 00 00 00 00
  0x0c1e80001360: 00 00 00 00 00 00 00 00 00 00 00 fa fa fa fa fa
  0x0c1e80001370: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c1e80001380: fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==2749==ABORTING

[cbnolok]

So:

• The help or defrag commands could be parsed before the ini and scripts loading, i'll do that.
• Defrag is a code i never messed with, but there's an invalid array memory access if the number of UIDs is 0, so that's a fix to do.
• When you close the server, or you passed some of the commands you mentioned before, Sphere terminates. The crashes occur because of the objects destruction order and pointer references to freed objects. Luckily this a non-issue, because the server has done everything it had to before destroying the world and loaded objects. We can ignore it, we could even brute-force a process termination without cleanup (calling destructors)

[mtwango]

Different error this time. Seems like it is trying to write messages to log, but log is not ready?

root@Mulambo-WS:/home/mulambo/xxx# ./SphereSvrX64_debug -?
Registered thread 'dummy' with ThreadHolder ID 0.
DEBUG:Setting thread (ThreadHolder ID 0, internal name 'dummy') system name: 'T_SphereStartup'.
DEBUG:Registered thread 'T_UnixTerm' with ThreadHolder ID 1.
DEBUG:Setting thread (ThreadHolder ID 1, internal name 'T_UnixTerm') system name: 'T_UnixTerm'.
DEBUG:Started thread 'T_UnixTerm' with ThreadHolder ID 1 and system ID 140343931389632.
SphereServer
Command Line Switches:
-C do not use colored console output (default: on).
-D Dump global variable DEFNAMEs to defs.txt.
-Gpath/to/saves/ Defrags sphere saves.
-Nstring Set the sphere name.
-P# Set the port number.
-Ofilename Output console to this file name
-Q Quit when finished.
AddressSanitizer:DEADLYSIGNAL
=================================================================
==4866==ERROR: AddressSanitizer: SEGV on unknown address 0x000000004800 (pc 0x55b39f8b254e bp 0x7ffc8581f000 sp 0x7ffc8581ef70 T0)
==4866==The signal is caused by a WRITE memory access.
    #0 0x55b39f8b254e in AbstractSphereThread::Strings::allocateBuffer() /home/mulambo/Source-X-linus/src/sphere/threads.cpp:856
    #1 0x55b39f99c570 in Str_GetTemp() /home/mulambo/Source-X-linus/src/common/sphere_library/sstringobjs.cpp:54
    #2 0x55b39f410396 in CEventLog::VEvent(unsigned int, char const*, ConsoleTextColor, __va_list_tag*) /home/mulambo/Source-X-linus/src/common/CLog.cpp:20
    #3 0x55b39f4107c4 in CEventLog::Event(unsigned int, char const*, ...) /home/mulambo/Source-X-linus/src/common/CLog.cpp:37
    #4 0x55b39f8a6cb1 in ThreadHolder::markThreadsClosing() /home/mulambo/Source-X-linus/src/sphere/threads.cpp:293
    #5 0x55b39e7f4896 in atexit_handler() (/home/mulambo/xxx/SphereSvrX64_debug+0x2a58896)
    #6 0x7fa46102e55c in __run_exit_handlers stdlib/exit.c:116
    #7 0x7fa46102e699 in __GI_exit stdlib/exit.c:146
    #8 0x7fa461017250 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:74
    #9 0x7fa461017304 in __libc_start_main_impl ../csu/libc-start.c:360
    #10 0x55b39e3884e0 in _start (/home/mulambo/xxx/SphereSvrX64_debug+0x25ec4e0)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /home/mulambo/Source-X-linus/src/sphere/threads.cpp:856 in AbstractSphereThread::Strings::allocateBuffer()
==4866==ABORTING

[cbnolok]

Should be fine now

[mtwango]

No errors now, but -? and -H doesn't return anything.

Using -Gsave/ ends hanging with 100% CPU thread usage (even with actual save with data) on Linux. On Windows it ends fine:

./SphereSvrX64_debug -Gsave/
Registered thread 'dummy' with ThreadHolder ID 0.
DEBUG:Setting thread (ThreadHolder ID 0, internal name 'dummy') system name: 'T_SphereStartup'.
Defragmentation (UID alteration) of SphereServer saves.
Use it on your risk and if you know what you are doing since it can possibly harm your server.
The process can take up to several hours depending on the CPU you have.
After finished, you will have your 'sphere*.scp' files converted and saved as 'sphere*.scp.new'.
DEBUG:Registered thread 'T_UnixTerm' with ThreadHolder ID 1.
DEBUG:Setting thread (ThreadHolder ID 1, internal name 'T_UnixTerm') system name: 'T_UnixTerm'.
DEBUG:Started thread 'T_UnixTerm' with ThreadHolder ID 1 and system ID 140100420740800.
Reading current UIDs: save/spherestatics.scp
Reading current UIDs: save/sphereworld.scp
Reading current UIDs: save/spherechars.scp
Totally having 0 unique objects (UIDs). Aborting defrag.

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 2413 root      20   0   20.0t 145388  55084 S 100.0   0.4   2:55.16 T_SphereStartup

I think -G switch should have separate issue, since there is more problems with it (doesn't sort spheremultis, check for accounts file in save folder)

09:08:Totally having 4284490 unique objects (UIDs), latest: 04029c40e
09:08:Quick-Sorting the UIDs array...
09:08:Updating UID-s in save/sphereaccu.scp to save/sphereaccu.scp.new
09:08:Cannot open file for reading. Skipped!
09:08:Updating UID-s in save/spherechars.scp to save/spherechars.scp.new

[cbnolok]

For sure defrag should be thoroughly checked and eventually fixed.
For -H, -?, here i use zsh on Linux and it works. Are you talking about Windows?