Skip to content

Improve crashlog #8373

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 19 commits into from
Jun 29, 2021
Merged

Improve crashlog #8373

merged 19 commits into from
Jun 29, 2021

Conversation

julianbrost
Copy link
Contributor

@julianbrost julianbrost commented Oct 14, 2020

The current crash handlers show some shortcomings, especially on Windows. For example, if an uncaught C++ exception occurs on Windows, currently the SEH handler is executed which prints no information about the actual exception. Also the stacktrace printed on Windows is of little use for release builds as it only prints (unknown function) without even an address. This is addressed by using boost instead of the custom stacktrace implementation.

New Crashlog Examples

Windows, Debug Build, SEH Exception
Caught unhandled SEH exception.
Current time: 2020-10-21 16:59:56 +0200

  Application version: v2.12.0-87-g5806c8985

System information:
  Platform: Windows
  Platform version: 8
  Kernel: Windows
  Kernel version: 6.2
  Architecture: x86_64

Build information:
  Compiler: MSVC 19.27.29112.0
  Build host: DESKTOP-ELL2TUA
  OpenSSL version: OpenSSL 1.1.1h  22 Sep 2020

Application information:

General paths:
  Config directory: C:\ProgramData\icinga2\etc\icinga2
  Data directory: C:\ProgramData\icinga2\var\lib\icinga2
  Log directory: C:\ProgramData\icinga2\var\log\icinga2
  Cache directory: C:\ProgramData\icinga2\var\cache\icinga2
  Spool directory: C:\ProgramData\icinga2\var\spool\icinga2
  Run directory: C:\ProgramData\icinga2\var\run\icinga2

Old paths (deprecated):
  Installation root: C:\Program Files\ICINGA2-debug\
  Sysconf directory: C:\ProgramData\icinga2\etc
  Run directory (base): C:\ProgramData\icinga2\var\run
  Local state directory: C:\ProgramData\icinga2\var

Internal paths:
  Package data directory: C:\Program Files\ICINGA2-debug\\share\icinga2
  State path: C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state
  Modified attributes path: C:\ProgramData\icinga2\var\lib\icinga2/modified-attributes.conf
  Objects path: C:\ProgramData\icinga2\var\cache\icinga2/icinga2.debug
  Vars path: C:\ProgramData\icinga2\var\cache\icinga2/icinga2.vars
  PID path: C:\ProgramData\icinga2\var\run\icinga2/icinga2.pid

SEH exception:
  Code: 0x2a
  Address: 00007FFFB3313E49
  Flags: 0

Stacktrace:
 0# boost::stacktrace::basic_stacktrace<std::allocator<boost::stacktrace::frame> >::init at C:\Local\boost_1_71_0\boost\stacktrace\stacktrace.hpp:75
 1# boost::stacktrace::basic_stacktrace<std::allocator<boost::stacktrace::frame> >::basic_stacktrace<std::allocator<boost::stacktrace::frame> > at C:\Local\boost_1_71_0\boost\stacktrace\stacktrace.hpp:127
 2# icinga::Application::SEHUnhandledExceptionFilter at C:\Users\Julian Brost\source\repos\icinga2\lib\base\application.cpp:959
 3# UnhandledExceptionFilter in KERNELBASE
 4# memset in ntdll
 5# _C_specific_handler in ntdll
 6# _chkstk in ntdll
 7# RtlRaiseException in ntdll
 8# RtlRaiseException in ntdll
 9# RaiseException in KERNELBASE
10# throw_test_exception at C:\Users\Julian Brost\source\repos\icinga2\lib\cli\daemoncommand.cpp:225
11# RunWorker at C:\Users\Julian Brost\source\repos\icinga2\lib\cli\daemoncommand.cpp:305
12# icinga::DaemonCommand::Run at C:\Users\Julian Brost\source\repos\icinga2\lib\cli\daemoncommand.cpp:701
13# Main at C:\Users\Julian Brost\source\repos\icinga2\icinga-app\icinga.cpp:660
14# main at C:\Users\Julian Brost\source\repos\icinga2\icinga-app\icinga.cpp:957
15# invoke_main at D:\agent\_work\9\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:79
16# __scrt_common_main_seh at D:\agent\_work\9\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
17# __scrt_common_main at D:\agent\_work\9\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:331
18# mainCRTStartup at D:\agent\_work\9\s\src\vctools\crt\vcstartup\src\startup\exe_main.cpp:17
19# BaseThreadInitThunk in KERNEL32
20# RtlUserThreadStart in ntdll

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***
Windows, Debug Build, C++ Exception
Caught unhandled exception.
Current time: 2020-10-21 17:00:20 +0200

  Application version: v2.12.0-87-g5806c8985

System information:
  Platform: Windows
  Platform version: 8
  Kernel: Windows
  Kernel version: 6.2
  Architecture: x86_64

Build information:
  Compiler: MSVC 19.27.29112.0
  Build host: DESKTOP-ELL2TUA
  OpenSSL version: OpenSSL 1.1.1h  22 Sep 2020

Application information:

General paths:
  Config directory: C:\ProgramData\icinga2\etc\icinga2
  Data directory: C:\ProgramData\icinga2\var\lib\icinga2
  Log directory: C:\ProgramData\icinga2\var\log\icinga2
  Cache directory: C:\ProgramData\icinga2\var\cache\icinga2
  Spool directory: C:\ProgramData\icinga2\var\spool\icinga2
  Run directory: C:\ProgramData\icinga2\var\run\icinga2

Old paths (deprecated):
  Installation root: C:\Program Files\ICINGA2-debug\
  Sysconf directory: C:\ProgramData\icinga2\etc
  Run directory (base): C:\ProgramData\icinga2\var\run
  Local state directory: C:\ProgramData\icinga2\var

Internal paths:
  Package data directory: C:\Program Files\ICINGA2-debug\\share\icinga2
  State path: C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state
  Modified attributes path: C:\ProgramData\icinga2\var\lib\icinga2/modified-attributes.conf
  Objects path: C:\ProgramData\icinga2\var\cache\icinga2/icinga2.debug
  Vars path: C:\ProgramData\icinga2\var\cache\icinga2/icinga2.vars
  PID path: C:\ProgramData\icinga2\var\run\icinga2/icinga2.pid

Error: test exception

Stacktrace:
 0# boost::stacktrace::basic_stacktrace<std::allocator<boost::stacktrace::frame> >::init at C:\Local\boost_1_71_0\boost\stacktrace\stacktrace.hpp:75
 1# boost::stacktrace::basic_stacktrace<std::allocator<boost::stacktrace::frame> >::basic_stacktrace<std::allocator<boost::stacktrace::frame> > at C:\Local\boost_1_71_0\boost\stacktrace\stacktrace.hpp:127
 2# `icinga::Application::ExceptionHandler'::`1'::catch$9 at C:\Users\Julian Brost\source\repos\icinga2\lib\base\application.cpp:889
 3# 0x00007FFFA2FF1080 in VCRUNTIME140_1D
 4# _NLG_Return2 in VCRUNTIME140_1D
 5# RtlCaptureContext2 in ntdll
 6# icinga::Application::ExceptionHandler at C:\Users\Julian Brost\source\repos\icinga2\lib\base\application.cpp:874
 7# terminate in ucrtbased
 8# __scrt_unhandled_exception_filter at D:\agent\_work\9\s\src\vctools\crt\vcstartup\src\utility\utility_desktop.cpp:96
 9# icinga::Application::SEHUnhandledExceptionFilter at C:\Users\Julian Brost\source\repos\icinga2\lib\base\application.cpp:917
10# UnhandledExceptionFilter in KERNELBASE
11# memset in ntdll
12# _C_specific_handler in ntdll
13# _chkstk in ntdll
14# RtlRaiseException in ntdll
15# RtlRaiseException in ntdll
16# RaiseException in KERNELBASE
17# CxxThrowException in VCRUNTIME140D
18# throw_test_exception at C:\Users\Julian Brost\source\repos\icinga2\lib\cli\daemoncommand.cpp:221
19# RunWorker at C:\Users\Julian Brost\source\repos\icinga2\lib\cli\daemoncommand.cpp:305
20# icinga::DaemonCommand::Run at C:\Users\Julian Brost\source\repos\icinga2\lib\cli\daemoncommand.cpp:701
21# Main at C:\Users\Julian Brost\source\repos\icinga2\icinga-app\icinga.cpp:660
22# main at C:\Users\Julian Brost\source\repos\icinga2\icinga-app\icinga.cpp:957
23# invoke_main at D:\agent\_work\9\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:79
24# __scrt_common_main_seh at D:\agent\_work\9\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
25# __scrt_common_main at D:\agent\_work\9\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:331
26# mainCRTStartup at D:\agent\_work\9\s\src\vctools\crt\vcstartup\src\startup\exe_main.cpp:17
27# BaseThreadInitThunk in KERNEL32
28# RtlUserThreadStart in ntdll

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***
Windows, Snapshot Build, SEH Exception
Caught unhandled SEH exception.
Current time: 2020-10-21 14:47:30 -0000

  Application version: v2.12.0-87-g5806c8985

System information:
  Platform: Windows
  Platform version: 8 (Server)
  Kernel: Windows
  Kernel version: 6.2
  Architecture: x86_64

Build information:
  Compiler: MSVC 19.26.28805.0
  Build host: WINBUILD2
  OpenSSL version: OpenSSL 1.1.1c  28 May 2019

Application information:

General paths:
  Config directory: C:\ProgramData\icinga2\etc\icinga2
  Data directory: C:\ProgramData\icinga2\var\lib\icinga2
  Log directory: C:\ProgramData\icinga2\var\log\icinga2
  Cache directory: C:\ProgramData\icinga2\var\cache\icinga2
  Spool directory: C:\ProgramData\icinga2\var\spool\icinga2
  Run directory: C:\ProgramData\icinga2\var\run\icinga2

Old paths (deprecated):
  Installation root: C:\Program Files\ICINGA2\
  Sysconf directory: C:\ProgramData\icinga2\etc
  Run directory (base): C:\ProgramData\icinga2\var\run
  Local state directory: C:\ProgramData\icinga2\var

Internal paths:
  Package data directory: C:\Program Files\ICINGA2\\share\icinga2
  State path: C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state
  Modified attributes path: C:\ProgramData\icinga2\var\lib\icinga2/modified-attributes.conf
  Objects path: C:\ProgramData\icinga2\var\cache\icinga2/icinga2.debug
  Vars path: C:\ProgramData\icinga2\var\cache\icinga2/icinga2.vars
  PID path: C:\ProgramData\icinga2\var\run\icinga2/icinga2.pid

SEH exception:
  Code: 0x2a
  Address: 00007FF8657C5299
  Flags: 0

Stacktrace:
 0# 0x00007FF7CB225D42 in icinga2
 1# 0x00007FF7CB1E6477 in icinga2
 2# UnhandledExceptionFilter in KERNELBASE
 3# memset in ntdll
 4# _C_specific_handler in ntdll
 5# _chkstk in ntdll
 6# RtlWalkFrameChain in ntdll
 7# RtlRaiseException in ntdll
 8# RaiseException in KERNELBASE
 9# 0x00007FF7CB428E5E in icinga2
10# 0x00007FF7CB414828 in icinga2
11# 0x00007FF7CB3F012F in icinga2
12# 0x00007FF7CB0F4027 in icinga2
13# 0x00007FF7CB0FC3A1 in icinga2
14# 0x00007FF7CB70D0DC in icinga2
15# BaseThreadInitThunk in KERNEL32
16# RtlUserThreadStart in ntdll

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***
Windows, Snapshot Build, C++ Exception
Caught unhandled exception.
Current time: 2020-10-21 14:47:41 -0000

  Application version: v2.12.0-87-g5806c8985

System information:
  Platform: Windows
  Platform version: 8 (Server)
  Kernel: Windows
  Kernel version: 6.2
  Architecture: x86_64

Build information:
  Compiler: MSVC 19.26.28805.0
  Build host: WINBUILD2
  OpenSSL version: OpenSSL 1.1.1c  28 May 2019

Application information:

General paths:
  Config directory: C:\ProgramData\icinga2\etc\icinga2
  Data directory: C:\ProgramData\icinga2\var\lib\icinga2
  Log directory: C:\ProgramData\icinga2\var\log\icinga2
  Cache directory: C:\ProgramData\icinga2\var\cache\icinga2
  Spool directory: C:\ProgramData\icinga2\var\spool\icinga2
  Run directory: C:\ProgramData\icinga2\var\run\icinga2

Old paths (deprecated):
  Installation root: C:\Program Files\ICINGA2\
  Sysconf directory: C:\ProgramData\icinga2\etc
  Run directory (base): C:\ProgramData\icinga2\var\run
  Local state directory: C:\ProgramData\icinga2\var

Internal paths:
  Package data directory: C:\Program Files\ICINGA2\\share\icinga2
  State path: C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state
  Modified attributes path: C:\ProgramData\icinga2\var\lib\icinga2/modified-attributes.conf
  Objects path: C:\ProgramData\icinga2\var\cache\icinga2/icinga2.debug
  Vars path: C:\ProgramData\icinga2\var\cache\icinga2/icinga2.vars
  PID path: C:\ProgramData\icinga2\var\run\icinga2/icinga2.pid

Error: test exception

Stacktrace:
 0# 0x00007FF7CB225D42 in icinga2
 1# 0x00007FF7CB73FD9E in icinga2
 2# 0x00007FF85A131080 in VCRUNTIME140_1
 3# _NLG_Return2 in VCRUNTIME140_1
 4# RtlCaptureContext in ntdll
 5# 0x00007FF7CB1B53CF in icinga2
 6# terminate in ucrtbase
 7# 0x00007FF7CB70D816 in icinga2
 8# 0x00007FF7CB1E6114 in icinga2
 9# UnhandledExceptionFilter in KERNELBASE
10# memset in ntdll
11# _C_specific_handler in ntdll
12# _chkstk in ntdll
13# RtlWalkFrameChain in ntdll
14# RtlRaiseException in ntdll
15# RaiseException in KERNELBASE
16# CxxThrowException in VCRUNTIME140
17# 0x00007FF7CB428E98 in icinga2
18# 0x00007FF7CB414828 in icinga2
19# 0x00007FF7CB3F012F in icinga2
20# 0x00007FF7CB0F4027 in icinga2
21# 0x00007FF7CB0FC3A1 in icinga2
22# 0x00007FF7CB70D0DC in icinga2
23# BaseThreadInitThunk in KERNEL32
24# RtlUserThreadStart in ntdll

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***
Linux, Debug Build, C++ Exception
Caught unhandled exception.
Current time: 2020-10-21 16:41:48 +0200

  Application version: v2.12.0-87-g5806c8985

System information:
  Platform: Arch Linux
  Platform version: Unknown
  Kernel: Linux
  Kernel version: 5.9.1-arch1-1
  Architecture: x86_64

Build information:
  Compiler: GNU 10.2.0
  Build host: wh
  OpenSSL version: OpenSSL 1.1.1h  22 Sep 2020

Application information:

General paths:
  Config directory: /usr/local/etc/icinga2
  Data directory: /usr/local/var/lib/icinga2
  Log directory: /usr/local/var/log/icinga2
  Cache directory: /usr/local/var/cache/icinga2
  Spool directory: /usr/local/var/spool/icinga2
  Run directory: /usr/local/var/run/icinga2

Old paths (deprecated):
  Installation root: /usr/local
  Sysconf directory: /usr/local/etc
  Run directory (base): /usr/local/var/run
  Local state directory: /usr/local/var

Internal paths:
  Package data directory: /usr/local/share/icinga2
  State path: /usr/local/var/lib/icinga2/icinga2.state
  Modified attributes path: /usr/local/var/lib/icinga2/modified-attributes.conf
  Objects path: /usr/local/var/cache/icinga2/icinga2.debug
  Vars path: /usr/local/var/cache/icinga2/icinga2.vars
  PID path: /usr/local/var/run/icinga2/icinga2.pid

Error: test exception

Stacktrace:
 0# __cxa_throw in /usr/local/lib64/icinga2/sbin/icinga2
 1# throw_test_exception() in /usr/local/lib64/icinga2/sbin/icinga2
 2# 0x0000563B1AE7C2C5 in /usr/local/lib64/icinga2/sbin/icinga2
 3# 0x0000563B1AE7CDBB in /usr/local/lib64/icinga2/sbin/icinga2
 4# icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const in /usr/local/lib64/icinga2/sbin/icinga2
 5# 0x0000563B1A9D8BAA in /usr/local/lib64/icinga2/sbin/icinga2
 6# main in /usr/local/lib64/icinga2/sbin/icinga2
 7# __libc_start_main in /usr/lib/libc.so.6
 8# _start in /usr/local/lib64/icinga2/sbin/icinga2

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***
ptrace: Operation not permitted.
The program is not being run.
Linux, Debug Build, abort()
Caught SIGABRT.
Current time: 2020-10-21 16:41:54 +0200

  Application version: v2.12.0-87-g5806c8985

System information:
  Platform: Arch Linux
  Platform version: Unknown
  Kernel: Linux
  Kernel version: 5.9.1-arch1-1
  Architecture: x86_64

Build information:
  Compiler: GNU 10.2.0
  Build host: wh
  OpenSSL version: OpenSSL 1.1.1h  22 Sep 2020

Application information:

General paths:
  Config directory: /usr/local/etc/icinga2
  Data directory: /usr/local/var/lib/icinga2
  Log directory: /usr/local/var/log/icinga2
  Cache directory: /usr/local/var/cache/icinga2
  Spool directory: /usr/local/var/spool/icinga2
  Run directory: /usr/local/var/run/icinga2

Old paths (deprecated):
  Installation root: /usr/local
  Sysconf directory: /usr/local/etc
  Run directory (base): /usr/local/var/run
  Local state directory: /usr/local/var

Internal paths:
  Package data directory: /usr/local/share/icinga2
  State path: /usr/local/var/lib/icinga2/icinga2.state
  Modified attributes path: /usr/local/var/lib/icinga2/modified-attributes.conf
  Objects path: /usr/local/var/cache/icinga2/icinga2.debug
  Vars path: /usr/local/var/cache/icinga2/icinga2.vars
  PID path: /usr/local/var/run/icinga2/icinga2.pid

Stacktrace:
 0# icinga::Application::SigAbrtHandler(int) in /usr/local/lib64/icinga2/sbin/icinga2
 1# 0x00007F21355970F0 in /usr/lib/libpthread.so.0
 2# gsignal in /usr/lib/libc.so.6
 3# abort in /usr/lib/libc.so.6
 4# throw_test_exception() in /usr/local/lib64/icinga2/sbin/icinga2
 5# 0x00005562895D02C5 in /usr/local/lib64/icinga2/sbin/icinga2
 6# 0x00005562895D0DBB in /usr/local/lib64/icinga2/sbin/icinga2
 7# icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const in /usr/local/lib64/icinga2/sbin/icinga2
 8# 0x000055628912CBAA in /usr/local/lib64/icinga2/sbin/icinga2
 9# main in /usr/local/lib64/icinga2/sbin/icinga2
10# __libc_start_main in /usr/lib/libc.so.6
11# _start in /usr/local/lib64/icinga2/sbin/icinga2

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

ptrace: Operation not permitted.
The program is not being run.
Linux, Snapshot Build, C++ Exception
Caught unhandled exception.
Current time: 2020-10-21 15:17:22 +0000

  Application version: v2.12.0-87-g5806c8985

System information:
  Platform: Debian GNU/Linux
  Platform version: 10 (buster)
  Kernel: Linux
  Kernel version: 5.9.1-arch1-1
  Architecture: x86_64

Build information:
  Compiler: GNU 8.3.0
  Build host: runner-hh8q3bz2-project-298-concurrent-0
  OpenSSL version: OpenSSL 1.1.1d  10 Sep 2019

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

Error: test exception

Stacktrace:
 0# __cxa_throw in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 1# 0x000055952B335A8D in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 2# 0x000055952B6741CA in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 3# icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 4# 0x000055952B876CD6 in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 5# main in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 6# __libc_start_main in /lib/x86_64-linux-gnu/libc.so.6
 7# _start in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***
Failed to launch GDB: No such file or directory
Linux, Snapshot Build, abort()
Caught SIGABRT.
Current time: 2020-10-21 15:17:29 +0000

  Application version: v2.12.0-87-g5806c8985

System information:
  Platform: Debian GNU/Linux
  Platform version: 10 (buster)
  Kernel: Linux
  Kernel version: 5.9.1-arch1-1
  Architecture: x86_64

Build information:
  Compiler: GNU 8.3.0
  Build host: runner-hh8q3bz2-project-298-concurrent-0
  OpenSSL version: OpenSSL 1.1.1d  10 Sep 2019

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

Stacktrace:
 0# icinga::Application::SigAbrtHandler(int) in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 1# 0x00007F8D14D40730 in /lib/x86_64-linux-gnu/libpthread.so.0
 2# gsignal in /lib/x86_64-linux-gnu/libc.so.6
 3# abort in /lib/x86_64-linux-gnu/libc.so.6
 4# 0x00005588DD801A92 in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 5# 0x00005588DDB401CA in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 6# icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 7# 0x00005588DDD42CD6 in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 8# main in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 9# __libc_start_main in /lib/x86_64-linux-gnu/libc.so.6
10# _start in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

Failed to launch GDB: No such file or directory

TODO

@icinga-probot icinga-probot bot added this to the 2.13.0 milestone Oct 14, 2020
@julianbrost julianbrost force-pushed the feature/improve-crashlog branch 5 times, most recently from 64ecccc to 460bd8f Compare October 21, 2020 11:08
julianbrost added a commit that referenced this pull request Oct 21, 2020
PR #8390 changed the default version of Visual Studio to 2019 because
PR #8373 introduces behavior that is incompatible with MSVC versions
shipped with older versions of Visual Studio.
julianbrost added a commit that referenced this pull request Oct 21, 2020
PR #8390 changed the default version of Visual Studio to 2019 because
PR #8373 introduces behavior that is incompatible with MSVC versions
shipped with older versions of Visual Studio.
@julianbrost julianbrost force-pushed the feature/improve-crashlog branch from 460bd8f to 3440f97 Compare October 21, 2020 14:03
@julianbrost
Copy link
Contributor Author

Requesting some preliminary reviews before also testing this on more platforms and older systems.

Copy link
Member

@Al2Klimov Al2Klimov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel free to squash the fixes into the respective comments.

@julianbrost julianbrost force-pushed the feature/improve-crashlog branch 4 times, most recently from bbf2853 to 8e3fbb3 Compare October 22, 2020 08:42
@julianbrost julianbrost force-pushed the feature/improve-crashlog branch from 8e3fbb3 to b213d85 Compare October 22, 2020 09:52
@julianbrost
Copy link
Contributor Author

julianbrost commented Oct 23, 2020

Testing on FreeBSD showed that the symbol name resolution boost::stacktrace returns broken values there.

Default format (operator<<(std::ostream, boost::stacktrace::stacktrace)):

 0# __cxa_throw in /usr/local/lib/icinga2/sbin/icinga2
 1# void in /usr/local/lib/icinga2/sbin/icinga2
 2# __cxa_rethrow in /lib/libcxxrt.so.1
 3# short in /usr/local/lib/icinga2/sbin/icinga2
 4# short in /usr/local/lib/icinga2/sbin/icinga2
 5# void in /usr/local/lib/icinga2/sbin/icinga2
 6# void in /usr/local/lib/icinga2/sbin/icinga2
 7# void in /usr/local/lib/icinga2/sbin/icinga2

Custom output iterating over the frames in the stack using frame.name() and boost::stacktrace::detail::location_from_symbol() (which is not a documented interface unfortunately):

  ( 0) 0x5666b0 __cxa_throw in /usr/local/lib/icinga2/sbin/icinga2
  ( 1) 0x531aed vtable for boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<std::out_of_range> > in /usr/local/lib/icinga2/sbin/icinga2
  ( 2) 0x801412ad9 __cxa_rethrow in /lib/libcxxrt.so.1
  ( 3) 0x89d8b6 std::__1::__assoc_sub_state::~__assoc_sub_state(void) in /usr/local/lib/icinga2/sbin/icinga2
  ( 4) 0x89aa01 std::__1::__assoc_sub_state::~__assoc_sub_state(void) in /usr/local/lib/icinga2/sbin/icinga2
  ( 5) 0x51914f vtable for boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<std::out_of_range> > in /usr/local/lib/icinga2/sbin/icinga2
  ( 6) 0x51539a vtable for boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<std::out_of_range> > in /usr/local/lib/icinga2/sbin/icinga2
  ( 7) 0x51510f vtable for boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<std::out_of_range> > in /usr/local/lib/icinga2/sbin/icinga2

Resolving the symbols from the Boost stacktrace using backtrace_symbols():

  ( 0) 0x5666b0 <__cxa_throw+0x120> at /usr/local/lib/icinga2/sbin/icinga2
  ( 1) 0x531aed <_ZN6icinga11Application16ExceptionHandlerEv+0x36d> at /usr/local/lib/icinga2/sbin/icinga2
  ( 2) 0x801412ad9 <__cxa_rethrow+0x159> at /lib/libcxxrt.so.1
  ( 3) 0x89d8b6 <_ZL15StartUnixWorkerRKNSt3__16vectorINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEENS4_IS6_EEEEbRKN6icinga6StringE+0x1326> at /usr/local/lib/icinga2/sbin/icinga2
  ( 4) 0x89aa01 <_ZNK6icinga13DaemonCommand3RunERKN5boost15program_options13variables_mapERKNSt3__16vectorINS6_12basic_stringIcNS6_11char_traitsIcEENS6_9allocatorIcEEEENSB_ISD_EEEE+0xd11> at /usr/local/lib/icinga2/sbin/icinga2
  ( 5) 0x51914f <_ZL4Mainv+0x3d3f> at /usr/local/lib/icinga2/sbin/icinga2
  ( 6) 0x51539a <main+0xba> at /usr/local/lib/icinga2/sbin/icinga2
  ( 7) 0x51510f <_start+0x10f> at /usr/local/lib/icinga2/sbin/icinga2

I see the following possible options:

  • Fall back to formatting the stacktrace saved with boost::stacktrace using backtrace_symbols() from execinfo.h on some platforms.
  • Use a custom stacktrace formatter that always prints the address and say this is good enough.

@julianbrost
Copy link
Contributor Author

I'll try to do the following: Add a new define ICINGA_STACKTRACE_USE_BACKTRACE_SYMBOLS. If it is set, use the Boost stacktrace with backtrace_symbols(), otherwise use the regular operator<< (this avoids using any Boost internals). Set this using CMake on FreeBSD (and possible other platforms, should there be more that show this strange behavior).

@Al2Klimov
Copy link
Member

Remember to inform the packagers.

@julianbrost
Copy link
Contributor Author

Remember to inform the packagers.

About what in particular? backtrace_symbols() was already in use before, so that should not make a difference. Or just a general hint that this might need tweaking on some platforms?

@Al2Klimov
Copy link
Member

About -DICINGA_STACKTRACE_USE_BACKTRACE_SYMBOLS.

@julianbrost
Copy link
Contributor Author

I would just set this in our CMakeLists.txt, so no need to update any packaging.

@julianbrost julianbrost force-pushed the feature/improve-crashlog branch 3 times, most recently from 0c6a5ff to 81bdef0 Compare October 27, 2020 08:10
…n and document behavior

The logic for selecting the traces to print stays the same, but there
are fewer nested ifs now. This changes the format of the returned string
a bit by adding a heading for both traces.
Maybe this will save the next person who has to look at this code some
time. Please don't blame me for the implementation, I'm just trying to
reconstruct what it does.
Older versions of MSVC fail to rethrow an unhandled C++ exception (using
`throw;`) in the termination handler (`std::set_terminate`), however
Icinga relies on this behavior in its crash handler
(`Application::ExceptionHandler`).
This makes the format more similar to what the uncaught C++ and SEH
exception handlers write. Previously there was no indication in the
crash log that a SIGABRT happened.
Unfortunately, the symbol resolution of boost::stacktrace is broken on
FreeBSD, therefore fall back to using backtrace_symbols() to print the
stack trace saved by Boost.

Additionally, -D_GNU_SOURCE is required on FreeBSD for the
_Unwind_Backtrace function used by boost::stacktrace.
So far, the check that actually sets HAVE_LIBEXECINFO was executed after
it was already used to add dependencies.
Needed by `boost::stacktrace` for `_Unwind_Backtrace()`.
@julianbrost julianbrost force-pushed the feature/improve-crashlog branch from 6cd09bb to 00caf3d Compare March 1, 2021 13:11
@julianbrost julianbrost added area/log Logging related consider backporting Should be considered for inclusion in a bugfix release enhancement New feature or request labels Mar 1, 2021
- Explicitly disable optimizations for MSVC
- Make stack_test_func_a bigger
@julianbrost julianbrost force-pushed the feature/improve-crashlog branch from 00caf3d to c5626cb Compare March 1, 2021 15:08
Copy link
Contributor

@N-o-X N-o-X left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Great work! 👍

@julianbrost
Copy link
Contributor Author

@Al2Klimov I just noticed that I pushed to this PR since your last review. Do you want to take another look at this before merging this?

@Al2Klimov Al2Klimov self-assigned this Jun 28, 2021
@Al2Klimov Al2Klimov self-requested a review June 29, 2021 11:37
@Al2Klimov Al2Klimov removed their assignment Jun 29, 2021
Copy link
Member

@Al2Klimov Al2Klimov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test subject

W/o vs. w/ this PR

Test base

Test results

Linux

DEBUG_ABORT

Before
root@debian-gnu-linux-10-vm:~# DEBUG_ABORT=1 icinga2 daemon -x warning
Caught SIGABRT.
Current time: 2021-06-29 12:39:57 +0200

[2021-06-29 12:39:57 +0200] critical/Application: Icinga 2 has terminated unexpectedly. Additional information can be found in '/var/log/icinga2/crash/report.1624963197.837719'
root@debian-gnu-linux-10-vm:~# cat /var/log/icinga2/crash/report.1624963197.837719
  Application version: v2.12.0-511-gb36cf3e08

System information:
  Platform: Debian GNU/Linux
  Platform version: 10 (buster)
  Kernel: Linux
  Kernel version: 4.19.0-10-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 8.3.0
  Build host: 0d188d083bb1
  OpenSSL version: OpenSSL 1.1.1d  10 Sep 2019

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
Stacktrace:

	(0) libc.so.6: gsignal (+0x10b) [0x7f5e99cf27bb]
	(1) libc.so.6: abort (+0x121) [0x7f5e99cdd535]
	(2) icinga2: <unknown function> (+0x2c3ad8) [0x5639288e8ad8]
	(3) icinga2: <unknown function> (+0x614dd0) [0x563928c39dd0]
	(4) icinga2: icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const (+0x9ab) [0x563928c3aa3b]
	(5) icinga2: <unknown function> (+0x823316) [0x563928e48316]
	(6) icinga2: main (+0xcf) [0x56392896218f]
	(7) libc.so.6: __libc_start_main (+0xeb) [0x7f5e99cdf09b]
	(8) icinga2: _start (+0x2a) [0x56392896223a]

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

Failed to launch GDB: No such file or directory
root@debian-gnu-linux-10-vm:~#
After
root@debian-gnu-linux-10-vm:~# DEBUG_ABORT=1 icinga2 daemon -x warning
Caught SIGABRT.
Current time: 2021-06-29 12:42:32 +0200

[2021-06-29 12:42:32 +0200] critical/Application: Icinga 2 has terminated unexpectedly. Additional information can be found in '/var/log/icinga2/crash/report.1624963352.489917'
root@debian-gnu-linux-10-vm:~# cat /var/log/icinga2/crash/report.1624963352.489917
Caught SIGABRT.
Current time: 2021-06-29 12:42:32 +0200

  Application version: v2.12.0-530-g74218807b

System information:
  Platform: Debian GNU/Linux
  Platform version: 10 (buster)
  Kernel: Linux
  Kernel version: 4.19.0-10-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 8.3.0
  Build host: 4dd22d1c669b
  OpenSSL version: OpenSSL 1.1.1d  10 Sep 2019

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

Stacktrace:
 0# icinga::Application::SigAbrtHandler(int) in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 1# 0x00007FC44C034730 in /lib/x86_64-linux-gnu/libpthread.so.0
 2# gsignal in /lib/x86_64-linux-gnu/libc.so.6
 3# abort in /lib/x86_64-linux-gnu/libc.so.6
 4# 0x0000556795553B42 in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 5# 0x00005567958A5380 in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 6# icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 7# 0x0000556795AB41F6 in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 8# main in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 9# __libc_start_main in /lib/x86_64-linux-gnu/libc.so.6
10# _start in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

Failed to launch GDB: No such file or directory
root@debian-gnu-linux-10-vm:~#

DEBUG_THROW_CXX

Before
root@debian-gnu-linux-10-vm:~# DEBUG_THROW_CXX=1 icinga2 daemon -x warning
[2021-06-29 12:41:10 +0200] critical/cli: Exception in main process: Error: test exception


	(0) icinga2: <unknown function> (+0x614dd0) [0x5582f9472dd0]
	(1) icinga2: icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const (+0x9ab) [0x5582f9473a3b]
	(2) icinga2: <unknown function> (+0x823316) [0x5582f9681316]
	(3) icinga2: main (+0xcf) [0x5582f919b18f]
	(4) libc.so.6: __libc_start_main (+0xeb) [0x7f06bfb1b09b]
	(5) icinga2: _start (+0x2a) [0x5582f919b23a]
root@debian-gnu-linux-10-vm:~#
After
root@debian-gnu-linux-10-vm:~# DEBUG_THROW_CXX=1 icinga2 daemon -x warning
[2021-06-29 12:43:23 +0200] critical/cli: Exception in main process: Error: test exception

Stacktrace:
 0# __cxa_throw in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 1# 0x0000556843B78B3D in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 2# 0x0000556843ECA380 in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 3# icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 4# 0x00005568440D91F6 in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 5# main in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 6# __libc_start_main in /lib/x86_64-linux-gnu/libc.so.6
 7# _start in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
root@debian-gnu-linux-10-vm:~#

Windows

DEBUG_ABORT

Before

Bildschirmfoto 2021-06-29 um 14 16 47

Only "Retry" gives a result:

[2021-06-29 12:16:50 +0000] critical/Application: Icinga 2 has terminated unexpectedly. Additional information can be found in 'C:\ProgramData\icinga2\var\log\icinga2/crash/report.1624969010.804000'
PS C:\Users\Administrator>
  Application version: v2.12.0-684-gfd0c9c39b

System information:
  Platform: Windows
  Platform version: 8
  Kernel: Windows
  Kernel version: 6.2
  Architecture: x86_64

Build information:
  Compiler: MSVC 19.29.30038.1
  Build host: DESKTOP-V1GF2GS
  OpenSSL version: OpenSSL 1.1.1h  22 Sep 2020

Application information:

General paths:
  Config directory: C:\ProgramData\icinga2\etc\icinga2
  Data directory: C:\ProgramData\icinga2\var\lib\icinga2
  Log directory: C:\ProgramData\icinga2\var\log\icinga2
  Cache directory: C:\ProgramData\icinga2\var\cache\icinga2
  Spool directory: C:\ProgramData\icinga2\var\spool\icinga2
  Run directory: C:\ProgramData\icinga2\var\run\icinga2

Old paths (deprecated):
  Installation root: C:\Program Files\ICINGA2\
  Sysconf directory: C:\ProgramData\icinga2\etc
  Run directory (base): C:\ProgramData\icinga2\var\run
  Local state directory: C:\ProgramData\icinga2\var

Internal paths:
  Package data directory: C:\Program Files\ICINGA2\\share\icinga2
  State path: C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state
  Modified attributes path: C:\ProgramData\icinga2\var\lib\icinga2/modified-attributes.conf
  Objects path: C:\ProgramData\icinga2\var\cache\icinga2/icinga2.debug
  Vars path: C:\ProgramData\icinga2\var\cache\icinga2/icinga2.vars
  PID path: C:\ProgramData\icinga2\var\run\icinga2/icinga2.pid
Caught unhandled SEH exception.
Current time: 2021-06-29 12:16:50 +0000

Stacktrace:

	(0): throw_test_exception+182
	(1): RunWorker+978
	(2): icinga::DaemonCommand::Run+2191
	(3): Main+13665
	(4): main+338
	(5): invoke_main+57
	(6): __scrt_common_main_seh+302
	(7): __scrt_common_main+14
	(8): mainCRTStartup+14
	(9): BaseThreadInitThunk+20
	(10): RtlUserThreadStart+33
***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***
After
Caught unhandled SEH exception.
Current time: 2021-06-29 12:34:33 +0000

  Application version: v2.12.0-704-ge6def2acf

System information:
  Platform: Windows
  Platform version: 8
  Kernel: Windows
  Kernel version: 6.2
  Architecture: x86_64

Build information:
  Compiler: MSVC 19.29.30038.1
  Build host: DESKTOP-V1GF2GS
  OpenSSL version: OpenSSL 1.1.1h  22 Sep 2020

Application information:

General paths:
  Config directory: C:\ProgramData\icinga2\etc\icinga2
  Data directory: C:\ProgramData\icinga2\var\lib\icinga2
  Log directory: C:\ProgramData\icinga2\var\log\icinga2
  Cache directory: C:\ProgramData\icinga2\var\cache\icinga2
  Spool directory: C:\ProgramData\icinga2\var\spool\icinga2
  Run directory: C:\ProgramData\icinga2\var\run\icinga2

Old paths (deprecated):
  Installation root: C:\Program Files\ICINGA2\
  Sysconf directory: C:\ProgramData\icinga2\etc
  Run directory (base): C:\ProgramData\icinga2\var\run
  Local state directory: C:\ProgramData\icinga2\var

Internal paths:
  Package data directory: C:\Program Files\ICINGA2\\share\icinga2
  State path: C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state
  Modified attributes path: C:\ProgramData\icinga2\var\lib\icinga2/modified-attributes.conf
  Objects path: C:\ProgramData\icinga2\var\cache\icinga2/icinga2.debug
  Vars path: C:\ProgramData\icinga2\var\cache\icinga2/icinga2.vars
  PID path: C:\ProgramData\icinga2\var\run\icinga2/icinga2.pid

SEH exception:
  Code: 0x80000003
  Address: 00007FFECE6C7475
  Flags: 0

Stacktrace:
 0# boost::stacktrace::basic_stacktrace<std::allocator<boost::stacktrace::frame> >::init at C:\local\boost_1_71_0-Win64\boost\stacktrace\stacktrace.hpp:75
 1# boost::stacktrace::basic_stacktrace<std::allocator<boost::stacktrace::frame> >::basic_stacktrace<std::allocator<boost::stacktrace::frame> > at C:\local\boost_1_71_0-Win64\boost\stacktrace\stacktrace.hpp:127
 2# icinga::Application::SEHUnhandledExceptionFilter at C:\Users\Administrator\icinga2\lib\base\application.cpp:958
 3# UnhandledExceptionFilter in KERNELBASE
 4# memset in ntdll
 5# _C_specific_handler in ntdll
 6# _chkstk in ntdll
 7# RtlWalkFrameChain in ntdll
 8# KiUserExceptionDispatcher in ntdll
 9# _threadid in ucrtbased
10# _threadid in ucrtbased
11# abort in ucrtbased
12# throw_test_exception at C:\Users\Administrator\icinga2\lib\cli\daemoncommand.cpp:225
13# RunWorker at C:\Users\Administrator\icinga2\lib\cli\daemoncommand.cpp:324
14# icinga::DaemonCommand::Run at C:\Users\Administrator\icinga2\lib\cli\daemoncommand.cpp:736
15# Main at C:\Users\Administrator\icinga2\icinga-app\icinga.cpp:660
16# main at C:\Users\Administrator\icinga2\icinga-app\icinga.cpp:946
17# invoke_main at d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:79
18# __scrt_common_main_seh at d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
19# __scrt_common_main at d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:331
20# mainCRTStartup at d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_main.cpp:17
21# BaseThreadInitThunk in KERNEL32
22# RtlUserThreadStart in ntdll

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

DEBUG_THROW_CXX

Before

Bildschirmfoto 2021-06-29 um 14 22 21

"Retry" just returns the shell.

After

All quiet on the western front.

DEBUG_THROW_SEH

Before
PS C:\Users\Administrator> $Env:DEBUG_THROW_SEH = 1
PS C:\Users\Administrator> .\icinga2\build\Bin\Release\Debug\icinga2.exe daemon
[2021-06-29 12:25:38 +0000] information/cli: Icinga application loader (version: v2.12.0-684-gfd0c9c39b; debug)
[2021-06-29 12:25:38 +0000] information/cli: Loading configuration file(s).
[2021-06-29 12:25:38 +0000] information/ConfigItem: Committing config item(s).
[2021-06-29 12:25:41 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.
[2021-06-29 12:25:41 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2021-06-29 12:25:41 +0000] information/ConfigItem: Instantiated 1 User.
[2021-06-29 12:25:41 +0000] information/ConfigItem: Instantiated 1 UserGroup.
[2021-06-29 12:25:41 +0000] information/ConfigItem: Instantiated 2 NotificationCommands.
[2021-06-29 12:25:41 +0000] information/ConfigItem: Instantiated 3 ServiceGroups.
[2021-06-29 12:25:41 +0000] information/ConfigItem: Instantiated 10 Notifications.
[2021-06-29 12:25:41 +0000] information/ConfigItem: Instantiated 9 Services.
[2021-06-29 12:25:41 +0000] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2021-06-29 12:25:41 +0000] information/ConfigItem: Instantiated 3 Zones.
[2021-06-29 12:25:41 +0000] information/ConfigItem: Instantiated 3 TimePeriods.
[2021-06-29 12:25:41 +0000] information/ConfigItem: Instantiated 1 Endpoint.
[2021-06-29 12:25:41 +0000] information/ConfigItem: Instantiated 1 WindowsEventLogLogger.
[2021-06-29 12:25:41 +0000] information/ConfigItem: Instantiated 244 CheckCommands.
[2021-06-29 12:25:41 +0000] information/ConfigItem: Instantiated 2 HostGroups.
[2021-06-29 12:25:41 +0000] information/ConfigItem: Instantiated 1 Host.
[2021-06-29 12:25:41 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2021-06-29 12:25:41 +0000] information/ScriptGlobal: Dumping variables to file 'C:\ProgramData\icinga2\var\cache\icinga2/icinga2.vars'
[2021-06-29 12:25:41 +0000] information/ConfigItem: Triggering Start signal for config items
[2021-06-29 12:25:41 +0000] information/NotificationComponent: 'notification' started.
[2021-06-29 12:25:41 +0000] information/CheckerComponent: 'checker' started.
[2021-06-29 12:25:41 +0000] information/ConfigItem: Activated all objects.
[2021-06-29 12:25:41 +0000] critical/Application: Icinga 2 has terminated unexpectedly. Additional information can be found in 'C:\ProgramData\icinga2\var\log\icinga2/crash/report.1624969541.631000'
PS C:\Users\Administrator>
  Application version: v2.12.0-684-gfd0c9c39b

System information:
  Platform: Windows
  Platform version: 8
  Kernel: Windows
  Kernel version: 6.2
  Architecture: x86_64

Build information:
  Compiler: MSVC 19.29.30038.1
  Build host: DESKTOP-V1GF2GS
  OpenSSL version: OpenSSL 1.1.1h  22 Sep 2020

Application information:

General paths:
  Config directory: C:\ProgramData\icinga2\etc\icinga2
  Data directory: C:\ProgramData\icinga2\var\lib\icinga2
  Log directory: C:\ProgramData\icinga2\var\log\icinga2
  Cache directory: C:\ProgramData\icinga2\var\cache\icinga2
  Spool directory: C:\ProgramData\icinga2\var\spool\icinga2
  Run directory: C:\ProgramData\icinga2\var\run\icinga2

Old paths (deprecated):
  Installation root: C:\Program Files\ICINGA2\
  Sysconf directory: C:\ProgramData\icinga2\etc
  Run directory (base): C:\ProgramData\icinga2\var\run
  Local state directory: C:\ProgramData\icinga2\var

Internal paths:
  Package data directory: C:\Program Files\ICINGA2\\share\icinga2
  State path: C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state
  Modified attributes path: C:\ProgramData\icinga2\var\lib\icinga2/modified-attributes.conf
  Objects path: C:\ProgramData\icinga2\var\cache\icinga2/icinga2.debug
  Vars path: C:\ProgramData\icinga2\var\cache\icinga2/icinga2.vars
  PID path: C:\ProgramData\icinga2\var\run\icinga2/icinga2.pid
Caught unhandled SEH exception.
Current time: 2021-06-29 12:25:41 +0000

Stacktrace:

	(0): icinga::DaemonCommand::Run+2191
	(1): Main+13665
	(2): main+338
	(3): invoke_main+57
	(4): __scrt_common_main_seh+302
	(5): __scrt_common_main+14
	(6): mainCRTStartup+14
	(7): BaseThreadInitThunk+20
	(8): RtlUserThreadStart+33
***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***
After
Caught unhandled SEH exception.
Current time: 2021-06-29 12:37:57 +0000

  Application version: v2.12.0-704-ge6def2acf

System information:
  Platform: Windows
  Platform version: 8
  Kernel: Windows
  Kernel version: 6.2
  Architecture: x86_64

Build information:
  Compiler: MSVC 19.29.30038.1
  Build host: DESKTOP-V1GF2GS
  OpenSSL version: OpenSSL 1.1.1h  22 Sep 2020

Application information:

General paths:
  Config directory: C:\ProgramData\icinga2\etc\icinga2
  Data directory: C:\ProgramData\icinga2\var\lib\icinga2
  Log directory: C:\ProgramData\icinga2\var\log\icinga2
  Cache directory: C:\ProgramData\icinga2\var\cache\icinga2
  Spool directory: C:\ProgramData\icinga2\var\spool\icinga2
  Run directory: C:\ProgramData\icinga2\var\run\icinga2

Old paths (deprecated):
  Installation root: C:\Program Files\ICINGA2\
  Sysconf directory: C:\ProgramData\icinga2\etc
  Run directory (base): C:\ProgramData\icinga2\var\run
  Local state directory: C:\ProgramData\icinga2\var

Internal paths:
  Package data directory: C:\Program Files\ICINGA2\\share\icinga2
  State path: C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state
  Modified attributes path: C:\ProgramData\icinga2\var\lib\icinga2/modified-attributes.conf
  Objects path: C:\ProgramData\icinga2\var\cache\icinga2/icinga2.debug
  Vars path: C:\ProgramData\icinga2\var\cache\icinga2/icinga2.vars
  PID path: C:\ProgramData\icinga2\var\run\icinga2/icinga2.pid

SEH exception:
  Code: 0x2a
  Address: 00007FFEFC885299
  Flags: 0

Stacktrace:
 0# boost::stacktrace::basic_stacktrace<std::allocator<boost::stacktrace::frame> >::init at C:\local\boost_1_71_0-Win64\boost\stacktrace\stacktrace.hpp:75
 1# boost::stacktrace::basic_stacktrace<std::allocator<boost::stacktrace::frame> >::basic_stacktrace<std::allocator<boost::stacktrace::frame> > at C:\local\boost_1_71_0-Win64\boost\stacktrace\stacktrace.hpp:127
 2# icinga::Application::SEHUnhandledExceptionFilter at C:\Users\Administrator\icinga2\lib\base\application.cpp:958
 3# UnhandledExceptionFilter in KERNELBASE
 4# memset in ntdll
 5# _C_specific_handler in ntdll
 6# _chkstk in ntdll
 7# RtlWalkFrameChain in ntdll
 8# RtlRaiseException in ntdll
 9# RaiseException in KERNELBASE
10# throw_test_exception at C:\Users\Administrator\icinga2\lib\cli\daemoncommand.cpp:233
11# RunWorker at C:\Users\Administrator\icinga2\lib\cli\daemoncommand.cpp:324
12# icinga::DaemonCommand::Run at C:\Users\Administrator\icinga2\lib\cli\daemoncommand.cpp:736
13# Main at C:\Users\Administrator\icinga2\icinga-app\icinga.cpp:660
14# main at C:\Users\Administrator\icinga2\icinga-app\icinga.cpp:946
15# invoke_main at d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:79
16# __scrt_common_main_seh at d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
17# __scrt_common_main at d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:331
18# mainCRTStartup at d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_main.cpp:17
19# BaseThreadInitThunk in KERNEL32
20# RtlUserThreadStart in ntdll

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

Conclusion

  • Linux: information which IMAO doesn’t matter gets lost
  • Windows: information which matters gets added

@Al2Klimov Al2Klimov merged commit 6048d0e into master Jun 29, 2021
@icinga-probot icinga-probot bot deleted the feature/improve-crashlog branch June 29, 2021 15:52
This was referenced Jul 2, 2021
@N-o-X N-o-X added backported Fix was included in a bugfix release and removed consider backporting Should be considered for inclusion in a bugfix release labels Nov 10, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/log Logging related backported Fix was included in a bugfix release enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants