Improve crashlog #8865

Al2Klimov · 2021-07-02T17:16:40Z

Backport of

Improve crashlog #8373

Al2Klimov · 2021-07-02T17:16:53Z

Blocked by

GitHub actions: build Windows #8862

julianbrost

Please rebase to remove commits not part of this PR so that GitHub doesn't show them as part of this PR.

On Windows, the termination handler is executed for uncaught C++ exceptions unless a SEH unhandled exception filter is also set. In this case, this filter has to explicitly chain the default filter to keep this behavior.

… filter

This is more similar to the normal exception crashlog which also states the problem and time at the beginning of the file.

Provides roughly the same functionality but works better on certain platforms (especially Windows) and is less code to maintain.

By default, DiagnosticInformation uses the stack trace saved when the exception was thrown, but this mechanism is not in use on Windows. Gathering a stacktrace in the terminate handler serves as a fallback.

…n and document behavior The logic for selecting the traces to print stays the same, but there are fewer nested ifs now. This changes the format of the returned string a bit by adding a heading for both traces.

Maybe this will save the next person who has to look at this code some time. Please don't blame me for the implementation, I'm just trying to reconstruct what it does.

Older versions of MSVC fail to rethrow an unhandled C++ exception (using `throw;`) in the termination handler (`std::set_terminate`), however Icinga relies on this behavior in its crash handler (`Application::ExceptionHandler`).

This makes the format more similar to what the uncaught C++ and SEH exception handlers write. Previously there was no indication in the crash log that a SIGABRT happened.

Unfortunately, the symbol resolution of boost::stacktrace is broken on FreeBSD, therefore fall back to using backtrace_symbols() to print the stack trace saved by Boost. Additionally, -D_GNU_SOURCE is required on FreeBSD for the _Unwind_Backtrace function used by boost::stacktrace.

So far, the check that actually sets HAVE_LIBEXECINFO was executed after it was already used to add dependencies.

Needed by `boost::stacktrace` for `_Unwind_Backtrace()`.

- Explicitly disable optimizations for MSVC - Make stack_test_func_a bigger

julianbrost · 2021-07-08T12:53:53Z

git rebase -i origin/support/2.12 to remove commits that are not actually part of this PR.

julianbrost · 2021-07-08T13:21:15Z

Commit with additional test code: dd7eabe

Linux C++ Exception

Caught unhandled exception.
Current time: 2021-07-08 13:11:40 +0000

  Application version: v2.12.4-50-gdd7eabeb0

System information:
  Platform: Debian GNU/Linux
  Platform version: 10 (buster)
  Kernel: Linux
  Kernel version: 5.12.14-arch1-1
  Architecture: x86_64

Build information:
  Compiler: GNU 8.3.0
  Build host: 047695450aea
  OpenSSL version: OpenSSL 1.1.1d  10 Sep 2019

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

Error: test exception

Stacktrace:
 0# __cxa_throw in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 1# 0x000055AF75C640DC in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 2# 0x000055AF75F218CE in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 3# icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 4# 0x000055AF75CEDC5E in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 5# main in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 6# __libc_start_main in /lib/x86_64-linux-gnu/libc.so.6
 7# _start in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***
Failed to launch GDB: No such file or directory

Linux SIGABRT

Caught SIGABRT.
Current time: 2021-07-08 13:11:49 +0000

  Application version: v2.12.4-50-gdd7eabeb0

System information:
  Platform: Debian GNU/Linux
  Platform version: 10 (buster)
  Kernel: Linux
  Kernel version: 5.12.14-arch1-1
  Architecture: x86_64

Build information:
  Compiler: GNU 8.3.0
  Build host: 047695450aea
  OpenSSL version: OpenSSL 1.1.1d  10 Sep 2019

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

Stacktrace:
 0# icinga::Application::SigAbrtHandler(int) in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 1# 0x00007FC436F54730 in /lib/x86_64-linux-gnu/libpthread.so.0
 2# gsignal in /lib/x86_64-linux-gnu/libc.so.6
 3# abort in /lib/x86_64-linux-gnu/libc.so.6
 4# 0x000056207736711E in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 5# 0x00005620776248CE in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 6# icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 7# 0x00005620773F0C5E in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 8# main in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 9# __libc_start_main in /lib/x86_64-linux-gnu/libc.so.6
10# _start in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

Failed to launch GDB: No such file or directory

Windows x86-64 SEH

Caught unhandled SEH exception.
Current time: 2021-07-08 13:17:03 -0000

  Application version: v2.12.4-50-gdd7eabeb0

System information:
  Platform: Windows
  Platform version: 8
  Kernel: Windows
  Kernel version: 6.2
  Architecture: x86_64

Build information:
  Compiler: MSVC 19.29.30038.1
  Build host: WINBUILD2
  OpenSSL version: OpenSSL 1.1.1k  25 Mar 2021

Application information:

General paths:
  Config directory: C:\ProgramData\icinga2\etc\icinga2
  Data directory: C:\ProgramData\icinga2\var\lib\icinga2
  Log directory: C:\ProgramData\icinga2\var\log\icinga2
  Cache directory: C:\ProgramData\icinga2\var\cache\icinga2
  Spool directory: C:\ProgramData\icinga2\var\spool\icinga2
  Run directory: C:\ProgramData\icinga2\var\run\icinga2

Old paths (deprecated):
  Installation root: C:\Program Files\ICINGA2\
  Sysconf directory: C:\ProgramData\icinga2\etc
  Run directory (base): C:\ProgramData\icinga2\var\run
  Local state directory: C:\ProgramData\icinga2\var

Internal paths:
  Package data directory: C:\Program Files\ICINGA2\\share\icinga2
  State path: C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state
  Modified attributes path: C:\ProgramData\icinga2\var\lib\icinga2/modified-attributes.conf
  Objects path: C:\ProgramData\icinga2\var\cache\icinga2/icinga2.debug
  Vars path: C:\ProgramData\icinga2\var\cache\icinga2/icinga2.vars
  PID path: C:\ProgramData\icinga2\var\run\icinga2/icinga2.pid

SEH exception:
  Code: 0x2a
  Address: 00007FFE1E015299
  Flags: 0

Stacktrace:
 0# 0x00007FF798017B55 in icinga2
 1# 0x00007FF797FD3E4F in icinga2
 2# UnhandledExceptionFilter in KERNELBASE
 3# memset in ntdll
 4# _C_specific_handler in ntdll
 5# _chkstk in ntdll
 6# RtlWalkFrameChain in ntdll
 7# RtlRaiseException in ntdll
 8# RaiseException in KERNELBASE
 9# 0x00007FF79822378F in icinga2
10# 0x00007FF79820DBCE in icinga2
11# 0x00007FF7981EDFBF in icinga2
12# 0x00007FF797EE233E in icinga2
13# 0x00007FF797EEBD81 in icinga2
14# 0x00007FF798504C38 in icinga2
15# BaseThreadInitThunk in KERNEL32
16# RtlUserThreadStart in ntdll

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

Windows x86-64 C++ Exception

Caught unhandled exception.
Current time: 2021-07-08 13:17:14 -0000

  Application version: v2.12.4-50-gdd7eabeb0

System information:
  Platform: Windows
  Platform version: 8
  Kernel: Windows
  Kernel version: 6.2
  Architecture: x86_64

Build information:
  Compiler: MSVC 19.29.30038.1
  Build host: WINBUILD2
  OpenSSL version: OpenSSL 1.1.1k  25 Mar 2021

Application information:

General paths:
  Config directory: C:\ProgramData\icinga2\etc\icinga2
  Data directory: C:\ProgramData\icinga2\var\lib\icinga2
  Log directory: C:\ProgramData\icinga2\var\log\icinga2
  Cache directory: C:\ProgramData\icinga2\var\cache\icinga2
  Spool directory: C:\ProgramData\icinga2\var\spool\icinga2
  Run directory: C:\ProgramData\icinga2\var\run\icinga2

Old paths (deprecated):
  Installation root: C:\Program Files\ICINGA2\
  Sysconf directory: C:\ProgramData\icinga2\etc
  Run directory (base): C:\ProgramData\icinga2\var\run
  Local state directory: C:\ProgramData\icinga2\var

Internal paths:
  Package data directory: C:\Program Files\ICINGA2\\share\icinga2
  State path: C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state
  Modified attributes path: C:\ProgramData\icinga2\var\lib\icinga2/modified-attributes.conf
  Objects path: C:\ProgramData\icinga2\var\cache\icinga2/icinga2.debug
  Vars path: C:\ProgramData\icinga2\var\cache\icinga2/icinga2.vars
  PID path: C:\ProgramData\icinga2\var\run\icinga2/icinga2.pid

Error: test exception

Stacktrace:
 0# 0x00007FF798017B55 in icinga2
 1# 0x00007FF798536AFE in icinga2
 2# 0x00007FFE17921080 in VCRUNTIME140_1
 3# _NLG_Return2 in VCRUNTIME140_1
 4# RtlCaptureContext in ntdll
 5# 0x00007FF797FA1041 in icinga2
 6# terminate in ucrtbase
 7# 0x00007FF79850555A in icinga2
 8# 0x00007FF797FD3B44 in icinga2
 9# UnhandledExceptionFilter in KERNELBASE
10# memset in ntdll
11# _C_specific_handler in ntdll
12# _chkstk in ntdll
13# RtlWalkFrameChain in ntdll
14# RtlRaiseException in ntdll
15# RaiseException in KERNELBASE
16# _NLG_Return2 in VCRUNTIME140_1
17# RtlCaptureContext in ntdll
18# 0x00007FF7981EDFBF in icinga2
19# 0x00007FF797EE233E in icinga2
20# 0x00007FF797EEBD81 in icinga2
21# 0x00007FF798504C38 in icinga2
22# BaseThreadInitThunk in KERNEL32
23# RtlUserThreadStart in ntdll

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

Windows x86 SEH

Caught unhandled SEH exception.
Current time: 2021-07-08 13:18:10 -0000

  Application version: v2.12.4-50-gdd7eabeb0

System information:
  Platform: Windows
  Platform version: 8
  Kernel: Windows
  Kernel version: 6.2
  Architecture: x86_64

Build information:
  Compiler: MSVC 19.29.30038.1
  Build host: WINBUILD2
  OpenSSL version: OpenSSL 1.1.1k  25 Mar 2021

Application information:

General paths:
  Config directory: C:\ProgramData\icinga2\etc\icinga2
  Data directory: C:\ProgramData\icinga2\var\lib\icinga2
  Log directory: C:\ProgramData\icinga2\var\log\icinga2
  Cache directory: C:\ProgramData\icinga2\var\cache\icinga2
  Spool directory: C:\ProgramData\icinga2\var\spool\icinga2
  Run directory: C:\ProgramData\icinga2\var\run\icinga2

Old paths (deprecated):
  Installation root: C:\Program Files (x86)\ICINGA2\
  Sysconf directory: C:\ProgramData\icinga2\etc
  Run directory (base): C:\ProgramData\icinga2\var\run
  Local state directory: C:\ProgramData\icinga2\var

Internal paths:
  Package data directory: C:\Program Files (x86)\ICINGA2\\share\icinga2
  State path: C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state
  Modified attributes path: C:\ProgramData\icinga2\var\lib\icinga2/modified-attributes.conf
  Objects path: C:\ProgramData\icinga2\var\cache\icinga2/icinga2.debug
  Vars path: C:\ProgramData\icinga2\var\cache\icinga2/icinga2.vars
  PID path: C:\ProgramData\icinga2\var\run\icinga2/icinga2.pid

SEH exception:
  Code: 0x2a
  Address: 777FAAF2
  Flags: 0

Stacktrace:
 0# 0x00DCB35F in icinga2
 1# 0x00D8B272 in icinga2
 2# UnhandledExceptionFilter in KERNELBASE
 3# RtlCaptureStackContext in ntdll
 4# RtlGetAppContainerNamedObjectPath in ntdll

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

Windows x86 C++ Exception

Caught unhandled exception.
Current time: 2021-07-08 13:18:21 -0000

  Application version: v2.12.4-50-gdd7eabeb0

System information:
  Platform: Windows
  Platform version: 8
  Kernel: Windows
  Kernel version: 6.2
  Architecture: x86_64

Build information:
  Compiler: MSVC 19.29.30038.1
  Build host: WINBUILD2
  OpenSSL version: OpenSSL 1.1.1k  25 Mar 2021

Application information:

General paths:
  Config directory: C:\ProgramData\icinga2\etc\icinga2
  Data directory: C:\ProgramData\icinga2\var\lib\icinga2
  Log directory: C:\ProgramData\icinga2\var\log\icinga2
  Cache directory: C:\ProgramData\icinga2\var\cache\icinga2
  Spool directory: C:\ProgramData\icinga2\var\spool\icinga2
  Run directory: C:\ProgramData\icinga2\var\run\icinga2

Old paths (deprecated):
  Installation root: C:\Program Files (x86)\ICINGA2\
  Sysconf directory: C:\ProgramData\icinga2\etc
  Run directory (base): C:\ProgramData\icinga2\var\run
  Local state directory: C:\ProgramData\icinga2\var

Internal paths:
  Package data directory: C:\Program Files (x86)\ICINGA2\\share\icinga2
  State path: C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state
  Modified attributes path: C:\ProgramData\icinga2\var\lib\icinga2/modified-attributes.conf
  Objects path: C:\ProgramData\icinga2\var\cache\icinga2/icinga2.debug
  Vars path: C:\ProgramData\icinga2\var\cache\icinga2/icinga2.vars
  PID path: C:\ProgramData\icinga2\var\run\icinga2/icinga2.pid

Error: test exception

Stacktrace:
 0# 0x00DCB35F in icinga2
 1# 0x00D5A329 in icinga2
 2# terminate in ucrtbase
 3# 0x0128E24C in icinga2
 4# 0x00D8AF72 in icinga2
 5# UnhandledExceptionFilter in KERNELBASE
 6# RtlCaptureStackContext in ntdll
 7# RtlGetAppContainerNamedObjectPath in ntdll

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

Al2Klimov self-assigned this Jul 2, 2021

icinga-probot bot added this to the 2.12.5 milestone Jul 2, 2021

icinga-probot bot added area/log Logging related consider backporting Should be considered for inclusion in a bugfix release enhancement New feature or request labels Jul 2, 2021

Al2Klimov force-pushed the feature/improve-crashlog-212 branch 2 times, most recently from 0e874af to bf1c33c Compare July 5, 2021 14:20

Al2Klimov removed their assignment Jul 5, 2021

Al2Klimov requested a review from julianbrost July 5, 2021 14:31

Al2Klimov marked this pull request as ready for review July 6, 2021 08:25

N-o-X removed the consider backporting Should be considered for inclusion in a bugfix release label Jul 6, 2021

julianbrost requested changes Jul 6, 2021

View reviewed changes

Al2Klimov force-pushed the feature/improve-crashlog-212 branch from bf1c33c to e4d34de Compare July 6, 2021 13:54

Al2Klimov requested a review from julianbrost July 6, 2021 13:55

Al2Klimov force-pushed the feature/improve-crashlog-212 branch from e4d34de to 60b9afc Compare July 7, 2021 15:16

julianbrost added 14 commits July 8, 2021 14:53

Run termination handler for uncaught C++ exceptions on Windows

b5017b7

On Windows, the termination handler is executed for uncaught C++ exceptions unless a SEH unhandled exception filter is also set. In this case, this filter has to explicitly chain the default filter to keep this behavior.

Use boost::stacktrace instead of custom implementation in Windows SEH…

9ebd812

… filter

Move error message and time to the beginning of the SEH crash log

8b67e4a

This is more similar to the normal exception crashlog which also states the problem and time at the beginning of the file.

Print details in uncaught SEH exception handler

4a29c39

Replace icinga::StackTrace with boost::stacktrace::stacktrace

8b2f463

Provides roughly the same functionality but works better on certain platforms (especially Windows) and is less code to maintain.

Pass fallback stacktrace to DiagnosticInformation in terminate handler

996f280

By default, DiagnosticInformation uses the stack trace saved when the exception was thrown, but this mechanism is not in use on Windows. Gathering a stacktrace in the terminate handler serves as a fallback.

Add documentation for cast_exception function

df59aa0

Restructure stack and context trace selection in DiagnosticInformatio…

9fcc781

…n and document behavior The logic for selecting the traces to print stays the same, but there are fewer nested ifs now. This changes the format of the returned string a bit by adding a heading for both traces.

Add some comments to __cxa_throw

2330ab5

Maybe this will save the next person who has to look at this code some time. Please don't blame me for the implementation, I'm just trying to reconstruct what it does.

docs: mention use of boost::stacktrace

51bb751

Windows: require at least MSVC 19.20 to build

cb3febf

Older versions of MSVC fail to rethrow an unhandled C++ exception (using `throw;`) in the termination handler (`std::set_terminate`), however Icinga relies on this behavior in its crash handler (`Application::ExceptionHandler`).

Begin crash log for SIGABRT with error message and timestamp

2310cdb

This makes the format more similar to what the uncaught C++ and SEH exception handlers write. Previously there was no indication in the crash log that a SIGABRT happened.

Crash handlers: use more compact string representation

a2e5cfd

julianbrost added 5 commits July 8, 2021 14:53

Add a test case for the stack trace formatter

1498727

CMakeLists: use HAVE_LIBEXECINFO only after actually checking for it

1a82a9f

So far, the check that actually sets HAVE_LIBEXECINFO was executed after it was already used to add dependencies.

Compile with -D_GNU_SOURCE

c3f77be

Needed by `boost::stacktrace` for `_Unwind_Backtrace()`.

Add comments to stack trace formatter and test case

d8ab328

Stacktrace test: try to prevent inlining even harder

7bfe896

- Explicitly disable optimizations for MSVC - Make stack_test_func_a bigger

julianbrost force-pushed the feature/improve-crashlog-212 branch from 60b9afc to 7bfe896 Compare July 8, 2021 12:53

julianbrost approved these changes Jul 8, 2021

View reviewed changes

julianbrost mentioned this pull request Jul 8, 2021

Icinga 2.12.5 #8890

Closed

julianbrost merged commit 289a872 into support/2.12 Jul 9, 2021

icinga-probot bot deleted the feature/improve-crashlog-212 branch July 9, 2021 06:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve crashlog #8865

Improve crashlog #8865

Al2Klimov commented Jul 2, 2021

Al2Klimov commented Jul 2, 2021

julianbrost left a comment

julianbrost commented Jul 8, 2021

julianbrost commented Jul 8, 2021

Improve crashlog #8865

Improve crashlog #8865

Conversation

Al2Klimov commented Jul 2, 2021

Backport of

Al2Klimov commented Jul 2, 2021

Blocked by

julianbrost left a comment

Choose a reason for hiding this comment

julianbrost commented Jul 8, 2021

julianbrost commented Jul 8, 2021

Linux C++ Exception

Linux SIGABRT

Windows x86-64 SEH

Windows x86-64 C++ Exception

Windows x86 SEH

Windows x86 C++ Exception