stacktrace without --stackTrace runtime overhead

[timotheecour]

Drawbacks of --stackTrace:

• --stackTrace incurs significant runtime overhead, and in fact {.push stacktrace:off.} is used at several places in nim codebase (eg lib/system/gc.nim).
  One reason for this runtime overhead is nimfr_ + popFrame inserted upon entering/exiting each function.

• --stackTrace is not flexible enough: it does not help with getting stacktraces when inside C/C++ code (eg if nim calls into C/C++ code)

• --stackTrace doesn't allow following use case: shipping code to customers without debug symbols while still allowing to symbolicate customer's stacktraces using a corresponding non-shipped .dSYM file containing DWARF debug info

IMO a better approach is possible (could use a new flag, maybe --stacktraceNative ; not to be confused with --debugger:native), based on using standard tools:

• libunwind or backtrace_symbols to get the stackframe addreses (libunwind is more accurate than backtrace_symbols, eg the backtrace_symbols often misses the parent frame when a signal handler is executing, which is the most important one)
• dladdr to get object file load addresses + symbol name
• cxa_demangle for demangling c++ symbols, or
• ndi files (generated by nim --debuginfo:on) to demangle nim symbols + get file/line/col info, eg:
  main main__Na4naun1nNzHO8Lb2OfwMQ /Users/timothee/git_clone/nim/timn/src/timn/tests/stacktrace/tstacktraces.nim 65 5
• atos (on mac), or addr2line (on linux) to get file+line info from stack address + object file load address

A subset of these tools could be sufficient for common use cases, eg:

• libunwind + dladdr + ndi files would be enough to get stacktraces for nim procs (with file+line info); while cxa_demangle and (atos or addr2line) would be needed for C/C++ code

notes

• --lineDir:on does not insert #line pragma before a nim proc declaration, so atos reports file/line info pointing to the cgen'd file instead of the original nim sources; however the ndi file has that info
• looking at the sources, I found one can do: --stacktrace:off -d:nativeStacktrace however this doesn't seem documented, it only works with nim c (not nim cpp, which produces codegen errors), and it doesn't demangle nim symbols nor shows any line numbers

requires

• osx: dsymutil needs to be called for debug builds to keep debug info #12735 osx: dsymutil needs to be called for debug builds to keep debug info

[Araq]

But we have the code for that, check system/excpt.nim line 140 and following. I never use it because it produces uglier stack traces and doesn't work on every OS.

[mratsim]

Having fast and good stacktraces would be a significant help for Nimbus as well.

We will soon onboard new users unfamiliar with Nim and that are not developers.

We need a fast client but we also need excellent debugging as we can't ask people to recompile.
Right now we try to work around with exhaustive logging but this requires us to not forget to add logs.

(when I say soon, an article on our testnet is scheduled before the end of the day)