feat: external crash reporter

[jpnurmi]

Context:

• Feature Request: Support option to launch Crash Modal Dialogue to collect User Feedback at Crash Time #1223

Depends on:

• feat: external crash reporter crashpad#131

Works with:

	Windows	Linux	macOS
inproc	✓	✓	✓
breakpad	✓	✓	✓
crashpad	✓	✓	✓

TODO:

• 1. Clean up & productize
• 2. Tests
• 3. Transfer envelope ownership
• 4. Rename to Crash Reporter
• 5. Split off separate PRs
  • feat: add sentry__path_get_mtime() #1317
  • feat: add sentry__process_spawn() #1318
  • ref: crashpad: pre-generate crash event ID #1319
  • feat: read and capture envelopes #1320
• 6. Docs
• 7. Last crash detection:
  • Last crash detection API parity between SDK's #1325

Usage:

1. On startup, register a path to an external crash reporter executable

   sentry_options_set_crash_reporter_path(options, "/path/to/my/crash-reporter-app");

2. When a crash is captured, the external crash reporter process is spawned and detached, and the path to a crash event envelope is passed as an argument.

Note

The external crash reporter is responsible for submitting the envelope to Sentry. It may ask for user consent or feedback, and attach additional data to the envelope.

The command-line argument allows using almost any app for debugging and testing. Text editor, web browser, anything that is capable of handling a single argument and displaying a plain-text JSON(L) file.

See tests/fixtures/crash_reporter/crash_reporter.c for a minimal crash reporter implementation used for integration tests.

Examples:

// https://github.com/bruno-garcia/sentry-crash-reporter
sentry_options_set_crash_reporter_path(options, "/path/to/Sentry.CrashReporter");

sentry_options_set_crash_reporter_path(options, "/usr/bin/xdg-open");

Close: #1223

[jpnurmi]

@sentry review

[seer-by-sentry]

On it! We are reviewing the PR and will provide feedback shortly.

[seer-by-sentry]

PR Description

This pull request introduces the ability to launch a separate feedback handler process upon application crashes. This allows for collecting user feedback or performing custom actions outside the main application process, ensuring that these operations don't interfere with crash reporting or application stability.

Click to see more

Key Technical Changes

1. New sentry_options configuration: Added feedback_handler_path to sentry_options to specify the path to the feedback handler executable. This includes both narrow and wide character versions of the API.
2. Process spawning: Implemented a cross-platform process spawning mechanism (sentry_process.h, sentry_process_unix.c, sentry_process_windows.c, sentry_process_none.c) to launch the feedback handler as a detached subprocess.
3. Crash event persistence: Modified the crash reporting backends (sentry_backend_breakpad.cpp, sentry_backend_crashpad.cpp, sentry_backend_inproc.c) to persist the crash event data to disk in a dedicated 'feedback' directory within the Sentry database before launching the handler. The path to this persisted event is passed as an argument to the feedback handler.
4. Event ID generation: The crashpad backend now generates a predictable event ID at startup, allowing the feedback handler to locate the event data even before the crash occurs. This is important for scenarios where the feedback handler needs to access the event data immediately after the crash.
5. Feedback file cleanup: Added logic to prune feedback files older than 2 hours to prevent excessive disk usage.

Architecture Decisions

1. Detached subprocess: The feedback handler is launched as a detached subprocess to avoid blocking the main application's crash reporting process and to prevent the feedback handler's failures from affecting the main application.
2. File-based event passing: The crash event data is passed to the feedback handler via a file on disk to avoid complex inter-process communication mechanisms and to ensure that the data is available even if the feedback handler crashes.
3. UUID-based filename: The event data file is named using the event's UUID to ensure uniqueness and prevent naming conflicts.

Dependencies and Interactions

1. sentry_options: The new feedback_handler_path option is added to the sentry_options struct, requiring changes to the options parsing and management logic.
2. Crash reporting backends: The crash reporting backends now depend on the process spawning mechanism to launch the feedback handler.
3. sentry_database: The sentry_database component is used to persist the crash event data to disk and to manage the 'feedback' directory.
4. sentry_uuid: The sentry_uuid component is used to generate unique filenames for the event data files.

Risk Considerations

1. Security: The feedback handler path should be carefully validated to prevent malicious executables from being launched. Input validation and sanitization are crucial to mitigate command injection vulnerabilities.
2. Resource exhaustion: The feedback handler could potentially consume excessive resources (CPU, memory, disk space). Resource limits and monitoring should be considered.
3. File system access: The feedback handler requires read access to the crash event data file. Permissions and access control should be carefully configured.
4. Error handling: Failures during process spawning or file persistence should be handled gracefully to prevent data loss or application instability.
5. Data privacy: Ensure that any user feedback collected by the handler adheres to privacy regulations and guidelines.

Notable Implementation Details

1. The sentry__process_spawn function uses fork() and execv() (or CreateProcessW() on Windows) to launch the feedback handler as a detached subprocess.
2. The sentry__run_write_feedback function copies the event envelope to the feedback directory.
3. The crashpad backend generates a predictable event ID at startup to allow the feedback handler to locate the event data even before the crash occurs.
4. The feedback directory is periodically cleaned up to remove old event data files.

[supervacuus]

because only the minidump endpoint understands msgpacked __sentry-breadcrumbN attachments. I suppose Crashpad needs to unpack and merge the two breadcrumb files to be able to hand over a proper event structure to the external crash reporter.

Oh, I didn't raise this as an issue because I expected you to have tested this already. The same applies to __sentry-event, of course, or any other MsgPack payload. Decoding and converting it to the JSON payload is the right call, then.

[supervacuus]

because only the minidump endpoint understands msgpacked __sentry-breadcrumbN attachments. I suppose Crashpad needs to unpack and merge the two breadcrumb files to be able to hand over a proper event structure to the external crash reporter.

Oh, I didn't raise this as an issue because I expected you to have tested this already. The same applies to __sentry-event, of course, or any other MsgPack payload. Decoding and converting it to the JSON payload is the right call, then.

Of course, another option would also be to expose a minidump endpoint client and add user feedback as a decodable payload to the minidump endpoint in relay. However, I think converting MsgPack to JSON on the client is less effort and still sensible.

[jpnurmi]

The reference GUI made with .NET 9.0 + Uno Platform is now available here: https://github.com/getsentry/sentry-desktop-crash-reporter (WIP!)

For example, launching it straight from the build directory on Windows:

sentry_options_set_external_crash_reporter_pathw(options, L"C:\\path\\to\\sentry-desktop-crash-reporter\\Sentry.CrashReporter\\bin\\Release\\net9.0-desktop\\Sentry.CrashReporter.exe");