[BUG] Uncaught exception in main thread doesn't halt JVM

[Bukhtawar]

Describe the bug

[2024-08-13T19:53:41,233][ERROR][o.o.b.Bootstrap          ] [a28ce3bd16681e0092591169b3fd0b0a] Exception
java.lang.IllegalStateException: Unable to restore remote index metadata
        at org.opensearch.index.recovery.RemoteStoreRestoreService.restore(RemoteStoreRestoreService.java:158)
        at org.opensearch.gateway.GatewayMetaState.start(GatewayMetaState.java:178)
        at org.opensearch.node.Node.start(Node.java:1546)
        at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:339)
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:413)
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
        at org.opensearch.cli.Command.main(Command.java:101)
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)

[2024-08-14T00:04:18,880][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [ee3fe29191a528a0f473bb2fa9b77f12] uncaught exception in thread [main]
java.lang.IllegalStateException: Unable to restore remote index metadata

The above stack trace was noted in a non-fatal exception(not an instance of Error) on main thread which didn't halt the JVM keeping the process in a stuck state. It would have been preferable to shut the running process to allow monitoring or nanny scripts to restart the process.

OpenSearch/server/src/main/java/org/opensearch/bootstrap/OpenSearchUncaughtExceptionHandler.java

Lines 53 to 77 in 411b279

	public void uncaughtException(Thread thread, Throwable t) {
	if (isFatalUncaught(t)) {
	try {
	onFatalUncaught(thread.getName(), t);
	} finally {
	// we use specific error codes in case the above notification failed, at least we
	// will have some indication of the error bringing us down
	if (t instanceof InternalError) {
	halt(128);
	} else if (t instanceof OutOfMemoryError) {
	halt(127);
	} else if (t instanceof StackOverflowError) {
	halt(126);
	} else if (t instanceof UnknownError) {
	halt(125);
	} else if (t instanceof IOError) {
	halt(124);
	} else {
	halt(1);
	}
	}
	} else {
	onNonFatalUncaught(thread.getName(), t);
	}
	}

Related component

Storage:Remote

To Reproduce

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

JVM should halt on uncaught exception on main thread

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[andrross]

I wonder if we should be catching arbitrary exceptions here and returning an error exit code?

OpenSearch/libs/cli/src/main/java/org/opensearch/cli/Command.java

Lines 100 to 116 in c0852f8

	try {
	mainWithoutErrorHandling(args, terminal);
	} catch (OptionException e) {
	// print help to stderr on exceptions
	printHelp(terminal, true);
	terminal.errorPrintln(Terminal.Verbosity.SILENT, "ERROR: " + e.getMessage());
	return ExitCodes.USAGE;
	} catch (UserException e) {
	if (e.exitCode == ExitCodes.USAGE) {
	printHelp(terminal, true);
	}
	if (e.getMessage() != null) {
	terminal.errorPrintln(Terminal.Verbosity.SILENT, "ERROR: " + e.getMessage());
	}
	return e.exitCode;
	}
	return ExitCodes.OK;

[soosinha]

For remote state restore flow, this was fixed to throw an Error instead of Exception so that the process exists: https://github.com/opensearch-project/OpenSearch/pull/15950/files

I will leave this issue open in case we want to modify the generic exception handling as well for exiting the process.