napi: crash when a callback throws and returns an invalid napi_value; napi_create_dataview returns napi_ok after throwing #36941

Version: deno 2.9.7 (also 2.9.5), Linux x86_64 and Windows x64. Same code on main (1b48a20).

Two related problems in how Deno handles a pending exception in Node-API, found while running the Node-API conformance tests (js-native-api/test_function and js-native-api/test_dataview crash).

1. A callback that throws and then returns an invalid napi_value crashes Deno.

Node ignores the callback's return value whenever an exception is pending (js_native_api_v8.cc):

if (!exceptionOccurred && (result != nullptr)) {
  cbinfo_.GetReturnValue().Set(V8LocalValueFromJsValue(result));
}

That check came in with nodejs/node#39476, for nodejs/node-addon-api#1025: the C++ wrapper with exceptions enabled returned an invalid value after throwing. Node's own test_function.c (TestBadReturnExceptionPending) checks it by returning (napi_value)0xFFFFFFFFF after napi_throw_error.

In Deno, call_fn in ext/napi/function.rs rethrows last_exception and then still calls rv.set(value), so it dereferences that value.

2. napi_create_dataview past the end of the buffer returns napi_ok.

Node throws a RangeError and returns napi_pending_exception (js_native_api_v8.cc). Deno throws the RangeError but returns the status of napi_throw_range_error, which is napi_ok, and leaves *result unwritten. An addon that checks the status, the usual pattern, then goes on to return *result. That value was never set, so it lands in problem 1.

Repro (two small standalone addons, built with gcc -shared -fPIC -I<node headers>):

static napi_value throw_return_invalid(napi_env env, napi_callback_info info) {
  napi_throw_error(env, NULL, "thrown");
  return (napi_value)0xFFFFFFFFF;
}

static napi_value typical(napi_env env, napi_callback_info info) {
  napi_value ab, dv = (napi_value)0xFFFFFFFFF;  /* stands in for an uninitialized local */
  void *data;
  napi_create_arraybuffer(env, 16, &data, &ab);
  if (napi_create_dataview(env, 8, ab, 12, &dv) != napi_ok) return NULL;
  return dv;
}

Results on Linux (node:24-bookworm image, 3 runs each, all identical):

throwReturnInvalid   node        threw Error thrown
throwReturnInvalid   deno 2.9.7  exit 139 (SIGSEGV), no output
typical              node        threw RangeError ERR_NAPI_INVALID_DATAVIEW_ARGS
typical              deno 2.9.7  exit 139 (SIGSEGV), no output
status               node        {"status":10,"resultWritten":false,"exceptionPending":true}
status               deno 2.9.7  {"status":0,"resultWritten":false,"exceptionPending":true}

Returning NULL or a valid value after throwing works in Deno today; only an invalid non-null value crashes. On Windows the two CTS files exit with 0xc0000005 on 2.9.5 and 2.9.7, 3 of 3 runs.

Expected: both calls throw, as in Node (Error: thrown and RangeError [ERR_NAPI_INVALID_DATAVIEW_ARGS]), and napi_create_dataview returns napi_pending_exception (10).

I have a small fix with regression tests in tests/napi and am opening a PR for it.