crypto: make generatePrime/checkPrime interruptible

The `generatePrime` and `checkPrime` functions in the `crypto`
module are only somewhat interruptible. This change makes it
possible to interrupt these more reliably. Note that generating
overly large primes can still take a long time and may not be
interruptible as this mechanism relies on a callback to check
for stopping conditions but OpenSSL may perform a long running
operation without calling the callback right away.

Fixes: #56449

Author: jasnell

doc/api/crypto.md

@@ -3940,6 +3940,13 @@ By default, the prime is encoded as a big-endian sequence of octets
 in an {ArrayBuffer}. If the `bigint` option is `true`, then a {bigint}
 is provided.
 
+The `size` of the prime will have a direct impact on how long it takes to
+generate the prime. The larger the size, the longer it will take. Because
+we use OpenSSL's `BN_generate_prime_ex` function, which provides only
+minimal control over our ability to interrupt the generation process,
+it is not recommended to generate overly large primes, as doing so may make
+the process unresponsive.
+
 ### `crypto.generatePrimeSync(size[, options])`
 
 <!-- YAML
@@ -3981,6 +3988,13 @@ By default, the prime is encoded as a big-endian sequence of octets
 in an {ArrayBuffer}. If the `bigint` option is `true`, then a {bigint}
 is provided.
 
+The `size` of the prime will have a direct impact on how long it takes to
+generate the prime. The larger the size, the longer it will take. Because
+we use OpenSSL's `BN_generate_prime_ex` function, which provides only
+minimal control over our ability to interrupt the generation process,
+it is not recommended to generate overly large primes, as doing so may make
+the process unresponsive.
+
 ### `crypto.getCipherInfo(nameOrNid[, options])`
 
 <!-- YAML

src/crypto/crypto_random.cc

@@ -28,6 +28,25 @@ using v8::Uint32;
 using v8::Value;
 
 namespace crypto {
+namespace {
+using BNGENCBPointer = DeleteFnPtr<BN_GENCB, BN_GENCB_free>;
+
+BNGENCBPointer getBN_GENCB(Environment* env) {
+  // The callback is used to check if the operation should be stopped.
+  // Currently, the only check we perform is if env->is_stopping()
+  // is true.
+  BNGENCBPointer cb(BN_GENCB_new());
+  BN_GENCB_set(
+      cb.get(),
+      [](int a, int b, BN_GENCB* cb) -> int {
+        Environment* env = static_cast<Environment*>(BN_GENCB_get_arg(cb));
+        return env->is_stopping() ? 0 : 1;
+      },
+      env);
+  return std::move(cb);
+}
+
+}  // namespace
 MaybeLocal<Value> RandomBytesTraits::EncodeOutput(
     Environment* env, const RandomBytesConfig& params, ByteSource* unused) {
   return v8::Undefined(env->isolate());
@@ -150,13 +169,14 @@ bool RandomPrimeTraits::DeriveBits(Environment* env,
   // Make sure the CSPRNG is properly seeded.
   CHECK(ncrypto::CSPRNG(nullptr, 0));
 
-  if (BN_generate_prime_ex(
-          params.prime.get(),
-          params.bits,
-          params.safe ? 1 : 0,
-          params.add.get(),
-          params.rem.get(),
-          nullptr) == 0) {
+  BNGENCBPointer cb = getBN_GENCB(env);
+
+  if (BN_generate_prime_ex(params.prime.get(),
+                           params.bits,
+                           params.safe ? 1 : 0,
+                           params.add.get(),
+                           params.rem.get(),
+                           cb.get()) == 0) {
     return false;
   }
 
@@ -189,12 +209,10 @@ bool CheckPrimeTraits::DeriveBits(
     ByteSource* out) {
 
   BignumCtxPointer ctx(BN_CTX_new());
+  BNGENCBPointer cb = getBN_GENCB(env);
 
   int ret = BN_is_prime_ex(
-            params.candidate.get(),
-            params.checks,
-            ctx.get(),
-            nullptr);
+      params.candidate.get(), params.checks, ctx.get(), cb.get());
   if (ret < 0) return false;
   ByteSource::Builder buf(1);
   buf.data<char>()[0] = ret;