Non-blocking backend for `Process` APIs

[armanbilge]

There are two interesting candidates for this:

• https://github.com/brettwooldridge/NuProcess
• https://github.com/jnr/jnr-process

Probably NuProcess is the better established of the two. jnr-process seems to expose a lower-level API. Suffice to say, both need a closer look, particularly at the threading model.

Similar to UnixSockets, we can add one of these as an optional dependency, and use reflection to try and detect it at runtime, otherwise falling back to the blocking JDK APIs.

[rossabaker]

Eek! Optional dependencies are a linker disaster waiting to happen. Even with fallbacks, the end user has to maintain a fragile dependency relationship instead of the standard transitivity and eviction warnings. Quoth the Maven (emphasis mine):

Optional dependencies are used when it's not possible (for whatever reason) to split a project into sub-modules. The idea is that some of the dependencies are only used for certain features in the project and will not be needed if that feature isn't used. Ideally, such a feature would be split into a sub-module that depends on the core functionality project. This new subproject would have only non-optional dependencies, since you'd need them all if you decided to use the subproject's functionality.

However, since the project cannot be split up (again, for whatever reason), these dependencies are declared optional. If a user wants to use functionality related to an optional dependency, they have to redeclare that optional dependency in their own project. This is not the clearest way to handle this situation, but both optional dependencies and dependency exclusions are stop-gap solutions.

I would strongly recommend a module for this sort of thing.

[rossabaker]

Some comparisons to other places we do classpath discovery:

1. SLF4J bindings, JDBC drivers: these implement a specified API, and the implementations are mostly called only called via that API, so the diamond risk is relatively low. Also, it wasn't our decision.
2. UnixDomainSocket: JNR is its own API with its own dependencies, so carries somewhat higher risk. It looks to have been stable the past few years, but a breaking change will be a sad day for FS2. A saving grace is that Java >= 17 adoption is asymptotically approaching 100%.
3. These process libraries: similar risks to UnixDomainSocket, but without Java 17 adoption relieving the problem over time.

All this magic potentially gets a better implicit on the algebra's companion object, but I don't think it's worth the classpath landmine.

[mpilquist]

Are the benefits of a non-blocking process backend worth the maintenance costs?

[rossabaker]

I have microlibrary fatigue, but I think that's a good solution here: it's there if you need it, and doesn't tax a foundational library when most people don't.

[armanbilge]

There's another option we can pursue now: on Loom / Virtual Threads hopefully the implementation has been fixed to be non-blocking under-the-hood (someone should check this ...). After typelevel/cats-effect#3870 is released we can run JDK Process operations on a VT executor if available and hopefully get a non-blocking backend.

[armanbilge]

hopefully the implementation has been fixed to be non-blocking under-the-hood (someone should check this ...)

The bad news is that it seems that every JDK Process is associated with a "process reaper" thread, so it's not exactly fixed. However, virtual threads can still help us by making Process#waitFor non-blocking, since it's just waiting for synchronization conditions from the reaper thread.

https://github.com/openjdk/jdk/blob/4fc72b8e4b22db2aa3217afb5b8c30d496589eb8/src/java.base/unix/classes/java/lang/ProcessImpl.java#L409-L419

[onsah]

Hi @armanbilge I would like to work on this issue. Can you assign it to me?