NodeJS native modules made difficult by distribution packages.

[nicolasnoble]

As a NodeJS native module developer, we've been relying on NodeJS' ABI to be able to publish pre-compiled binary packages to ease the installation process.

We recently discovered that the Debian / Ubuntu / Arch Linux packages (and maybe more ?) aren't ABI compatible with the official NodeJS distributions, which breaks pre-built binary packages in difficult ways. More specifically, these distributions are shipping NodeJS 8 that is linked against OpenSSL 1.1. The official NodeJS distribution is linking against OpenSSL 1.0, and there has been ABI breaking changes between the two versions. Therefore, a NodeJS 8 native module built against the official runtime will fail to work properly on Debian or Arch Linux's runtime.

The package we are publishing (grpc) is affected by this, but I also managed to identify at least a second package that is also affected: uws. The initial issue was reported and (painfully) investigated over on the gRPC bug tracker: grpc/grpc-node#341

I've then filed a detailed issue on Ubuntu's issue tracker to expose the problem with a reproduction case here.

I'm not sure what would the NodeJS' stance be on this issue, hence me creating this issue here to discuss the problem. I believe that the ABI breakage from Debian and Arch Linux is an oversight and an honest mistake, but I'm not sure what the resolution should be. I know that Arch Linux has an openssl-1.0 package that the nodejs package can depend upon, but I don't think this is viable for Debian / Ubuntu.

My opinion is that at the simplest, the Node foundation should publish Vendoring Guidelines, describing what it means to ship a correct NodeJS runtime, including notes on how to properly expose ABI compatible symbols for native modules.

cc @ofrobots.

[ofrobots]

/cc @nodejs/tsc. IMO, we need to have stronger guidelines about ABI compatibility for vendors and distributors shipping Node.js. If something advertises itself as "Node.js 8.11.1" it should be compatible with the official Node.js 8.11.1.

[addaleax]

@ofrobots I remember talking about this in a TSC meeting before the Node 10 release (with @rvagg I think)… we knew this was coming. :/

In retrospect, maybe we should have had different NODE_MODULE_VERSION values for the two OpenSSL lines we support. Would that make sense as a pattern for the future?

[nicolasnoble]

@addaleax mangling the NODE_MODULE_VERSION should work, yes, but this multiplies the number of packages that need to be distributed. As of today, grpc is shipping with 123 precompiled binaries for a big matrix combination of runtimes (node vs electron), platform (Windows, Linux, MacOS), architecture, and runtime versions, and we have a fairly convoluted and complex build script to generate all of these. Adding yet another dimension to this matrix doesn't make me feel warm and fuzzy, especially since it would now mean that the build script needs to acquire two different "versions" of the same node version, potentially using two different sources.

I wouldn't be looking forward to this :-)

[ofrobots]

OpenSSL is probably the main suspect where the ABI between different distributors could end up differing; but there could be other environmental differences, e.g. choice of the C++ compiler, C++ std library or build flags that that make a non-official release potentially incompatible with official releases.

I think having different NODE_MODULE_VERSION values would have help this specific case of the two OpenSSL release lines, but I think there is a general problem if the ABI ends up being different in other ways.

I guess my question is: Is it a problem that distributor ships an ABI incompatible version of Node that uses the same NODE_MODULE_VERSION? I think it would good if the Node.js project could build consensus on this question, and then publish clear guidelines for vendors.

My personal opinion is that ABI incompatibility fragments the ecosystem and pushes the cost of figuring out compatibility to our users. We should document clear guidelines about ABI compatibility for distributors. Native module authors now also have to worry about shipping multiple versions of the binaries for the same Node version, as @nicolasnoble points out.

If it is not compatible with Node, it is not Node.

[nicolasnoble]

I discovered another, fairly edge case package: termux's nodejs package for Android.

termux/termux-packages#2674

[joyeecheung]

cc @nodejs/delivery-channels

[kapouer]

Open-source distributions implicitely assume users of Node will always compile addons and not install untrusted binaries when it can be avoided. Maybe checking for NODE_MODULE_VERSION is not enough, and shared libraries against which Node has been compiled by distributor should also be taken into account ?

[nicolasnoble]

@kapouer I world say they are assuming wrong of that's the case. Also your suggestion is basically what @addaleax said a few comments up.

[kapouer]

@nicolasnoble my point was to make sure that what decision goes for openssl goes also for uv, zlib, openssl, c-ares, nghttp2, http-parser, icu, should multiple versions of one of these deps be supported.

[nicolasnoble]

Right, which would in effect make prebuilt binaries impossible to ship for anything but the official runtimes available on the website. This is an important trade-off to consider.

[kapouer]

Maybe N-API is the solution, but i'm not even sure of that.

[nicolasnoble]

I don't think that N-API does anything about transitive dependencies such as openssl or libuv, no.

[kapouer]

It is really important to emphasize the fact that this kind of problem is a general problem of open-source distributions, not particularly related to Node. There is no quick fix and the situation with Node ABI is not as simple as it looks.
The simplest solution is to fallback to building the addon from source - because open-source distributions make that really easy to achieve. sharp proposes this.
Alternatives are:

• working around a c++ addon or relying on a general addon already known to work (node-ffi ?)
• distribute the c++ addon in the open-source distribution itself. Depending on the addon, it is either very quick (because popular) or not. Summoning "at nodejs/delivery-channels" with a clear request might help.
• flatpak, docker, etc... which are generic and working solutions