Switch to node 8 / yarn

[edmundoa]

Node 8.9.1 was released a few days ago and it is a LTS release. As we are still in an early stage for 3.0, we can switch to it and get some improvements in our builds while minimizing the risks of this update. This version comes along with npm 5, which made important changes in the way local dependencies are installed:

npm install ./packages/subdir will now create a symlink instead of a regular installation. file://path/to/tarball.tgz will not change – only directories are symlinked.

We are heavily affected by that change in npm 5, as our internal graylog-web-plugin and eslint-config-graylog packages will be installed in a completely different way as before, and so will their dependencies.

On the past few days I have been trying to find some solution, and in the end I decided to go with Yarn in the future. It is a drop-in replacement of npm, so we could still go back to npm if we are not satisfied, and we only need to do some small changes to keep our builds working as they were before.

This PR switches our builds to Node 8.9.1 and Yarn 1.3.2, adapts some configuration files to that change, and also fixes an issue in the way we ignored files in the graylog-web-plugin module. The build should still work in npm v4, as long as Yarn is also installed in the system.

Before merging this change we should ensure it works for a couple of people, both in full rebuilds with maven, as in development builds.

[edmundoa]

@bernd already asked, and yes, the warnings are afaik "okay" because Yarn is warning that some of our required packages are coming through graylog-web-plugin, instead of depending on them directly.

[edmundoa]

I also want to explain a bit more about what I saw during my tests with local modules for future reference. I hope I don't miss anything, but it's in the end a complex process and I only grasp some things that are done there. I hope it helps to understand what's going on, since we can't write comments in the package.json file.

npm v4.6

Doing npm install in the graylog2-web-interface results in:

1. Calling the prepare and prepublish lifecycle events in the local modules (more importantly in graylog-web-plugin). That is installing the dependencies for the module and compiling with babel
2. Create a package of any local dependencies (aka npm pack)
3. Fetching dependencies
4. Call preinstall lifecycle event for local modules in a staging area
5. Installing dependencies, which included moving the packed local modules into the node_modules folder
6. Call postinstall lifecycle event for local modules in the node_modules folder

Both preinstall and postinstall events are called on the module once it is unpacked, so there is no source code available any more.

Also all binaries installed are linked in the node_modules/.bin folder.

yarn v1.3

Doing yarn install in the graylog2-web-interface results in:

1. Fetching dependencies
2. Installing dependencies. Local modules are copied to node_modules as they are in the file system
3. Call preinstall lifecycle event for local modules in the node_modules folder
4. Call postinstall lifecycle event for local modules in the node_modules folder

As you can see, there's no call to prepare and prepublish when using Yarn and local modules. Also preinstall and postinstall include all files in the local module, so we can build them in one of those events, which is what we do. Transitive dependencies of local modules are handled like any other packages dependencies, therefore being hoisted in the upper node_modules folder.

Binaries are also linked in the node_modules/.bin folder.

npm v5.5

This version has different behaviours when the local dependency is in a subfolder, or on a different tree.

First for a subfolder, like when executing in graylog2-web-interface:

1. Fetching dependencies
2. Installing dependencies. This links local modules in the node_modules folder. Transitive dependencies from the local modules are hoisted in the upper node_modules folder
3. Call preinstall lifecycle event for local modules in the linked folder
4. Call postinstall lifecycle event for local modules in the linked folder

This case is quite similar to what Yarn v1.3 does, just using symlinks instead of copying folders.

Now let's see what happens when installing a local module from a different subtree, like when executing npm install in graylog-plugin-map-widget:

1. Fetching dependencies
2. Installing dependencies. This links local modules in the node_module folder, but does not hoist transitive dependencies
3. Call preinstall lifecycle event for local dependencies in the linked folder
4. Call postinstall lifecycle event for local dependencies in the linked folder

The little change on how dependencies are installed makes that our plugins will have very little dependencies, as most of them come through the (local) graylog-web-plugin module. For instance, with npm 5 we cannot call webpack, eslint or babel, without depending directly on them first, creating a package of graylog-web-plugin (.tgz` files have the old behaviour), or installing the package from a registry.

I haven't found a way of using the "old" behaviour, just some (mostly unanswered) questions in github.

[edmundoa]

@bernd also saw another issue with yarn.lock including a reference to my local cache folder. That is a known issue: yarnpkg/yarn#4388

[kroepke]

Testing the production build worked well for me. The unmet peer dependencies are still there, of course, but other than that I did not notice anything unusual.

[bernd]

Please do!

[kroepke]

👍