Support for caching yarn workspaces' node_modules

[kamsar]

Which problem is this feature request solving?

https://community.netlify.com/t/gatsby-js-yarn-workspaces-and-caching-node-modules/6571

Given a structure such as this with Yarn Workspaces:

package.json - monorepo package defines yarn workspaces
yarn.lock - this root yarn.lock has ALL the dependencies in it for the whole set of workspace packages
node_modules - contains ALL modules for the workspace that are not specific to a single package (module hoisting)
packages/
    package1/
        {other package files including src}
        package.json - a monorepo package that we are deploying to Netlify
        yarn.lock - a fake empty lockfile so Netlify Build uses yarn (not actually used by yarn at all)
        netlify.toml - settings for this deployment
        node_modules - contains symlinks and packages specific to `package1` only (i.e. eslint would be in /node_modules)

We need to set our base directory to packages/package1 so that the netlify.toml is read correctly. However doing this means /node_modules is NOT cached, only /packages/package1/node_modules which is practically empty. This means every build is adding ~120 sec to re-download the root modules from npm.

Describe the solution you'd like

• Detect yarn workspaces in use (presence of workspaces key in some ancestor package.json would indicate this)
• Automatically use yarn instead of npm without needing the fake yarn.lock in the package folder (nice to have)
• Cache both the workspace and local node_modules folders when a workspace is being used

A more manual solution might be to allow specifying paths to cache explicitly, or setting an environment variable.

Describe alternatives you've considered

Build plugins let me cache arbitrary things but unfortunately:

• They currently run after dependencies have been installed, so they are too late to expand the explicitly cached modules before buildbot has already done a full install
• Build plugin cache explicitly refuses to cache dotfiles (hidden files), and the yarn cache is corrupt without these so even with a build plugin yarn is unable to successfully restore cache

Can you submit a pull request?

Possibly with some suggestions as to desired approach.

[ehmicky]

Hi @kamsar,

Thanks for filing this issue.
This is a feature we are currently missing.
This has been already reported in #399, #196, #432 and #433. Also related to #465.

[plum-dawg]

Has there been any movement on this issue?

[kaganjd]

Exciting update about Yarn workspaces: we’ve made improvements to how we cache dependencies for these projects. You can try out our new caching by creating an environment variable called NETLIFY_YARN_WORKSPACES and setting it to "true"

We want your feedback before we roll this out more widely- please let us know here or in the forum what your experience is like! You can also read more about this change here: https://answers.netlify.com/t/improved-caching-for-yarn-workspaces/36066

[JGAntunes]

#526 is now running for all of our builds, closing this issue 👍

[qixiaobo]

So sad to see this after 2 hours waste~