[Bug?]: Yarn 3.2 breaks node_modules with docker volumes

[basicdays]

Self-service

• I'd be willing to implement a fix

Describe the bug

Yarn 3.1 works as expected. Yarn 3.2 will fail on yarn install with the following error:

➤ YN0000: ┌ Link step
➤ YN0001: │ Error: EBUSY: resource busy or locked, rmdir '/workspace/node_modules'
➤ YN0000: └ Completed in 4s 519ms

This occurs in the following environment:

• Yarn 3.2
• Using yarn with nodeLinker: node-modules
• Running within a Docker container
• Using a Docker volume for the node_modules folder. This is used in my case with VSCode's devcontainer feature, but it can happen with just using Docker directly.

To reproduce

If necessary, I can create a tiny git repo that has Docker and yarn setup to reproduce this. Otherwise just creating a Docker container with a volume at node_modules and running any kind of yarn install. The host machine will require Docker and Docker Compose for it to run.

Environment

System:
  OS: Linux 5.10 Debian GNU/Linux 10 (buster) 10 (buster)
  CPU: (8) x64 Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
Binaries:
  Node: 12.22.7 - /tmp/xfs-560392c2/node
  Yarn: 3.2.0 - /tmp/xfs-560392c2/yarn
  npm: 6.14.15 - /usr/local/bin/npm

Additional context

Attempting to delete a folder within a Docker container that is managed by a Docker volume will never work. Something within yarn 3.2 switched to attempting to delete the folder completely. I've just used a simple bash script if I ever need to clear out the node_modules directory for whatever reason:

#!/usr/bin/env bash
shopt -s dotglob nullglob
rm -rf "node_modules/*"

[merceyz]

Possibly a duplicate of #4172, can you try with the latest canary release?

yarn set version canary

[coopermaruyama]

I am experiencing the same issue. Using the canary version results in a different error, even with sudo:

➤ YN0000: ┌ Link step
➤ YN0001: │ Error: EPERM: operation not permitted, open '/app/.yarn/install-state.gz'
➤ YN0000: └ Completed in 3s 401ms
➤ YN0000: Failed with errors in 1m 1s

[coopermaruyama]

Got it working. It works fine, my issue was that the .yarn/install-state.gz file was being created on the host machine and then synced into the container.

If the file is created by the container instance it works fine.

If you need to be able to run yarn install on both the host and container, make sure this file does not get synced with the following (if using docker-compose):

volumes:
  - "/app/.yarn"
  - $PWD/.yarn/plugins:/app/.yarn/plugins
  - $PWD/.yarn/releases:/app/.yarn/releases
  - $PWD/.yarn/patches:/app/.yarn/patches
  - $PWD/.yarn/sdks:/app/.yarn/sdks
  - $PWD/.yarn/versions:/app/.yarn/versions

This excludes .yarn from being synced, and then includes the directories which you probably do want to sync.

[basicdays]

Possibly a duplicate of #4172, can you try with the latest canary release?

yarn set version canary

@merceyz Looks like canary dropped support for node 12. We're in process of updating node versions, but haven't completed that yet so I'm unable to test this quick. I'll see if I can find a way to do that later today though.

[basicdays]

@merceyz Looks like the canary successfully did a full install in a workspaces yarn project. Each workspace as well as the root workspace uses a Docker volume on the node_modules directories, and they all worked. I noticed that canary is v4-rc.2. Is a fix going to get backported to v3 at all, or is that unlikely to happen?

[basicdays]

Since this seems to be fixed in an upcoming version of yarn, I'm just going to close this.

[modbender]

Setting yarn version to canary doesn't work everytime.

I still get errors like this:

➤ YN0001: │ Error: While persisting /home/user/webproject/frontend/.yarn/cache/next-npm-13.0.7-ad4d095ec2-de001a674c.zip/node_modules/next/ -> /home/user/webproject/frontend/node_modules/next ENOMEM: not enough memory, open '/home/user/webproject/frontend/node_modules/next/dist/shared/lib/match-remote-pattern.js'

Random package in path each time.