Yarn requires yarn release .cjs file to be committed to git repo

[ildella]

Self-service

• I'd be willing to implement a fix

Describe the bug

When I upgrade a project to use yarn canary, it downloads the yarn release file in a local project folder, and writes the path in the .yarnrc.yml. The upgrade to yarn v4 is then successful.

I do not push the released file to git as I consider a very bad practice to pollute all my repositories with dependencies file of any kind.

Then when someone else tries to run yarn, it fails as any yarn command will look for the local file - which does not exists on git.

It should be downloading it if there's no local copy.

To reproduce

1. upgrade a yarn 1.x project to yarn canary following the step by step guide: https://yarnpkg.com/migration/guide
2. gitignore the whole .yarn/* folder
3. now everything works, commit and push
4. someone else - maybe the github workflow - checks out the project and try to run yarn - really any command even --version:

node:internal/modules/cjs/loader:1051
  throw err;
  ^

Error: Cannot find module '/{path-to-project}/.yarn/releases/yarn-4.0.0-rc.51.cjs'
    at Module._resolveFilename (node:internal/modules/cjs/loader:1048:15)
    at Module._load (node:internal/modules/cjs/loader:901:27)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:83:12)
    at node:internal/main/run_main_module:23:47 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

Node.js v20.7.0

Environment

System:
    OS: Linux 6.2 Ubuntu 23.04 23.04 (Lunar Lobster)
    CPU: (8) x64 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz
  Binaries:
    Node: 20.7.0 - /tmp/xfs-a472fb66/node
    Yarn: 4.0.0-rc.51 - /tmp/xfs-a472fb66/yarn
    npm: 10.1.0 - ~/n/bin/npm
  npmPackages:
    jest: ^29.7.0 => 29.7.0

[ildella]

Possible solutions:

1. yarn checks yarnPath file to exist, and instead of failing just download it.
2. there is a yarn command to download the desired release in a systm folder, like $HOME/.yarn/releases...

Second option would be great, but we must be sure it can work on CI, like github workflows.

[ildella]

ok actually the explanation of this whole thing is here:

https://yarnpkg.com/configuration/yarnrc

In the last line of the config file 😦

So the solution is:

1. uninstall and remove all old versions of yarn, installed either with npm install -g yarn or with https://yarnpkg.com/install.sh
2. install with corepack as explained here -> https://nodejs.org/api/corepack.html which means corepack prepare yarn@stable --activate
3. now it won't be necessary for yarn to write the release file path in yarnrc.yml it actually won't even download it as it's been instlled globally by corepack...

In the end the problem is that the step by step guide does not say to eradicate old versions of yarn for this to work properly.
https://yarnpkg.com/migration/guide

[polarathene]

Yes the docs need to be updated. I think many others agree with you that committing versions of yarn and the cache (with good intent of reproducible builds) is not good for the git repo.

At the very least git LFS would be more appropriate to keep the core repo history light-weight. It's less of an issue if you want to archive the project, but I don't see the appeal otherwise as git is not really meant to manage binaries like this (hence git lfs, but that's additional setup).

---

Your solution to use corepack prepare will work for a global yarn on your system.

However for other devs they should have enabled corepack (corepack enable) and your package.json should have a packageManager field that references what version of yarn to use yarn@3.6.3 for example (stable is not permitted here) and whenever your project is used, corepack will ensure it's run with the appropriate release of yarn for that project.

Related: #5744 (comment)

[ildella]

Clear.

The problem I see is that the guide are:

1. It explain how to migrate a single project, rather than upgrading yarn for the system. That's it should be made more clear.
2. Following that guide, user will end up either committing the /release file to git or having failures. Both scenario are unlikely to be welcome.

This is what a guide to install yarn 2+ at system level could be:

    Make sure you're using Node 18+
    Uninstall all existing versions of yarn from your system.
    Run `corepack enable`. See [Corepack](https://nodejs.org/api/corepack.html) for reference. 
    Run `corepack prepare yarn@stable --activate`  to install the desired version. (link to available versions)
    
Now each project must (can?) set which version of yarn to use adding a line like this to package.json:

"packageManager": "yarn@4.0.0-rc.50"

or

"packageManager": "yarn@3.6.3".

At this point in an individual project just run `yarn install` and the project will be upgraded. 
A new file `.yarnrc.yml` will be generated at this stage, and should be published to the version control system.

That will allow user to keep old 1.x project as they are - I believe - and smoothly transition just the ones they wish.

[arcanis]

It explain how to migrate a single project, rather than upgrading yarn for the system. That's it should be made more clear.

That's because we don't want you to install Yarn at the system level. We want it to be managed per-project, like any other dependency.

[ildella]

But yarn is not like any other dependency. And installing it that way force users to have yarn actual bundle in their VCS, which is bad.

Other dependencies works fine in a folder which is never on VCS.

What are the motivations behind this choice?

[koshic]

But yarn is not like any other dependency.

Why? Every project (scripts, .yarnrc, yarn.lock, etc.) depends on the particular yarn version. Keeping yarn in VCS eliminates all 'global installation' risks, and don't force developers to dance around upgrade / downgrade when switching projects. We already have such experience with Node - 100% backward compatibility is impossible on the long run, and we have to use tools like nvm to choose appropriate version.

Even if sometimes yarn way is a bit... strange / uncommon, ability to 'lock' package manager is a very important feature. Sure, corepack can provide more or less similar experience, but it still requires manual activation and has an inconvenient offline mode.

[arcanis]

And installing it that way force users to have yarn actual bundle in their VCS, which is bad.

You say it's bad, I say "I've been doing it for the past seven years and both git and my coworkers are perfectly happy with it". Yarn is 2MB, there's absolutely no problem whatsoever keeping it in the VCS - even perfwise, it has no practical impact on its own (if you check-in the .yarn/cache folder as well it's another story, but that's orthogonal and entirely optional).

With that being said, you're not the first one raising such concerns, so from the 4.0 forward we are moving towards making Corepack the preferred way to install Yarn - but that's still on a per-project basis (ie with the packageManager field rather than by using the --activate flag), for the reasons described by @koshic.

So once we release the 4.0 stable (which should happen soon), yarn init -2 will set this field, but won't add the Yarn binary inside the newly created project anymore.

[polarathene]

we don't want you to install Yarn at the system level. We want it to be managed per-project, like any other dependency.
from the 4.0 forward we are moving towards making Corepack the preferred way to install Yarn - but that's still on a per-project basis

I'm fully in support of packageManager in package.json, but I don't know why you should hide how to use yarn outside of a project (eg for yarn dlx).

I think it's worth highlighting corepack prepare yarn@stable --activate, while making the packageManager approach the endorsed approach for projects.

I say "I've been doing it for the past seven years and both git and my coworkers are perfectly happy with it".
Yarn is 2MB, there's absolutely no problem whatsoever keeping it in the VCS - even perfwise, it has no practical impact on its own

• If you commit all 26 releases on Github since Aug 2019, and we average those at 2MB each, that's 52MB, for a 2MB dependency?
• If you also were doing that with the v4 RC releases, that's over 50, another 100MB.

Both aren't that realistic, but shows over time how that can build up depending how often you'd bump that dependency in the project.

You could just as easily keep an archive of those releases out of the project or use git LFS. In your projects it may not be an issue but it's generally wasteful lumping in such data into your git history.

There's plenty of tooling out there that can get the right version and automate that for you too if you need that (as mentioned with nvm as an example). Why not also commit Node and everything else for a full reproducible build as these can be important as time passes too 🤷‍♂️

(if you check-in the .yarn/cache folder as well it's another story, but that's orthogonal and entirely optional).

This gets a pass because of how much more excessive it is in size? It's the same problem, you just experience it sooner.

[arcanis]

I'm fully in support of packageManager in package.json, but I don't know why you should hide how to use yarn outside of a project (eg for yarn dlx).

Corepack has a "transparent" mechanism - some commands (currently only dlx) are run against Yarn 2+, even if 1.x is the global binary. It's not perfect, but it can be improved!

I think it's worth highlighting corepack prepare yarn@stable --activate, while making the packageManager approach the endorsed approach for projects.

It's fine to add it to the documentation - but we should make it clear that this is discouraged in general because [reasons described; basically a small FAQ answering the points from this thread]. Then people decide on their own (clearly, the current wording isn't clear enough).

Both aren't that realistic, but shows over time how that can build up depending how often you'd bump that dependency in the project.

(I doubt we'll convince each other, so I don't try to address the "is it good or bad" aspect, just the practicality of it)

Exactly, they aren't realistic - in practice I wouldn't expect you to upgrade Yarn that often. I checked on our work repository, in the past 4y we upgraded it 41 times - and I'm a Yarn maintainer, so sometimes I update in-between releases!

Using LFS for that would have been adding a significant hurdle (everyone working on the repo needs a plugin, for starter, and hosting isn't always free) to solve what's essentially a non-problem.

There's plenty of tooling out there that can get the right version and automate that for you too if you need that (as mentioned with nvm as an example)

And Corepack! Which I actually designed, advocated for, and implemented. I agree that not having to check-in the binary is the best situation - but it wasn't possible until recently, and in the meantime checking-in the binary was actually reasonable.

[ildella]

So once we release the 4.0 stable (which should happen soon), yarn init -2 will set this field, but won't add the Yarn binary inside the newly created project anymore.

So let me recap:

1. there will be a fix before final v4 is released so that the per-project approach - with no corepack prepare - will NOT require any binary in the VCS
2. The package.json entry will guarantee that regardless one has yarn installed globally or not, the proper version will run a specific project.
3. the reccomended way will be per-project. With point number 1 in place is actually ok as it won't have potentially undesirable side effects later.

I think it's a win :)

[arcanis]

there will be a fix before final v4 is released so that the per-project approach - with no corepack prepare - will NOT require any binary in the VCS

It already doesn't, you can set packageManager in your package.json, remove the yarnPath setting (and file), and it'll work just fine as long as you have Corepack enabled. It's just that Yarn 4.0+ will do that automatically on yarn init -2.

[Rikaelus]

It already doesn't, you can set packageManager in your package.json, remove the yarnPath setting (and file), and it'll work just fine as long as you have Corepack enabled. It's just that Yarn 4.0+ will do that automatically on yarn init -2.

This is a slight tangent and different situation, so if this is better it is own issue then just say so, but there's a lot of good context in this thread that makes for an excellent jumping-off point.

We're attempting to update a Docker-based project from having one of its services use Yarn 4 (upgrading from 3) and I'm really running afoul of the new Corepack requirements. Unlike @ildella we've never had to store the .cjs file in the repository but we do have tools in place that install it (and all other dependencies) when preparing the environment. That same tool also includes checks to make sure everything looks configured. The tool, for example, tests for the presence of .yarn/releases/yarn-<ver>.cjs and, if it's not present it automatically begins configuring it.

The first issue we had was this:
HOME=/tmp/.yarn-global yarn set version 4.1.1 =>

error This project's package.json defines "packageManager": "yarn@4.1.1". However the current global version of Yarn is 1.22.22.

Presence of the "packageManager" field indicates that the project is meant to be used with Corepack, a tool included by default with all official Node.js distributions starting from 16.9 and 14.19.
Corepack must currently be enabled by running corepack enable in your terminal. For more information, check out https://yarnpkg.com/corepack. 
Failed

I got by that by changing the command to HOME=/tmp/.yarn-global corepack yarn set version 4.1.1 but... I have no idea where Corepack is putting the .cjs file for our tool to see if it already exists. And I really need our project operating on files located only within the project directory structure for Docker voluming purposes. Can this be achieved? Is there any way to simply remove Corepack from the picture? From what I can tell it'll buy us nothing in a Dockerized environment but looks like it could cause us a good chunk of grief.