Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Yarn on Linux resolves to "wrong" global package folder #13

Open
byCedric opened this issue May 5, 2019 · 12 comments
Open

Yarn on Linux resolves to "wrong" global package folder #13

byCedric opened this issue May 5, 2019 · 12 comments
Labels
💵 Funded on Issuehunt This issue has been funded on Issuehunt

Comments

@byCedric
Copy link

byCedric commented May 5, 2019

Issuehunt badges

As mentioned in sindresorhus/resolve-global#3. Although it's already mentioned in #12, I'll create this issue as requested.

It looks like Yarn is resolved to a "wrong" global directory on Linux. On my MacOS/OSX this works perfectly though.

OSX:

> npx ava                                                                                          
⠼ { globalDirs:
   { npm:
      { prefix: '/usr/local',
        packages: '/usr/local/lib/node_modules',
        binaries: '/usr/local/bin' },
     yarn:
      { prefix: '/Users/cedric/.config/yarn',
        packages: '/Users/cedric/.config/yarn/global/node_modules',
        binaries: '/Users/cedric/.config/yarn/global/node_modules/.bin' } } }

Linux (through docker run -v $PWD:/code -w /code node:10 npx ava):

root@061d5e7d715e:/code# npx ava
⠧ { globalDirs:
   { npm:
      { prefix: '/usr/local',
        packages: '/usr/local/lib/node_modules',
        binaries: '/usr/local/bin' },
     yarn:
      { prefix: '/usr/local',
        packages: '/usr/local/global/node_modules',
        binaries: '/usr/local/global/node_modules/.bin' } } }

I don't think the /usr/local/global is the proper folder for yarn right? When I manually add something globally through Yarn on the docker/linux instance, I get this path:

root@af923ba739ae:/code# yarn global add dog-names
yarn global v1.13.0
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
warning Your current version of Yarn is out of date. The latest version is "1.15.2", while you're on "1.13.0".
info To upgrade, run the following command:
$ curl --compressed -o- -L https://yarnpkg.com/install.sh | bash
success Installed "dog-names@1.0.2" with binaries:
      - dog-names
Done in 2.47s.
root@af923ba739ae:/code# which dog-names
/usr/local/bin/dog-names
IssueHunt Summary

Sponsors (Total: $50.00)

Become a sponsor now!

Or submit a pull request to get the deposits!

Tips

@vladimyr
Copy link
Contributor

vladimyr commented May 5, 2019

@byCedric I promised to help so I spent some time digging through yarn source and here is what I was able to figure out (in random order):

global package dir

Ofc there isn't a single documentation page inside wonderful yarn docs that lists all possible configuration values so you can only dig through source code in order to figure out that global package dir is determined in following manner:
https://github.com/yarnpkg/yarn/blob/v1.16.0/src/config.js#L346-L349

this.globalFolder = opts.globalFolder || String(this.getOption('global-folder', true));
if (this.globalFolder === 'undefined') {
  this.globalFolder = constants.GLOBAL_MODULE_DIRECTORY;
}

where getOption means aggregation of a lot of things:

  1. cli invocation flag --global-folder
  2. corresponding env variable: $npm_config_global_folder (case insensitive ⚠️)
  3. entry (global-folder) in user's .yarnrc
  4. entry (global-folder) in global <global prefix>/etc/yarnrc

Because nobody really knows that, it will typically default to GLOBAL_MODULE_DIRECTORY defined here:
https://github.com/yarnpkg/yarn/blob/v1.16.0/src/constants.js#L64

export const GLOBAL_MODULE_DIRECTORY = path.join(DATA_DIRECTORY, 'global');

You see this and think: ok that makes sense. I got: /Users/cedric/.config/yarn/global/node_modules as result so (global) DATA_DIRECTORY is constant with value of: <homedir>/.config/yarn/global. Err, not; DATA_DIRECTORY isn't a constant at all, it is platform and environment dependent and gets calculated here:
https://github.com/yarnpkg/yarn/blob/v1.16.0/src/util/user-dirs.js#L9-L24
which roughly translates to (after you deobfuscate it):

const getYarnHomeDir = () => {
    if (isWindows || !process.getuid) {
        return os.homedir();
    }

    const isRootUser = process.getuid() === 0;
    return isRootUser ? path.resolve('/usr/local/share') : os.homedir();
};

const getYarnDataDir = () => {
    if (isWindows) {
        if (process.env.LOCALAPPDATA) {
            return path.join(process.env.LOCALAPPDATA, 'Yarn/Data');
        }

        return path.join(getYarnHomeDir(), '.config/yarn');
    }

    if (process.env.XDG_DATA_HOME) {
        return path.join(process.env.XDG_DATA_HOME, 'yarn');
    }

    return path.join(getYarnHomeDir(), '.config/yarn');
};

Long story short, in order to reliably determine global package dir one needs to:

  1. read yarn's configuration from env and search for global-folder setting
  2. read user's .yarnrc and check there for both --global-folder (which is hardcoded cli flag) and global-folder key
  3. check global <global prefix>/etc/yarnrc for same (meaning you need to calculate <global prefix> too but that's not really yarn's global prefix...)
  4. fallback to lets call it constant fallback global dir typically found in user's homedir or XDG_DATA_HOME if set

Where 2. and 3. are especially funky having in mind that (which is ofc undocumented, why bother documenting it...) yarnpkg/yarn#3320 (comment) yarn uses custom rc format also known as "stringified lockfiles" (I didn't invented that, I swear: yarnpkg/yarn#4134 (comment)) so you can't really pull it through ini parser. In order to fight that insanity I made some hacky wrapper around ini parser:

const readYarnrc = filePath => {
    try {
        const contents = fs.readFileSync(filePath, 'utf8');
        const lines = contents.split(/\r?\n/g);
        const rePrefix = /^(?:--)?prefix\s+/;
        const prefix = lines.reduce((prefix, line) => {
            line = line.trim();
            return rePrefix.test(line) ? line.replace(rePrefix, '') : prefix;
        }, undefined);
        if (prefix) {
            return ini.parse(`prefix=${prefix}`).prefix;
        }
    } catch (_) { }
};

that serves me well so far 🤞

I said earlier that you have to calculate <global prefix> in order to read global yarnrc. Now let's deep dive into that. You'll find code that does that here (together with lovely comment above): https://github.com/yarnpkg/yarn/blob/v1.16.0/src/registries/npm-registry.js#L40-L57
and that can be rewritten as:

const getYarnGlobalPrefix = () => {
    if (process.env.PREFIX) {
        return process.env.PREFIX;
    }

    if (isWindows) {
        return path.dirname(process.execPath);
    }

    const prefix = path.dirname(path.dirname(process.execPath));
    return path.join(process.env.DESTDIR || '', prefix);
};

I said it's not real <global prefix> because it is meant to be used only to locate yarn's installation dir and then you join it with etc/yarnrc and get global config candidate that might contain different global prefix inside and if not different fallback will be used 🥴

Now; going back to current state of art that is present inside global-dirs; you can easily see that there are bunch of changes required to happen before your issue gets (reliably) resolved 😭
I'm still in code typing business and once and if I get something ready in sense that it covers all use cases I'll propose PR.

That's it for now. Stay tuned for next episode of @vladimyr fighting yarn with working title prefix + /bin = binaries 🍿

@byCedric
Copy link
Author

byCedric commented May 6, 2019

Haha wow @vladimyr, you do your research thoroughly and I like it ❤️ Thanks for helping/checking this out! It's insane how Yarn does this right now. I imagine a developer at Yarn, who put the "lovely comment" above, looking at that and thinking "naa, I'm not touching that..." 😅

If there is anything I can help with, spam me and I'll appear!

@IssueHuntBot
Copy link

@byCedric has funded $50.00 to this issue.


@byCedric
Copy link
Author

I personally love it to get this fixed, so I'm throwing in some funds so it can get attention 😄 Hope it's not to much pressure though!

@TiagoDanin
Copy link
Contributor

TiagoDanin commented Jun 9, 2019

I think I found the cause of problem and I believe macOS also has the same problem.

@byCedric What output of yarn global bin (macOS) ?

@byCedric
Copy link
Author

@TiagoDanin I get this 😄

> yarn global bin
/usr/local/bin

@vladimyr
Copy link
Contributor

I personally love it to get this fixed, so I'm throwing in some funds so it can get attention Hope it's not to much pressure though!

I don't do this because of funds but I don't have problem with it either, I don't mind really. If it were up to me I'd have better ideas how to waste few dozen bucks and they mostly revolve around good food and beverage 😁

No pressure made it is just that I don't have enough time lately to tackle this but I'll give my best to finish it this weekend 🤞

@TiagoDanin
Copy link
Contributor

Nice! yarn/src/cli/commands/global.js L120-123 this can help you

@tunnckoCore
Copy link

tunnckoCore commented Nov 9, 2019

Oooh god, what a read and insanity... :D Btw, it's strange to get different results, since both linux and macos does not go to the isWindows() case.

@byCedric, what linux is that on the docker? oh it's the node's image.

That's why I proposed there (yarnpkg/yarn#5806) to add few real ENV variables, so we can get it easily.
No reasponse for year and a half, so some comments to support it would be great.

@vladimyr
Copy link
Contributor

vladimyr commented Nov 9, 2019

It is been a while and I forgot all delicate intricacies of yarn internals but I haven't changed my opinion and feelings towards yarn's way of doing things. That being said hard things will always remain hard. There isn't and should not be any magical shortcut. If you need to figure out yarn's global dirs you need to go through same steps that yarn does.

@tunnckoCore I'm sympathetic to your cause 👍 but I can't give you 👍 for proposed solution because it breaks canonical rules. Environment (variables) control behavior of a program not other way around. We would end up in huge mess if every binary starts to modify environment per its needs. User is and should always be in control of an environment while binaries could adapt to it if needed. Asking for yarn to pollute environment with environment variables that aren't effectively under user's control (at least aren't created by user) is essentially asking for it to behave wildly non-standard and something that will likely never be implemented. If you want to get some info from yarn you need to ask it by invoking it. yarn isn't and does not provide init shell script that you'll invoke as part of your shell profile and that would be the only place where temporary modification (per session) of environment can be considered allowed. What you proposed can be done only if yarn installer starts modifying your shell profile and/or env config and that is big no no due to reasons explained earlier. Correct me if I did not understand you correctly.

@tunnckoCore
Copy link

tunnckoCore commented Nov 10, 2019

Agree. But many programs is doing that while installation - not that many but still. Also, Yarn has standalone installer too, so actually the user will be able to control the env. The thing is that user will provide, for example the global dir, yarn is guaranteed to use it and user is guaranteed to know what is it since he set it.

Yes, this doesn't help in the other cases when you install it through package manager or such, but still one case less to care in such issues like this one.

Also, adding 1 or 2 env variables isn't really that big pollution.

Or another possibility can be to be opt-in (the setting of env vars), through flag or command. Nah, that's stupid.

Anyway, you are correct to some extent.

@vladimyr
Copy link
Contributor

Agree. But many programs is doing that while installation - not that many but still.

For example?

The thing is that user will provide, for example the global dir, yarn is guaranteed to use it and user is guaranteed to know what is it since he set it.

So you are envisioning something like:

$ YARN_GLOBAL_DIR=/path/to/global/dir ./yarn-installer

which is basically reinventing PREFIX environment variable but for this specific case. Whether or not it can be solved using PREFIX users still aren't required to use YARN_GLOBAL_DIR meaning that having in-code fallback is required. Because that constant or calculated value can not be queried you are asking yarn to store it back into environment variable, but how? In order to store something inside environment you need to either change /etc/environment or modify user's shell rc/profile. First you are probably not allowed to touch anyway and second is something that isn't considered good practice. Which brings me to:

Also, adding 1 or 2 env variables isn't really that big pollution.

It is not matter of a quantity it is about how you are planning to do that in the first place. I may be wrong but I believe it is technically impossible to do that in transparent and widely accepted manner. 🤔

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
💵 Funded on Issuehunt This issue has been funded on Issuehunt
Projects
None yet
Development

No branches or pull requests

5 participants