Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(tooling): add script for getting nested dependency data #9734

Merged
merged 6 commits into from
Dec 22, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions tasks/nmHoisting/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# Nested `node_modules`

When packages request different versions of the same dependency, Yarn tries to
make both packages happy by installing two copies of that dependency. With a
little bit of hand-waving here, for the `node-modules` linker, since dependencies
have to exist at a single place on disk, Yarn has to decide which one gets
hoisted to the top level (i.e. `node_modules/dependency`) and which one has to
be nested
(`node_modules/package-requesting-different-version/node_modules/dependency`).
As far as I can tell, it does this in a more or less sensible way by hoisting
the version that's requested the most.

But when our own `@redwoodjs` packages become nested, it's usually a (painful)
problem for us and for our users. This script identifies which of CRWA's
dependencies have a lot of nested dependencies:

```bash
yarn node ./tasks/nmHoisting/nmHoisting.mjs
```

Here's a snippet of the nested dependencies for the `@redwoodjs` packages as of
version `v6.5.1`:

```json5
{
"hoistedNodeModules": {
// ...
"@redwoodjs/internal": {
"source-map": "0.7.4"
},
"@redwoodjs/cli": {
"decamelize": "5.0.1"
},
"@redwoodjs/testing": {
"@types/node": "18.18.9"
},
"@redwoodjs/api": {
"@whatwg-node/fetch": "0.9.14"
},
"@redwoodjs/telemetry": {
"@whatwg-node/fetch": "0.9.14"
},
"@redwoodjs/prerender": {
"@whatwg-node/fetch": "0.9.14"
},
"@redwoodjs/structure": {
"lru-cache": "7.18.3"
},
"@redwoodjs/graphql-server": {
"@graphql-tools/utils": "10.0.11",
"@graphql-tools/utils/cjs": "null",
"@graphql-tools/schema": "10.0.2",
"@graphql-tools/schema/cjs": "null"
},
// ...
}
}
```

You can also see a visualization by opening [nmHoistingVisualize.html](./nmHoistingVisualize.html):
89 changes: 89 additions & 0 deletions tasks/nmHoisting/nmHoisting.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
#!/usr/bin/env node
/* eslint-env node */
// @ts-check

import { cd, fs, os, path, within, $ } from 'zx'

async function main() {
const TMP_DIR = os.tmpdir()
const TIMESTAMP = (await $`date +%Y%m%d_%H%M%S`).stdout.trim()
let CRWA_DIR = path.join(`crwa_${TIMESTAMP}`)

const projectProvided = !!process.argv[2]

if (projectProvided) {
CRWA_DIR = process.argv[2]
}

let data = {
version: '',
node_modules: [],
}

await within(async () => {
cd(TMP_DIR)

if (!projectProvided) {
await $`yarn create redwood-app ${CRWA_DIR} -y`
}

cd(CRWA_DIR)

data.version = (
await $`jq -r '.devDependencies."@redwoodjs/core"' < package.json`
).stdout.trim()

let stdout = (
await $`find ./node_modules -mindepth 2 -maxdepth 3 -type d -name node_modules`
).stdout
.trim()
.split('\n')

data.node_modules = await batchLines(stdout)
})

const dataS = `\

rawData.push(\`${JSON.stringify(data, null, 2)}\`)
`

await fs.appendFile(new URL(`nmHoistingData.js`, import.meta.url), dataS)
}

main()

// ------------------------
// Helpers
// ------------------------

function batchLines(lines) {
return lines.reduce(async (objP, line) => {
const obj = await objP

let depLines = (await $`find ${line} -name 'package.json'`).stdout
.trim()
.split('\n')

const name = line.match(
/^\.\/node_modules\/(?<package>.+)\/node_modules$/
)[1]
obj[name] = await batchDepLines(depLines)

return obj
}, Promise.resolve({}))
}

function batchDepLines(depLines) {
return depLines.reduce(async (objP, depLine) => {
const obj = await objP

const version = (await $`cat ${depLine} | jq -r .version`).stdout.trim()

const name = depLine.match(
/[^\.]+\/node_modules\/(?<package>.+)\/package.json$/
)[1]
obj[name] = version

return obj
}, Promise.resolve({}))
}
Loading
Loading