Can not follow nft's reason why to include some dependencies

[janpio]

A recent Prisma release lead to multiple people reporting problems with deploying their projects to Vercel, although we have end to end tests of projects that are deployed to Vercel - and those worked just fine with this version of course, or we would not have released it.

We could fortunately pinpoint the commit that introduced the problem (as we publish each commit as a dev version of prisma) and understood that a require.resolve('@prisma/engines') lead to nft treating this whole package as required and included it in the archive.
@prisma/engines usually is only needed by Prisma CLI prisma, and not by @prisma/client which should be included in the serverless function. As the Engines package is really big, that was very bad. We could fix this problem by replacing this line with eval("require.resolve('@prisma/engines')") fortunately as nft does not "follow" this.

But that did not explain why this only happened in some user projects, and not our end to end test projects 😕

So we looked at the user's project and finally noticed a pattern: They all use Nextjs with Nexus and Nexus Plugin Prisma. Our end to end tests do not. Based on that information we created a minimal reproduction repository that fails to deploy to Vercel because of the archive size: https://github.com/janpio/prisma-limit-repro

This is what I see when we try to deploy that repo:

00:37:45.626  	Warning: Max serverless function size of 50 MB compressed or 250 MB uncompressed hit
00:37:45.627  	Serverless Function's page: api/graphql.js
00:37:45.631  	Large Dependencies                  Uncompressed size  Compressed size
00:37:45.631  	node_modules/@prisma/engines                   113 MB          39.6 MB
00:37:45.631  	node_modules/.prisma/client                   45.3 MB          16.1 MB
00:37:45.631  	node_modules/prisma/build                     5.27 MB          1.09 MB
00:37:45.631  	node_modules/@prisma/client                   3.93 MB           823 kB
00:37:45.631  	node_modules/busboy/deps                       618 kB           196 kB
00:37:45.631  	node_modules/encoding/node_modules             329 kB           179 kB
00:37:45.632  	node_modules/iconv-lite/encodings              303 kB           172 kB
00:37:45.632  	node_modules/lodash/lodash.js                  544 kB          96.4 kB
00:37:45.632  	All dependencies                               172 MB          59.1 MB
00:37:45.632  	Max serverless function size was exceeded for 1 function
00:37:45.752  	Created all serverless functions in: 37.102s

After checking out the reproduction repository, and running npm install, npm run generate and npm run build, finally npm install -D @vercel/nft, I can output the list of files that nft picks with npx nft print ./.next/server/pages/api/graphql.js > nft-print.log - and indeed the whole node_modules/@prisma/engines is included:

...
node_modules/@prisma/engines/dist/download.d.ts
node_modules/@prisma/engines/dist/download.js
node_modules/@prisma/engines/dist/download.js.map
node_modules/@prisma/engines/dist/index.d.ts
node_modules/@prisma/engines/dist/index.js
node_modules/@prisma/engines/dist/index.js.map
node_modules/@prisma/engines/download/index.js
node_modules/@prisma/engines/download/index.js.map
node_modules/@prisma/engines/introspection-engine-debian-openssl-1.1.x
node_modules/@prisma/engines/migration-engine-debian-openssl-1.1.x
node_modules/@prisma/engines/package.json
node_modules/@prisma/engines/prisma-fmt-debian-openssl-1.1.x
node_modules/@prisma/engines/query-engine-debian-openssl-1.1.x
...

I then used this small script look into why one of those binaries in @prisma/engines are included:

const { nodeFileTrace } = require("@vercel/nft");
(async () => {
  const files = ["./.next/server/pages/api/graphql.js"];
  const { fileList, reasons } = await nodeFileTrace(files);
  //console.log(fileList)

  async function traceReasons(path) {
    let reason = reasons[path];
    while (reason) {
      console.log(reason);
      if (reason.parents.length > 1) {
        console.warn("More than one parent, choosing first one");
      }
      reason = reasons[reason.parents[0]];
    }
  }
  console.log("----------");
  traceReasons("node_modules/@prisma/engines/query-engine-debian-openssl-1.1.x");
})();

This outputs something similar to:

codespace ➜ /workspaces/prisma-limit-repro (master ✗) $ node nft.js
----------
{
  type: 'asset',
  ignored: false,
  parents: [ 'node_modules/prisma/build/index.js' ]
}
{
  type: 'dependency',
  ignored: false,
  parents: [ 'node_modules/@prisma/client/scripts/postinstall.js' ]
}
{
  type: 'dependency',
  ignored: false,
  parents: [ 'node_modules/nexus-plugin-prisma/dist/builder.js' ]
}
{
  type: 'dependency',
  ignored: false,
  parents: [ 'node_modules/nexus-plugin-prisma/dist/plugin.js' ]
}
{
  type: 'dependency',
  ignored: false,
  parents: [ 'node_modules/nexus-plugin-prisma/dist/index.js' ]
}
{
  type: 'dependency',
  ignored: false,
  parents: [ '.next/server/pages/api/graphql.js' ]
}
{ type: 'initial', ignored: false, parents: [] }

Now we already know that node_modules/prisma/build/index.js is related to our bug - there we had the misguided require.resolve('@prisma/engines'). But I could not figure out how we go from node_modules/nexus-plugin-prisma/dist/builder.js to node_modules/@prisma/client/scripts/postinstall.js and then node_modules/prisma/build/index.js in the first place.

nexus-plugin-prisma does not include/require the postinstall script of @prisma/client in any way I could find, and the Postinstall script does not require the CLI build in a way I could follow.

Can you please help me understand and see what I am failing to see here?

PS: I committed the repo state where I did the investigation (including .next and node_modules) at janpio/prisma-limit-repro#10 / https://github.com/janpio/prisma-limit-repro/tree/nft-investigation as well.

[janpio]

Ah, just remembered that we had a theory how it gets from Nexus Plugin Prisma to the postinstall hook, but we could not confirm that:

https://github.com/graphql-nexus/nexus-plugin-prisma/blob/fce0ffb5270007124444a36365691bf085538095/src/index.ts#L62
https://github.com/graphql-nexus/nexus-plugin-prisma/blob/fce0ffb5270007124444a36365691bf085538095/src/index.ts#L25

This reads the package.json file of @prisma/client (to get the version...) where the postinstall hook of course is defined and mentioned. But removing those lines did not remove the picking up of that connection by nft.

[guybedford]

Great write up, you've followed exactly the right steps here it seems @janpio.

I think the reason for this is the logic in https://github.com/graphql-nexus/nexus-plugin-prisma/blob/fce0ffb5270007124444a36365691bf085538095/src/builder.ts#L191 is locating asset references to node_modules/@prisma/client which we then can't trace further so can't know if there might be code directly loading something within that path. As a result we then glob and analyze the entire @prisma/client folder, causing the inclusion of @prisma/client/scripts/postinstall.js.

To fix this in nexus-plugin-prisma, you could similarly use eval to hide the @prisma/client path, but I think your original fix seems fine here too.

[songkeys]

Can we have a built-in option to run a script like @janpio did to output the reasons into some readable format so that we can figure out what the root of an introduced file is at a glance?

for example:

{
  "reasons": {
    "entry.js": {
      "node_modules/x/index.js": {
        "node_modules/x/util.js": { /* ... */ }
        "node_modules/x/app.js": { /* ... */ }
        // ...
      },
      "node_modules/y/index.js": { /* ... */ }
      // ...
    }
  }
}

[janpio]

@guybedford Thanks for the analysis, I will try to confirm that experimentally. Really want to understand this one.

This also gave me an idea: Would it be possible for nft to track the line number that caused it to include something in the tree vs. just the file/parent? Then it would be trivial to look at the correct code.

[janpio]

Thanks again @guybedford and @styfle, I could confirm that the line you pointed out indeed triggered this behavior. Wrapping it in eval made the problem go away:

prisma/prisma#6932 (comment)
https://github.com/graphql-nexus/nexus-plugin-prisma/pull/1089/files
janpio/prisma-limit-repro#11

<3

[kamilogorek]

Thanks @janpio, your investigation helped us a lot.

In case someone wants to use the original script now, here's version translated for Map/Set that @vercel/nft uses now:

const { nodeFileTrace } = require("@vercel/nft");
(async () => {
  const files = ["./.next/server/pages/api/test1.js"];
  const { fileList, reasons } = await nodeFileTrace(files);

  async function traceReasons(path) {
    let reason = reasons.get(path);
    while (reason) {
      console.log(reason);
      const [parent] = reason.parents;
      if (reason.parents.size > 1) {
        console.warn("More than one parent, choosing first one");
      }
      reason = reasons.get(parent);
    }
  }
  console.log("----------");
  traceReasons("node_modules/@sentry/cli/sentry-cli");
})();

[styfle]

This feature was added in #282 using nft why [input] [fileinquestion]