[NEXT-638] Next.js in a monorepo does not include file referenced in code

[millsp]

Verify canary release

• I verified that the issue exists in the latest Next.js canary release

Provide environment information

Operating System:
      Platform: linux
      Arch: x64
      Version: #66~20.04.1-Ubuntu SMP Wed Jan 25 09:41:30 UTC 2023
    Binaries:
      Node: 19.0.1
      npm: 8.19.2
      Yarn: 1.22.17
      pnpm: 7.24.2
    Relevant packages:
      next: 13.1.7-canary.17
      eslint-config-next: N/A
      react: 18.2.0
      react-dom: 18.2.0

Which area(s) of Next.js are affected? (leave empty if unsure)

No response

Link to the code that reproduces this issue

https://github.com/millsp/nextjs-monorepo-file-not-found

To Reproduce

• Clone the provided reproduction repository
• Follow steps in README of each branch
• There are two branches:
  1. A JS-based code branch
     Which shows the described bug and how the workaround fixes it
  2. A TS-based code branch
     Which shows the described bug and how the workaround is broken

Describe the Bug

An unexpected file not found error happens from an imported package that reads from the local file system (using Next.js in a monorepo).

In this monorepo, I have a package called service and another one called file-reader. Here’s the representation of the project:

.
├── packages
│   ├── file-reader
│   │   ├── index.js
│   │   ├── node_modules
│   │   ├── package.json
│   │   └── file.txt
│   └── service
│       ├── next.config.js
│       ├── next-env.d.ts
│       ├── node_modules
│       ├── package.json
│       └── pages
│           └── api
│               └── test.js
├── pnpm-workspace.yaml
└── README.md

service needs to use file-reader, and file-reader needs to read file.txt, but running the code with next build or next dev, and opening the API endpoint, we will then encounter an ENOENT: no such file or directory error:

Server Error
Error: ENOENT: no such file or directory, open '/nextjs-monorepo-notfound/packages/service/.next/server/pages/api/file.txt'

Analysis why this is happening

Since v13.1.7-canary.11 (which included #45864), the result of pnpm exec next build includes the file.txt in the correct location:

.next/standalone
├── node_modules
├── package.json
└── packages
    ├── file-reader
    │   ├── file.txt
    │   ├── index.js
    │   └── package.json
    └── service
        ├── .env
        ├── .next
        │   └── server
        │       ├── pages
        │       │   ├── api 
        │       │   │   ├── test.js // this bundle includes `file-reader`
        │       │   │   └── test.js.nft.json
        ├── package.json
        └── server.js

(Before v13.1.7-canary.11 that was not the case and you had to use outputFileTracingRoot: "../../" manually to make it work - thanks @ijjk for that timely fix!).

So why is the file still not found when executing the code, although it is included directly?

Our current understanding is that because there is a bundling step, file-reader will get bundled as the API endpoint (or as a chunk). However, the files this bundle is looking for (here file.txt) are not present right next to it. Remember, file-reader wants to open a file called file.txt and expects it to be near its index.js.

So for this to work, we would need to have a tree like this instead:

.next/standalone
├── node_modules
├── package.json
└── packages
    ├── file-reader
    │   ├── file.txt
    │   ├── index.js
    │   └── package.json
    └── service
        ├── .env
        ├── .next
        │   └── server
        │       ├── pages
        │       │   ├── api
+       │       │   │   ├── file.txt
        │       │   │   ├── test.js // this bundle includes `file-reader`
        │       │   │   └── test.js.nft.json
        ├── package.json
        └── server.js

Our attempt at finding a workaround

We tried marking file-reader as external:

webpack: (config, { buildId, dev, isServer, defaultLoaders, webpack }) => {
   config.externals = [...config.externals, 'file-reader']
   return config
},

This works perfectly when your codebase is JS-only, but breaks completely as soon as you use TypeScript. Because marking as external skips the Next.js transpilation step, then we end up with new errors instead:

Server Error
SyntaxError: Cannot use import statement outside a module

This error happened while generating the page. Any console logs will be displayed in the terminal window.

/nextjs-monorepo-notfound/packages/file-reader/index.ts:1
import path from 'path'
^^^^^^

SyntaxError: Cannot use import statement outside a module
    at Object.compileFunction (node:vm:360:18)

(We have also tried transpilePackages in this context, but that didn’t have any effect)

This shows how bundling is causing these path issues.

Expected Behavior

Next.js should include the files so they are found at runtime, and the user does not get an error.

In other words, Next.js should be able to understand how to copy the correct files near the generated bundles and include these in their respective nft.json files for a valid standalone build.

Why does Prisma care

In short, this also affects Prisma Client. Instead of the file-reader in the reproduction, this would be a generated Prisma Client. It includes a schema.prisma and a Query Engine file that live next to its code. When you run your app, the schema.prisma is not found. We have many issues about this:

prisma/prisma#12853
prisma/prisma#12921
prisma/prisma#12588
prisma/prisma#13233
prisma/prisma#12823
prisma/prisma#14566
prisma/prisma#17687

And these issues are full with related, but slightly different, reports of the same problem.

After some research, we figured out that what they all shared was that they are using Next.js in a monorepo context — and then could understand and reproduce the problem without Prisma as well. That became our reproduction above.

For the Prisma Client to work well in a Next.js app in a monorepo context, we need to find a solution for that problem, and ideally not a workaround.

Which browser are you using? (if relevant)

No response

How are you deploying your application? (if relevant)

No response

_NEXT-638

[ijjk]

Hi, after taking a look into the provided reproduction this doesn't exactly look like a bug with the file tracing, will try to clear up some confusion.

In the provided js branch the file-reader is being transpiled and bundled into the server bundle which causes __dirname to now reference the .next/server/api folder instead of the original packages/file-reader folder.

If the package is meant to be bundled/transpiled by webpack and shouldn't be external it can instead use the new URL() handling which copies the file automatically e.g.

const fs = require('fs')

function getFileContents() {
  const fileUrl = new URL('./file.txt', import.meta.url)
  return fs.readFileSync(fileUrl.pathname, 'utf8')
}

module.exports = { getFileContents }

When the externals modification in next.config.js is used it prevents it from being bundled and the __dirname is the correct value and the file tracing works as expected. For a published package this should work properly as long as the file being referenced is present on the disk before the webpack build starts.

In the provided ts branch the transpilePackages config conflicts with the externals workaround as the tranpsilePackages config requires bundling/transpiling the package meanwhile the externals prevents that. Ideally any packages are not requiring the TypeScript from another directly and instead file-reader is transpiled to JS and then required and then can be treated as external.

If the above context doesn't provide enough information to resolve this upstream I'd be happy to provide more examples/specifics to alleviate this.

[basememara]

Does this make sense to the Prisma team? The suggestion to use new URL() must be done in the Prisma binary to resolve this without workarounds?

[janpio]

(We were on a company offsite last week, catching up right now and will definitely take a look at @ijjk's suggestion. The changes would need to be applied in Prisma Client.)

[millsp]

Hey @ijjk thanks for your answer. I can confirm that changing path.join(__dirname, 'file.txt') to new URL('./file.txt', import.meta.url) causes the file to be properly copied in a Next.js project. I was happy to see it even handles possible file collisions by renaming the copied files.

However, when you are not in a Next.js project, I found that it didn't work. Since Prisma is a library, it will also need to work with plain Node.js (or other runtimes), outside of Next.js. Since import.meta.url is an ESM construct, attempting to use it in a CJS script will create an error at parsing time:

$ node file-reader/index.js

/home/millsp/Work/repros/nextjs-monorepo-notfound/packages/file-reader/index.js:4
const filePath = new URL('./file.txt', import.meta.url)
                                              ^^^^

SyntaxError: Cannot use 'import.meta' outside a module
    at Object.compileFunction (node:vm:360:18)
    at wrapSafe (node:internal/modules/cjs/loader:1088:15)

What I understood so far is that Next.js is able to understand import.meta.url even in CJS scripts and will transform it via webpack. With that transformation, it copies the files, updates nft.json files, and also completely removes import.meta.url to transform it to a webpack injected variable. Since this does not happen in simple Node.js projects, we have that parsing error.

Do you know about an alternative that would work for both CJS, ESM and Node.js, Next.js projects?

[basememara]

Just wondering if this a Next.js problem at this point, or still related? It sounds like by changing to URL will fix Next.js and probably other stacks, but breaks plain Node.js projects. Currently it's working the other way around at the moment (works for Node.js projects but broken for other stacks). Would it make sense to do something like isNode ? __dirname : URL?

[millsp]

@basememara import.meta.url unfortunately produces a syntax error on CJS modules, so a runtime check does not help.

[basememara]

Thx @millsp, I have another idea but sketchy and wanted to share in case there's something there. How about having ESM and CJS variants for the file-reader package. In package.json:

{
  ...
  "scripts": {
    "build": "npx tsc --build",
    "build:cjs": "npx tsc --project tsconfig-cjs.json"
  }
  ...
}

In tsconfig-cjs.json:

{
  "extends": "./tsconfig.json",
  "compilerOptions": {
    "module": "commonjs"
  }
}

This way we can inject values at compile time?