nx-distrbuted-task on ubuntu-4-cores Cannot find module project-graph-worker.js

[StefanLobbenmeierObjego]

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[x] Bug report  
[ ] Performance issue
[ ] Feature request
[ ] Documentation issue or request
[ ] Support request
[ ] Other... Please describe:

Current behavior

When starting on ubuntu-latest-4-cores, the nx-distributed-task step fails with this error:

Cannot find module '/home/runner/work/_actions/e-square-io/nx-distributed-task/v2/project-graph-worker.js'

Expected behavior

No error, just like on ubuntu-latest

Minimal reproduction of the problem with instructions

1. Create a github action runner with 4 cores called ubuntu-4-cores
2. Run e-square-io/nx-distributed-task@v2 on this runner

What is the motivation / use case for changing the behavior?

The nx project I use has one big project and many small ones. I would like to run the big project on a bigger runner to speed up CI. So I use 4 cores for that one, but unfortunately this seems to be incompatible with this project

Environment

- Nx version: the issue is in the bunled nx version, which is 13.10.6, the project uses 16.9.1
- Node version: 18.18.0
- Platform:  ubuntu

Others:

Logs

Run e-square-io/nx-distributed-task@v2
  with:
    target: test
    projects: objego-client
    uploadOutputs: true
    nxCloud: false
    maxParallel: 3
    debug: false
  env:
    PNPM_HOME: /home/runner/setup-pnpm/node_modules/.bin
Run actions/github-script@v6
  with:
    script: const { default: setEnv } = require('/home/runner/work/_actions/e-square-io/nx-distributed-task/v[2](https://github.com/objego/objego-client/actions/runs/9854372667/job/27206990967?pr=1372#step:7:2)/set-env.js');
  const { default: main } = require('/home/runner/work/_actions/e-square-io/nx-distributed-task/v2/main.js');
  
  setEnv({
    "target": "test",
    "projects": "objego-client",
    "uploadOutputs": "true",
    "nxCloud": "false",
    "args": "",
    "distribution": "",
    "bucket": "",
    "maxParallel": "[3](https://github.com/objego/objego-client/actions/runs/9854372667/job/27206990967?pr=1372#step:7:3)",
    "workingDirectory": "",
    "debug": "false"
  }, process);
  await main(context, core, exec, glob, __original_require__);
  
    github-token: ***
    debug: false
    user-agent: actions/github-script
    result-encoding: json
    retries: 0
    retry-exempt-status-codes: [4](https://github.com/objego/objego-client/actions/runs/9854372667/job/27206990967?pr=1372#step:7:4)00,401,403,404,422
  env:
    PNPM_HOME: /home/runner/setup-pnpm/node_modules/.bin

 >  NX   Cannot find module '/home/runner/work/_actions/e-square-io/nx-distributed-task/v2/project-graph-worker.js'

[StefanLobbenmeierObjego]

I am currently checking the nx source code for this:

The worker is created here:

function createWorkerPool(numberOfWorkers) {
    const res = [];
    for (let i = 0; i < numberOfWorkers; ++i) {
        res.push(new (require('worker_threads').Worker)((0, path_1.join)(__dirname, './project-graph-worker.js'), {
            env: process.env,
        }));
    }
    return res;
}

Looking up the stack we see this:

function buildExplicitDependenciesUsingWorkers(jsPluginConfig, ctx, totalNumOfFilesToProcess, builder) {
    const numberOfWorkers = Math.min(totalNumOfFilesToProcess, getNumberOfWorkers());
    const bins = splitFilesIntoBins(ctx, totalNumOfFilesToProcess, numberOfWorkers);
    const workers = createWorkerPool(numberOfWorkers);
    let numberOfExpectedResponses = bins.length;
 	// ...

function buildExplicitDependencies(jsPluginConfig, ctx, builder) {
    let totalNumOfFilesToProcess = totalNumberOfFilesToProcess(ctx);
    // using workers has an overhead, so we only do it when the number of
    // files we need to process is >= 100 and there are more than 2 CPUs
    // to be able to use at least 2 workers (1 worker per CPU and
    // 1 CPU for the main thread)
    if (totalNumOfFilesToProcess < 100 || getNumberOfWorkers() <= 2) {
        return buildExplicitDependenciesWithoutWorkers(jsPluginConfig, ctx, builder);
    }
    else {
        return buildExplicitDependenciesUsingWorkers(jsPluginConfig, ctx, totalNumOfFilesToProcess, builder);
    }
}

it seems when there are 2 or less threads, nx does not use workers, so it never runs into that path.

[StefanLobbenmeierObjego]

Because the file is an external file, it seems webpack is not aware of this. This is the resulting .js after compilation:

                                    const numberOfWorkers = Math.min(totalNumOfFilesToProcess, getNumberOfWorkers()), bins = function(ctx, totalNumOfFilesToProcess, numberOfWorkers) {
                                        const filesPerBin = Math.round(totalNumOfFilesToProcess / numberOfWorkers / 5) + 1, bins = [];
                                        let currentProjectFileMap = {}, currentNumberOfFiles = 0;
                                        for (const source of Object.keys(ctx.filesToProcess)) for (const f of Object.values(ctx.filesToProcess[source])) currentProjectFileMap[source] || (currentProjectFileMap[source] = []), currentProjectFileMap[source].push(f), currentNumberOfFiles++, currentNumberOfFiles >= filesPerBin && (bins.push(currentProjectFileMap), currentProjectFileMap = {}, currentNumberOfFiles = 0);
                                        return bins.push(currentProjectFileMap), bins;
                                    }(ctx, totalNumOfFilesToProcess, numberOfWorkers), workers = function(numberOfWorkers) {
                                        const res = [];
                                        for (let i = 0; i < numberOfWorkers; ++i) res.push(new (__webpack_require__(71267).Worker)((0, path_1.join)(__dirname, "./project-graph-worker.js"), { env: process.env }));
                                        return res;
                                    }(numberOfWorkers);

So you can see here that still the name of the file is used. Not sure if there is a webpack setting to correct this. Or maybe we can just turn off nx workers with some other option.

[StefanLobbenmeierObjego]

The project-graph-worker.js was removed anyway in 16.8.0, see this PR:
nrwl/nx#18752, mentioned in https://github.com/nrwl/nx/releases/tag/16.8.0

So this might also go away by upgrading the bundled nx version.

For now as a workaround, I added the env variable to limit the nx workers for the project graph using NX_PROJECT_GRAPH_MAX_WORKERS: '2':

            - name: Execute
              uses: e-square-io/nx-distributed-task@v2
              id: execute
              env:
                  NX_PROJECT_GRAPH_MAX_WORKERS: '2'
              with:
                  target: ${{ matrix.target }}
                  projects: ${{ matrix.projects }}
                  uploadOutputs: true
                  nxCloud: false
                  args: ''

[ronnetzer]

Hi @StefanLobbenmeierObjego, thank you for reporting the issue.
Unfortunately we're not maintaining the repo anymore and I strongly suggest to look for alternatives, one that I can recommend is nx-aws-cache

[StefanLobbenmeierObjego]

I see, thank you for your suggestion, will look into that.