NX No such file or directory (os error 2) - when using the new remote cache from 20.8

[ianldgs]

Current Behavior

Sometimes, when running tasks (happened with test and build), it fails with:

 NX   No such file or directory (os error 2)

Expected Behavior

Target runs normally

GitHub Repo

No response

Steps to Reproduce

1. Set NX_SELF_HOSTED_REMOTE_CACHE_SERVER to a remote cache server. E.g. this one
2. Run targets like test/build

Got this failure locally (macos) semi-consistently with a next.js app and a large .next/cache folder in dist.

It went away after I cleaned up all the cache:

• find apps -type d -path "*\.next/cache*" | xargs rm -rf
• rm -rf node_modules/.cache
• rm -rf .nx
• nx reset

But it's now happening on CI.

Nx Report

NX   Report complete - copy this into the issue template

Node           : 22.14.0
OS             : darwin-arm64
Native Target  : aarch64-macos
pnpm           : 10.4.0

nx                     : 20.8.0
@nx/js                 : 20.8.0
@nx/jest               : 20.8.0
@nx/eslint             : 20.8.0
@nx/workspace          : 20.8.0
@nx/cypress            : 20.8.0
@nx/devkit             : 20.8.0
@nx/esbuild            : 20.8.0
@nx/eslint-plugin      : 20.8.0
@nx/module-federation  : 20.8.0
@nx/next               : 20.8.0
@nx/node               : 20.8.0
@nx/plugin             : 20.8.0
@nx/react              : 20.8.0
@nx/rollup             : 20.8.0
@nx/storybook          : 20.8.0
@nx/vite               : 20.8.0
@nx/web                : 20.8.0
@nx/webpack            : 20.8.0
typescript             : 5.7.3
---------------------------------------
Community plugins:
@jscutlery/semver      : 5.3.1
@nx-tools/nx-container : 6.1.0
@nxkit/playwright      : 3.0.2
ngx-deploy-npm         : 8.2.0
---------------------------------------
Local workspace plugins:
         @imc-trading/webgui
         @imc-trading/nx-web
---------------------------------------
Cache Usage: 272.04 MB / 92.64 GB

Failure Logs

NX   Successfully ran target test for 38 projects and 5 tasks they depend on

Nx read the output from the cache instead of running the command for 5 out of 43 tasks.

 NX   No such file or directory (os error 2)
Pass --verbose to see the stacktrace.

Time for 'loading dotenv files' 1.3593339999999898
Time for 'code-loading' 53.17854199999999
Time for 'create-project-graph-async' 10.137083000000018
Time for 'total for sendMessageToDaemon()' 8.409125000000017
Time for 'deserialize daemon response' 3.732250000000022
TRACE nx::native::db: process{id=45749}: Creating connection to "/Users/iserpa/code/frontend/.nx/workspace-data/46B517E3-35EA-597B-BF52-B92D070073D4.db"
TRACE nx::native::db::initialize: process{id=45749}: Getting lock on db lock file
TRACE nx::native::db::initialize: process{id=45749}: Got lock on db lock file
TRACE nx::native::db::initialize: process{id=45749}: Checking if current existing database is compatible with Nx 20.8.0
TRACE nx::native::db::initialize: process{id=45749}: Database is compatible with Nx 20.8.0
Time for 'total for sendMessageToDaemon()' 1.2373329999999783
Time for 'deserialize daemon response' 0.006832999999971889
TRACE nx::native::tasks::details: Recording task details
Time for 'hashMultipleTasks' 13.501750000000015
Time for 'total for sendMessageToDaemon()' 10.54725000000002
Time for 'deserialize daemon response' 0.5096249999999714
TRACE nx::native::cache::cache: GET 1385638370963531659
TRACE nx::native::cache::cache: GET 1385638370963531659 70.041µs
TRACE hyper_util::client::legacy::pool: retrieve{hash=1385638370963531659}: checkout waiting for idle connection: ("http", nx-cache.intra)
DEBUG log: retrieve{hash=1385638370963531659}: starting new connection: http://nx-cache.intra/    
TRACE hyper_util::client::legacy::connect::http: retrieve{hash=1385638370963531659}: Http::connect; scheme=Some("http"), host=Some("nx-cache.intra"), port=None
Time for 'total for sendMessageToDaemon()' 1.5515829999999369
Time for 'deserialize daemon response' 0.00483299999996234
DEBUG hyper_util::client::legacy::connect::http: retrieve{hash=1385638370963531659}: connecting to 10.0.0.1:80
TRACE log: retrieve{hash=1385638370963531659}: registering event source with poller: token=Token(5115243008), interests=READABLE | WRITABLE    
DEBUG hyper_util::client::legacy::connect::http: connected to 10.0.0.1:80
TRACE hyper_util::client::legacy::client: http1 handshake complete, spawning background dispatcher task
TRACE hyper_util::client::legacy::pool: checkout dropped for ("http", nx-cache.intra)
TRACE hyper_util::client::legacy::pool: put; add idle connection for ("http", nx-cache.intra)
DEBUG hyper_util::client::legacy::pool: pooling idle connection for ("http", nx-cache.intra)
TRACE nx::native::cache::http_remote_cache: HTTP response status: 200 OK
TRACE nx::native::cache::http_remote_cache: Downloaded 134 bytes from remote cache
TRACE nx::native::cache::http_remote_cache: Retrieved terminal output from cache: 45 bytes
TRACE nx::native::cache::http_remote_cache: Retrieved exit code from cache: 0
TRACE nx::native::cache::http_remote_cache: Extracted tarball to /Users/iserpa/code/frontend/.nx/cache/1385638370963531659
TRACE nx::native::cache::cache: applying remote cache results: "1385638370963531659" (/Users/iserpa/code/frontend/.nx/cache/1385638370963531659)
TRACE nx::native::cache::expand_outputs: No glob patterns found, checking if entries exist
TRACE nx::native::cache::cache: Removing expanded outputs: []
TRACE nx::native::cache::cache: Copying Files from Cache "/Users/iserpa/code/frontend/.nx/cache/1385638370963531659" -> "/Users/iserpa/code/frontend"
TRACE hyper_util::client::legacy::pool: idle interval checking for expired
TRACE nx::native::cache::expand_outputs: No glob patterns found, checking if entries exist
TRACE nx::native::cache::cache: Removing expanded outputs: []
TRACE nx::native::cache::cache: Copying Files from Cache "/Users/iserpa/code/frontend/.nx/cache/1385638370963531659" -> "/Users/iserpa/code/frontend"
TRACE nx::native::cache::expand_outputs: No glob patterns found, checking if entries exist
TRACE nx::native::cache::cache: Removing expanded outputs: []
TRACE nx::native::cache::cache: Copying Files from Cache "/Users/iserpa/code/frontend/.nx/cache/1385638370963531659" -> "/Users/iserpa/code/frontend"
TRACE nx::native::cache::expand_outputs: No glob patterns found, checking if entries exist
TRACE nx::native::cache::cache: Removing expanded outputs: []
TRACE nx::native::cache::cache: Copying Files from Cache "/Users/iserpa/code/frontend/.nx/cache/1385638370963531659" -> "/Users/iserpa/code/frontend"
TRACE nx::native::cache::expand_outputs: No glob patterns found, checking if entries exist
TRACE nx::native::cache::cache: Removing expanded outputs: []
TRACE nx::native::cache::cache: Copying Files from Cache "/Users/iserpa/code/frontend/.nx/cache/1385638370963531659" -> "/Users/iserpa/code/frontend"
TRACE nx::native::cache::expand_outputs: No glob patterns found, checking if entries exist
TRACE nx::native::cache::cache: Removing expanded outputs: []
TRACE nx::native::cache::cache: Copying Files from Cache "/Users/iserpa/code/frontend/.nx/cache/1385638370963531659" -> "/Users/iserpa/code/frontend"

—————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

 NX   Successfully ran target test for project xxx


Task Execution Timings:
{
  "xxx:test": null
}
Time for 'total for sendMessageToDaemon()' 11.31420799999978
Time for 'deserialize daemon response' 0.03341700000237324

 NX   No such file or directory (os error 2)

Error: No such file or directory (os error 2)
    at DbCache.applyRemoteCacheResults (/Users/iserpa/code/frontend/node_modules/.pnpm/nx@20.8.0_@swc-node+register@1.9.2_@swc+core@1.5.29_@swc+helpers@0.5.15__@swc+types@0.1_b4fae504402af20c09600375d5c44547/node_modules/nx/src/tasks-runner/cache.js:119:27)
    at DbCache.get (/Users/iserpa/code/frontend/node_modules/.pnpm/nx@20.8.0_@swc-node+register@1.9.2_@swc+core@1.5.29_@swc+helpers@0.5.15__@swc+types@0.1_b4fae504402af20c09600375d5c44547/node_modules/nx/src/tasks-runner/cache.js:100:22)
    at async TaskOrchestrator.applyCachedResult (/Users/iserpa/code/frontend/node_modules/.pnpm/nx@20.8.0_@swc-node+register@1.9.2_@swc+core@1.5.29_@swc+helpers@0.5.15__@swc+types@0.1_b4fae504402af20c09600375d5c44547/node_modules/nx/src/tasks-runner/task-orchestrator.js:130:30)
    at async Promise.all (index 0)
    at async TaskOrchestrator.applyCachedResults (/Users/iserpa/code/frontend/node_modules/.pnpm/nx@20.8.0_@swc-node+register@1.9.2_@swc+core@1.5.29_@swc+helpers@0.5.15__@swc+types@0.1_b4fae504402af20c09600375d5c44547/node_modules/nx/src/tasks-runner/task-orchestrator.js:126:21)
    at async TaskOrchestrator.applyFromCacheOrRunTask (/Users/iserpa/code/frontend/node_modules/.pnpm/nx@20.8.0_@swc-node+register@1.9.2_@swc+core@1.5.29_@swc+helpers@0.5.15__@swc+types@0.1_b4fae504402af20c09600375d5c44547/node_modules/nx/src/tasks-runner/task-orchestrator.js:223:40)
    at async TaskOrchestrator.executeNextBatchOfTasksUsingTaskSchedule (/Users/iserpa/code/frontend/node_modules/.pnpm/nx@20.8.0_@swc-node+register@1.9.2_@swc+core@1.5.29_@swc+helpers@0.5.15__@swc+types@0.1_b4fae504402af20c09600375d5c44547/node_modules/nx/src/tasks-runner/task-orchestrator.js:85:13)
    at async Promise.all (index 0)
    at async TaskOrchestrator.run (/Users/iserpa/code/frontend/node_modules/.pnpm/nx@20.8.0_@swc-node+register@1.9.2_@swc+core@1.5.29_@swc+helpers@0.5.15__@swc+types@0.1_b4fae504402af20c09600375d5c44547/node_modules/nx/src/tasks-runner/task-orchestrator.js:61:9)
    at async defaultTasksRunner (/Users/iserpa/code/frontend/node_modules/.pnpm/nx@20.8.0_@swc-node+register@1.9.2_@swc+core@1.5.29_@swc+helpers@0.5.15__@swc+types@0.1_b4fae504402af20c09600375d5c44547/node_modules/nx/src/tasks-runner/default-tasks-runner.js:69:16)

TRACE nx::native::db::connection: Closing database connection

Package Manager Version

pnpm@10.4.0

Operating System

• macOS
• Linux
• Windows
• Other (Please specify)

Additional Information

No response

[ianldgs]

@FrozenPandaz / @AgentEnder I'm happy to work with you guys to debug this.
Let me know if there's any other flag or pre-release that we could get more details on the problem, like a stack trace or smth

[ianldgs]

Ok, I got it consistently working again, locally. Tracked it to the attached cache tarball.

Running the test target of one of our projects produced it, and it gets used as the remote cache for following runs.
The project uses nextjs and jest, and there are actually no tests in that project.

ls -lah cache-1385638370963531659
total 16
drwx------@   4 iserpa  staff   128B Apr 16 12:47 .
drwx------@ 205 iserpa  staff   6.4K Apr 16 12:47 ..
-rwxrwxrwx    1 iserpa  staff     4B Jan  1  1970 code
-rwxrwxrwx    1 iserpa  staff    45B Jan  1  1970 terminalOutput

cat cache-1385638370963531659/code # (empty output, nothing inside this file)
cat cache-1385638370963531659/terminalOutput 
No tests found, exiting with code 0

cache-1385638370963531659.tgz

[ianldgs]

Update:
found some stack trace by running the failing task in isolation with --verbose. Added it to the description.
found trace logs by enabling NX_NATIVE_LOGGING=trace, also added to the description.

[ianldgs]

Seems to be related to tasks that declare outputs, but none of those outputs are in the cache.

For example, jest and vitest declare outputs (coverage directory), but if you run tests without coverage, it won't produce those outputs.

    "test": {
      "executor": "@nx/jest:jest",
      "outputs": ["{workspaceRoot}/coverage/{projectRoot}"],
...
    "test": {
      "executor": "@nx/vite:test",
      "outputs": ["{options.reportsDirectory}"],

Then, when the cache results are being applied, and it looks for those outputs, which don't exist, it throws that "No such file or directory" error.

[ianldgs]

@FrozenPandaz / @AgentEnder / @vsavkin can we please get an update on this? Is there a plan to include a fix, at least before the official v21 launch?

[AgentEnder]

Hi, yes this needs to be fixed. Its on my list, just tending to some other stuff still too.