NextJS 13 returns fetch failed with UND_ERR_CONNECT_TIMEOUT after running in production mode for a day

[prunusnira]

Summary

Environment

next 13.4
node 18.16.0
running with pm2

Problem

Sometimes nextjs freezes and stop responding. (on production mode, we found that after running for 23 hours)
it returns 500 Internal Server Error on browser, and UND_ERR_CONNECT_TIMEOUT on server.
for me, it does not look like memory leak problem because pm2 says it uses memory under 100MB
When this happens, even pm2 does not restart automatically. it detects nextjs as status: online

Check additional information for detailed server error

is there someone who have same problem? or any workaround on this?

I can't post code because it is already too big code bases and security issue

Additional information

detailed error

TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11457:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async invokeRequest (/opt/mall-front/node_modules/next/dist/server/lib/server-ipc/invoke-request.js:17:12)
    at async invokeRender (/opt/mall-front/node_modules/next/dist/server/lib/router-server.js:254:29)
    at async handleRequest (/opt/mall-front/node_modules/next/dist/server/lib/router-server.js:447:24)
    at async requestHandler (/opt/mall-front/node_modules/next/dist/server/lib/router-server.js:464:13)
    at async Server.<anonymous> (/opt/mall-front/node_modules/next/dist/server/lib/start-server.js:117:13) {
  cause: ConnectTimeoutError: Connect Timeout Error
      at onConnectTimeout (/opt/mall-front/node_modules/next/dist/compiled/undici/index.js:1:82152)
      at /opt/mall-front/node_modules/next/dist/compiled/undici/index.js:1:81644
      at Immediate._onImmediate (/opt/mall-front/node_modules/next/dist/compiled/undici/index.js:1:82034)
      at process.processImmediate (node:internal/timers:476:21) {
    code: 'UND_ERR_CONNECT_TIMEOUT'
  }

environment

    Operating System:
      Platform: linux
      Arch: x64
      Version: #1 SMP PREEMPT_DYNAMIC Tue Jun  6 00:47:06 UTC 2023
    Binaries:
      Node: 18.16.0
      npm: 9.5.1
      Yarn: 1.22.19
      pnpm: N/A
    Relevant Packages:
      next: 13.4.19
      eslint-config-next: 13.4.3
      react: 18.2.0
      react-dom: 18.2.0
      typescript: 5.2.2
    Next.js Config:
      output: N/A

### Example

_No response_

[ADTC]

Next.js 14 removes undici. Try upgrading.

[ADTC]

If all your tests pass I don't see a problem. Note though that it requires Node 18 and above.

Btw, it should be noted that the native fetch inside Node 18+ is undici so I don't guarantee this problem will go away. But it's worth a shot. (See #57875 for more info.)

[ADTC]

Note: https://nextjs.org/blog/next-14

Minimum Node.js version is now 18.17

I see that your Node is one step too old at 18.16. You'll have to upgrade that too.

[prunusnira]

after updated our project to 14.0.1, at least we didn't see any undici related error. thanks
(currently going to 14.0.3 to fix other bug)

[ornakash]

We upgraded to 14.2.3 and we still get the same error.

[zackproser]

I'm hitting the same error with 14.2.3

[muhammad-rizwan1]

Hi,
not sure if I have to raise a new question or continue with this as I'm having undici related error as well.

My Next.js version 14.2.3 and Node v20.11.1 but still getting the error,

digest: "3554945235"
Cause: ConnectTimeoutError: Connect Timeout Error
    at onConnectTimeout (node:internal/deps/undici/undici:7492:28)
    at node:internal/deps/undici/undici:7448:50
    at Immediate._onImmediate (node:internal/deps/undici/undici:7478:37)
    at process.processImmediate (node:internal/timers:478:21)
    at process.callbackTrampoline (node:internal/async_hooks:130:17) {
  code: 'UND_ERR_CONNECT_TIMEOUT'
} 

[alarm109]

Me too. We are using Vercel, in the logs I see that some request go through without errors. Others have one or two of these TypeError: fetch failed, UND_ERR_CONNECT_TIMEOUT.

"next": "14.1.2"
"node": ">=18.17.0"

TypeError: fetch failed
    at node:internal/deps/undici/undici:12345:11
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at runNextTicks (node:internal/process/task_queues:64:3)
    at process.processImmediate (node:internal/timers:449:9)
    at process.topLevelDomainCallback (node:domain:160:15)
    at process.callbackTrampoline (node:internal/async_hooks:128:24)
    at async globalThis.fetch (/var/task/node_modules/.pnpm/next@14.1.2_@babel+core@7.24.0_react-dom@18.2.0_react@18.2.0/node_modules/next/dist/compiled/next-server/app-route.runtime.prod.js:6:36091) {
  cause: ConnectTimeoutError: Connect Timeout Error
      at onConnectTimeout (node:internal/deps/undici/undici:7492:28)
      at node:internal/deps/undici/undici:7448:50
      at Immediate._onImmediate (node:internal/deps/undici/undici:7480:13)
      at process.processImmediate (node:internal/timers:478:21)
      at process.topLevelDomainCallback (node:domain:160:15)
      at process.callbackTrampoline (node:internal/async_hooks:128:24) {
    code: 'UND_ERR_CONNECT_TIMEOUT'
  }
}

[ornakash]

We have the same issue

[hrc7505]

getting same issue during development. In production it is working fine but in localhost it is giving this issue many times. And after getting this issue I have to stop my work for some hours. On next day, it starts working again without any issues. I am randomly getting this issue.

Node: 18.17.1
Nextjs: 14.2.3

[ioExpander]

We're running nextjs 14.1.3, Vercel Node 18, and randomly getting this error on some api calls (trpc). It seems that subsequent requests are ok on that same endpoint.

[Omid-Jafari]

i have same issue with my next14

[Andriy-Kulak]

I am getting this locally randomly when trying to run next.js usin 14.1.4 and node v18.17.1

[lookevink]

randomly getting this in production while fetching mailchimp transactional from a serverless function. maxDuration set to 300 so it's not that

TypeError: fetch failed
    at node:internal/deps/undici/undici:12618:11
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  cause: ConnectTimeoutError: Connect Timeout Error
      at onConnectTimeout (node:internal/deps/undici/undici:7760:28)
      at node:internal/deps/undici/undici:7716:50
      at Immediate._onImmediate (node:internal/deps/undici/undici:7748:13)
      at process.processImmediate (node:internal/timers:476:21)
      at process.callbackTrampoline (node:internal/async_hooks:128:17) {
    code: 'UND_ERR_CONNECT_TIMEOUT'
  }
}

using next 14.0.4

[ornakash]

We get the exact same issue

[andremendonca03]

Hey, just out of curiosity, which API are you trying to fetch?

[lookevink]

mailchimp. managed to curb the inconsistent errors by setting an exponential backoff function to retry until the mail is sent. going to try to play with initial timeout a little this upcoming week. which ones are you fetching?

[andremendonca03]

Im using the Slack messaging API to send a chat message after a form submission.
Do you have any reference code you could share to help with these inconsistent errors?

[lookevink]

Never had this issue with Postmark. 40000ms is ridiculous.

Error sending mail: Z [AxiosError]: timeout of 40000ms exceeded
    at y.<anonymous> (/var/task/.next/server/app/api/services/mailchimp/wh/route.js:3:9058)
    at y.emit (node:events:517:28)
    at Timeout.<anonymous> (/var/task/.next/server/chunks/9555.js:1:17261)
    at listOnTimeout (node:internal/timers:569:17)
    at process.processTimers (node:internal/timers:512:7)
    at tj.request (/var/task/.next/server/app/api/services/mailchimp/wh/route.js:3:20579)
    at async t5 (/var/task/.next/server/app/api/services/mailchimp/wh/route.js:74:497)
    at async t5 (/var/task/.next/server/app/api/services/mailchimp/wh/route.js:74:882)
    at async t5 (/var/task/.next/server/app/api/services/mailchimp/wh/route.js:74:882) {
  code: 'ECONNABORTED',
  config: {
    transitional: {
      silentJSONParsing: true,
      forcedJSONParsing: true,
      clarifyTimeoutError: false
    },
    adapter: [ 'xhr', 'http', 'fetch' ],
    transformRequest: [ [Function (anonymous)] ],
    transformResponse: [ [Function (anonymous)] ],
    timeout: 40000,
    xsrfCookieName: 'XSRF-TOKEN',
    xsrfHeaderName: 'X-XSRF-TOKEN',
    maxContentLength: -1,
    maxBodyLength: -1,
    env: { FormData: [Function], Blob: [class Blob] },
    validateStatus: [Function: validateStatus],
    headers: Object [AxiosHeaders] {
      Accept: 'application/json, text/plain, */*',
      'Content-Type': 'application/json',
      'User-Agent': 'axios/1.7.2',
      'Content-Length': '526',
      'Accept-Encoding': 'gzip, compress, deflate, br'
    },
    method: 'post',
    url: 'https://mandrillapp.com/api/1.0/messages/send-template',
    data: '{"key":"obfuscated"
  },
  request: <ref *1> Writable {
    _writableState: WritableState {
      objectMode: false,
      highWaterMark: 16384,
      finalCalled: false,
      needDrain: false,
      ending: false,
      ended: false,
      finished: false,
      destroyed: false,
      decodeStrings: true,
      defaultEncoding: 'utf8',
      length: 0,
      writing: false,
      corked: 0,
      sync: true,
      bufferProcessing: false,
      onwrite: [Function: bound onwrite],
      writecb: null,
      writelen: 0,
      afterWriteTickInfo: null,
      buffered: [],
      bufferedIndex: 0,
      allBuffers: true,
      allNoop: true,
      pendingcb: 0,
      constructed: true,
      prefinished: false,
      errorEmitted: false,
      emitClose: true,
      autoDestroy: true,
      errored: null,
      closed: false,
      closeEmitted: false,
      [Symbol(kOnFinished)]: []
    },
    _events: [Object: null prototype] {
      response: [Function (anonymous)],
      error: [Function (anonymous)],
      socket: [Array]
    },
    _eventsCount: 3,
    _maxListeners: undefined,
    _options: {
      maxRedirects: 21,
      maxBodyLength: Infinity,
      protocol: 'https:',
      path: '/api/1.0/messages/send-template',
      method: 'POST',
      headers: [Object: null prototype],
      agents: [Object],
      auth: undefined,
      family: undefined,
      beforeRedirect: [Function: tr],
      beforeRedirects: [Object],
      hostname: 'mandrillapp.com',
      port: '',
      agent: undefined,
      nativeProtocols: [Object],
      pathname: '/api/1.0/messages/send-template'
    },
    _ended: false,
    _ending: true,
    _redirectCount: 0,
    _redirects: [],
    _requestBodyLength: 526,
    _requestBodyBuffers: [ [Object] ],
    _onNativeResponse: [Function (anonymous)],
    _currentRequest: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 7,
      _maxListeners: undefined,
      outputData:...

[andremendonca03]

This discussion have been rolling for almost a year now and I am just wondering why have no one raised an issue for this yet.
I have just created one but not sure if it covers all cases:
#66373

[ngroenewold95]

I am also getting a bunch of undici errors as well recently. These are the 3 main ones

"next": "^14.2.3",
node v20.9.0

`TypeError: fetch failed
    at node:internal/deps/undici/undici:12618:11
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  cause: Error: connect ETIMEDOUT 76.76.21.241:443
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1555:16)
      at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:128:17) {
    errno: -110,
    code: 'ETIMEDOUT',
    syscall: 'connect',
    address: '76.76.21.241',
    port: 443
  }`
  

  `TypeError: fetch failed
    at node:internal/deps/undici/undici:12618:11
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  cause: ConnectTimeoutError: Connect Timeout Error
      at onConnectTimeout (node:internal/deps/undici/undici:7760:28)
      at node:internal/deps/undici/undici:7716:50
      at Immediate._onImmediate (node:internal/deps/undici/undici:7748:13)
      at process.processImmediate (node:internal/timers:476:21)
      at process.callbackTrampoline (node:internal/async_hooks:128:17) {
    code: 'UND_ERR_CONNECT_TIMEOUT'
  }
}`

`TypeError: fetch failed
    at node:internal/deps/undici/undici:12618:11
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  cause: [Error: C0AFB780CE7F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:ssl/record/ssl3_record.c:355:
  ] {
    library: 'SSL routines',
    reason: 'wrong version number',
    code: 'ERR_SSL_WRONG_VERSION_NUMBER'
  }
}`

[rafalzawadzki]

Since a few days getting very similar, also undici:

cause: HeadersTimeoutError: Headers Timeout Error
      at Timeout.onParserTimeout [as callback] (node:internal/deps/undici/undici:8942:32)
      at Timeout.onTimeout [as _onTimeout] (node:internal/deps/undici/undici:6983:17)
      at listOnTimeout (node:internal/timers:573:17)
      at process.processTimers (node:internal/timers:514:7) {
    code: 'UND_ERR_HEADERS_TIMEOUT'

both locally and in prod on Vercel.

Sometimes after deleting .next and rebuilding the first request goes through but subsequent fail. Somehow prod seems more stable, locally happens all the time.

tried upgrading 14.0.4 -> 14.2.3 but unfortunately both are affected.

[samyarkd]

is there a solution for this? i get the same error with node 21 and next 14.2.3

[rafalzawadzki]

no-one seems to know, unfortunately.

more convo in this issue too: vercel/vercel#11692

[Tuscan-blue]

I get the same error with node v21.7.3 and next v14.2.4.

 [cause]: ConnectTimeoutError: Connect Timeout Error
      at onConnectTimeout (node:internal/deps/undici/undici:6621:28)
      at node:internal/deps/undici/undici:6573:50
      at Immediate._onImmediate (node:internal/deps/undici/undici:6603:37)
      at process.processImmediate (node:internal/timers:478:21)
      at process.callbackTrampoline (node:internal/async_hooks:130:17) {
    code: 'UND_ERR_CONNECT_TIMEOUT'
  }

[taylor-lindores-reeves]

Currently an issue in production:

TypeError: fetch failed
    at node:internal/deps/undici/undici:12502:13
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  [cause]: ConnectTimeoutError: Connect Timeout Error
      at onConnectTimeout (node:internal/deps/undici/undici:6635:28)
      at node:internal/deps/undici/undici:6587:50
      at Immediate._onImmediate (node:internal/deps/undici/undici:6619:13)
      at process.processImmediate (node:internal/timers:478:21)
      at process.callbackTrampoline (node:internal/async_hooks:130:17) {
    code: 'UND_ERR_CONNECT_TIMEOUT'
  }
}

Although it works in staging environment (Preview) on Vercel, but doesn't in Production environment

[codewithelvin]

The same situation. Couple of hours ago Nextjs app in Docker container stopped funtioning normally.
I looked at container's logs using docker logs and noticed the error below:

 ⨯ TypeError: fetch failed
    at node:internal/deps/undici/undici:12618:11
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async fetchExternalImage (/app/node_modules/next/dist/server/image-optimizer.js:565:17)
    at async NextNodeServer.imageOptimizer (/app/node_modules/next/dist/server/next-server.js:650:48)
    at async cacheEntry.imageResponseCache.get.incrementalCache (/app/node_modules/next/dist/server/next-server.js:182:65)
    at async /app/node_modules/next/dist/server/response-cache/index.js:90:36
    at async /app/node_modules/next/dist/lib/batcher.js:45:32 {
  cause: ConnectTimeoutError: Connect Timeout Error
      at onConnectTimeout (node:internal/deps/undici/undici:7760:28)
      at node:internal/deps/undici/undici:7716:50
      at Immediate._onImmediate (node:internal/deps/undici/undici:7748:13)
      at process.processImmediate (node:internal/timers:476:21)
      at process.callbackTrampoline (node:internal/async_hooks:128:17) {
    code: 'UND_ERR_CONNECT_TIMEOUT'
  }
}

Note that container was up almost 2 weeks and had no issue.

I tried to restart Container, and it does not helped, then restarted NGINx. Updated OS itself (Ubuntu) and restarted, but that doesn't helped either.

[taylor-lindores-reeves]

This is still an issue for us. Considering moving away from Vercel at this point.

[samyarkd]

i get this in docker

[Coderamrin]

I'm getting this on developement.

[dylantientcheu]

@Coderamrin, what do you do to get this on dev. Can you help me reproduce?

[ieuanign]

I also got this on development. Restarting my machine makes it back to normal

[TasmeTime]

same here

[cause]: ConnectTimeoutError: Connect Timeout Error
at onConnectTimeout (node:internal/deps/undici/undici:6635:28)
at node:internal/deps/undici/undici:6587:50
at Immediate._onImmediate (node:internal/deps/undici/undici:6619:13)
at process.processImmediate (node:internal/timers:478:21)
at process.callbackTrampoline (node:internal/async_hooks:130:17) {
code: 'UND_ERR_CONNECT_TIMEOUT'
}

Node v20.15.0
NextJs 14.2.3

[mr-mikey]

Hey guys

So I've resolved my issue by creating this small script and attaching that to the next build (ofc install undici beforehand)

const { setGlobalDispatcher, EnvHttpProxyAgent } = require('undici');

// sets proxy user from env variables
if (process.env.HTTPS_PROXY || process.env.HTTP_PROXY) {
  console.log('Using proxy user:', process.env.HTTP_PROXY || process.env.HTTPS_PROXY);
  console.log('Setting no_proxy to:', process.env.NO_PROXY);
  setGlobalDispatcher(new EnvHttpProxyAgent());
}

and attaching it to the build in package.json
"build": "node -r './proxy_fixer' node_modules/next/dist/bin/next build",

[rafalzawadzki]

@mr-mikey this looks like a sensible solution (in absence of any solutions tbh :)) but do you mind explaining a bit more what it does? does it require setting up a proxy and adding it to HTTPS_PROXY variable?

[mr-mikey]

Pretty much the connection was getting blocked at this stage, so I've dig around a little bit and found out that in order to make undici work properly in my company's environment, I needed to pass the company's proxy with the above "fix-script", because passing it in any other way wasn't sufficient.

This was the only option that worked for me and up to this day its still working

[Shub3am]

I faced this issue while i was working on a slack slash commands API, vercel provides a waitUntil feature which immediately close the response but the waitUntil request can proceed.

https://vercel.com/docs/functions/functions-api-reference#waituntil

[Shub3am]

here's the code which worked for us