SIGNAL USERS READ THIS FIRST: code: 'ERR_INTERNAL_ASSERTION' in internalConnectMultiple

[yuki12321]

Version

20.0.0

Platform

Darwin MacBookPro.local 22.3.0 Darwin Kernel Version 22.3.0: Mon Jan 30 20:42:11 PST 2023; root:xnu-8792.81.3~2/RELEASE_X86_64 x86_64

Subsystem

No response

What steps will reproduce the bug?

I ran the Storybook installation on Next.js 13.3.0 and this is what I see.

How often does it reproduce? Is there a required condition?

No response

What is the expected behavior? Why is that the expected behavior?

No response

What do you see instead?

$ npx sb init

 storybook init - the simplest way to add a Storybook to your project. 

 • Detecting project type. ✓
 • Adding Storybook support to your "Next" app
node:internal/assert:14
    throw new ERR_INTERNAL_ASSERTION(message);
    ^

Error [ERR_INTERNAL_ASSERTION]: This is caused by either a bug in Node.js or incorrect usage of Node.js internals.
Please open an issue with this stack trace at https://github.com/nodejs/node/issues

    at new NodeError (node:internal/errors:399:5)
    at assert (node:internal/assert:14:11)
    at internalConnectMultiple (node:net:1106:3)
    at Timeout.internalConnectMultipleTimeout (node:net:1637:3)
    at listOnTimeout (node:internal/timers:575:11)
    at process.processTimers (node:internal/timers:514:7) {
  code: 'ERR_INTERNAL_ASSERTION'
}

Node.js v20.0.0

Additional information

No response

[arllop]

this happened when i open signal app on mac laptop. has this been fixed? thanks.

[tniessen]

@arllop If Signal does not override NODE_OPTIONS, the workaround in #47644 (comment) might work.

[bnoordhuis]

I've pinned this issue in the (possibly in vain) hope it'll stem the tide of duplicate bug reports.

[Shnub]

Also got this just now from Signal (6.20.0) after waking macOS from sleep with Signal running:

Unhandled Error

Error [ERR_INTERNAL_ASSERTION]: This is caused by either a bug in Node.js or incorrect usage of Node.js internals.
Please open an issue with this stack trace at https://github.com/nodejs/node/issues

    at new NodeError (node:internal/errors:399:5)
    at assert (node:internal/assert:14:11)
    at internalConnectMultiple (node:net:1077:3)
    at afterConnectMultiple (node:net:1532:5)

[bnoordhuis]

Vain hope indeed, dear god. I've changed the title, let's hope it's enough for Signal users to put two and two together 🤦

[Shnub]

While I see how this is annoying for developers here, do kindly keep in mind that this error message is shown to Signal end users, very few of which are developers. And the message explicitly asks to open an issue here, so that's what folks are doing. With the error message itself being basically gibberish to laypeople, it's little wonder some end up opening redundant issues with little more than good intentions.

[joyeecheung]

I wonder if we should just disable network_family_autoselection by default again until the bug is fixed. The assert() we have is under the assumption that this should rarely get hit and when it does get hit we want to know about it. In this case this gets hit too often that the assert() here already becomes more annoying than it is helpful.

[tniessen]

@joyeecheung According to @ShogunPanda, this will be fully resolved by 20.3.0. However, even then, the implementation does not conform to the Happy Eyeballs RFC and may result in timeouts that did not occur in previous versions of Node.js, so I wouldn't be opposed to disabling it by default.

[ShogunPanda]

I can confirm I fixed all variants of this bug so after 20.3.0 all should be good.
About deviating from RFC, well, we never committed to have a compliant implementation of that, but just a loose one.

Disabling it again by default will cause harm to people with broken IPv6 stack.

The best course of action, IMVHO, would be to wait for 20.3.0 to settle a little bit and see if bug reports stop. If not we disable it. WDYT?

[RBCK]

Unhandled Error

Error [ERR_INTERNAL_ASSERTION]: This is caused by either a bug in Node.js or incorrect usage of Node.js internals.
Please open an issue with this stack trace at https://github.com/nodejs/node/issues

at new NodeError (node:internal/errors:399:5)
at assert (node:internal/assert:14:11)
at internalConnectMultiple (node:net:1077:3)
at afterConnectMultiple (node:net:1532:5)

[Veercodeprog]

you can include the NODE_OPTIONS environment variable in the script command to avoid this error. For example:
json

"scripts": {
"start": "NODE_OPTIONS=no-network-family-autoselection node your-app.js"
}

[ShogunPanda]

@Veercodeprog Confirmed. This can temporarily disable the problem until 20.3.0 is released.

[Veercodeprog]

what the correct solution then. i can see this mainly arises while trying to upload images from server to cloudinary.

[ShogunPanda]

There's no other required on your side. This bug was in node and I already fixed it. It will be publicly available once 20.3.0 is released.

Once you update to 20.3.0 everything will work as it was before.

[Veercodeprog]

do we get this error in stable versions too
?

[brian6932]

There's no other required on your side. This bug was in node and I already fixed it. It will be publicly available once 20.3.0 is released.

Once you update to 20.3.0 everything will work as it was before.

I've updated to 20.3.0, however I still exhibit this issue.
It does seem like it's fixed some IPv6 timeout issues for me tho.

[ShogunPanda]

@brian6932 Can you post here for which hosts this failed and the dig query for them? Thanks!

[brian6932]

@ShogunPanda I don't have an issue with a domain resolve, connecting with net/tls works fine for me, but a library I use exhibits the issue

[ShogunPanda]

I see. Can you please name which library and how to reproduce the issue?

[brian6932]

@ShogunPanda KararTY/dank-twitch-irc#13

[Veercodeprog]

Maybe a way of node to suggest code improvement.

[Veercodeprog]

Can be fixed without changing any node settings or package,if we improve our code

[ShogunPanda]

Hello. The fix is in the PR #48464. I'll let you know once it lands so you can try this again a nightly build.

[bnoordhuis]

@indutny Fedor, can you let me know when Signal picks up the fix so I can unpin this issue again? Cheers.

[krousseau]

This is still occurring for me with node 20.4:

Error [ERR_INTERNAL_ASSERTION]: This is caused by either a bug in Node.js or incorrect usage of Node.js internals.
Please open an issue with this stack trace at https://github.com/nodejs/node/issues

    at new NodeError (node:internal/errors:405:5)
    at assert (node:internal/assert:14:11)
    at internalConnectMultiple (node:net:1115:3)
    at Timeout.internalConnectMultipleTimeout (node:net:1683:3)
    at listOnTimeout (node:internal/timers:575:11)
    at process.processTimers (node:internal/timers:514:7)

[tniessen]

@krousseau Could you please confirm that this does indeed happen with Node.js 20.4.0? Is there any additional information you could provide? cc @ShogunPanda

[dmcr]

Have been redirected here as per my closed duplicated ticket.

So as I understand it this is still an active issue in v20.4 and is related to the network area with the current solutions being to:
Apply NODE_OPTIONS="--no-network-family-autoselection" or to use node ^v19.

I have been redirected here but this is also closed so presumably the best place to follow as to when ^v20 is usable is now #48763 #request/request#3458

I can confirm that going back to v19 resolves this issue in production for me.

[ShogunPanda]

@dmcr can you please provide a repro repo or at least which host are you trying to connect to and how your DNS resolves such host?

[AMNKramer]

Issue still occurring for node 20.5.1 with the same stacktrace.
Randomly starts occurring and keeps happening for multiple hours after which it randomly stopped.
Node is running as a kubernetes service and not using workers or websockets (like #48763).
Using the docker image node:20.5.1

[ShogunPanda]

@Rand0mF In order to address this, I need to know which host your node app is connecting to and how your local (or Kubernetes system) is resolving it. Can you provide me such info?

[AMNKramer]

@ShogunPanda Unfortunately I don't know.. The app is doing many queries to all kinds of hosts. The error logs don't provide any useful information for that, that's the only thing which is printed. Reverse engineering is difficult as it seems to occur completely at random, I didn't yet find a way to reproduce it.

Error [ERR_INTERNAL_ASSERTION]: This is caused by either a bug in Node.js or incorrect usage of Node.js internals. Please open an issue with this stack trace at https://github.com/nodejs/node/issues
    at new NodeError (node:internal/errors:405:5)
    at assert (node:internal/assert:14:11)
    at internalConnectMultiple (node:net:1118:3)
    at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
    at listOnTimeout (node:internal/timers:575:11)
    at process.processTimers (node:internal/timers:514:7)

[ainsleyclark]

Very sporadic error which seems to be difficult to debug locally as @Rand0mF mentioned.
export NODE_OPTIONS=--no-network-family-autoselection works.
All v20.x versions are producing the same error.

[ShogunPanda]

@ainsleyclark Is the same for you? Can you provide a list of which hosts the production system is connecting to and how these are resolved by the production system DNS servers?

[pimterry]

I'm seeing lots of these errors too, on Node v20.8.0. Similar to other reports, it seems to be very intermittent and only happening under load. I can't confirm the addresses involved either unfortunately (software is running on end user machines, I just see exception reports).

[AMNKramer]

@ShogunPanda
list of hosts will still be difficult, here's the dig output from within the container for

cluster-external hosts

dig google.com

; <<>> DiG 9.18.19 <<>> google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7886
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;google.com.			IN	A

;; ANSWER SECTION:
google.com.		30	IN	A	142.250.185.142

;; Query time: 2 msec
;; SERVER: 10.251.0.10#53(10.251.0.10) (UDP)
;; WHEN: Thu Nov 16 14:03:26 UTC 2023
;; MSG SIZE  rcvd: 55

cluster-internal hosts

dig hello-world.default.svc.cluster.local

; <<>> DiG 9.18.19 <<>> hello-world.default.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13182
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;hello-world.default.svc.cluster.local. IN A

;; ANSWER SECTION:
hello-world.default.svc.cluster.local. 30 IN A	10.251.3.203

;; Query time: 8 msec
;; SERVER: 10.251.0.10#53(10.251.0.10) (UDP)
;; WHEN: Thu Nov 16 14:08:01 UTC 2023
;; MSG SIZE  rcvd: 71

any other flags which would be helpful?

[ShogunPanda]

Nope, that should be it. I will keep you posted.

[stefanrows]

Still experiencing the issue with Node 20^. Switching to Node 18 solves the problem for me.

[ShogunPanda]

This should be solved by #51045. Will keep you as soon as this reaches 20.x.

[guillenotfound]

Any updates on when this is going to be fixed for 20.x @ShogunPanda ?

[brian6932]

It has been for a long time @guillenotfound https://nodejs.org/en/blog/release/v20.12.0#new-connection-attempt-events.
Latest node release's v21.7.1...

[ShogunPanda]

@guillenotfound It has been release in 20.12.0.