Happy Eyeballs support (address IPv6 issues in Node 17)

[andreialecu]

What is the problem this feature will solve?

It appears that node 17 has merged this PR: #39987 which may result in IPv6 addresses being sorted first.

As per an answer in this reddit thread from @treysis: https://www.reddit.com/r/ipv6/comments/qbr8jc/comment/hhmg7uz/?context=3 it appears that Happy Eyeballs is not implemented however.

The lack of Happy Eyeballs support can result in connections failing. In particular, there are a lot of data points related to similar support lacking from the OkHttp library which is used heavily on Android and results in connectivity issues for a relatively big number of users. Ref: square/okhttp#506 (comment)

The problem is that consumer routers sometimes glitch out on IPv6.

A router I manage in one particular location, which is a consumer grade TP-Link router, will routinely hang on IPv6 connections about once per month, until it is restarted. More on this here: square/okhttp#6954 (comment)

I have noticed that Node 17 will then pretty much hang on outgoing connections for up to one minute before eventually being able to connect. Restarting the router resolves the issue.

What is the feature you are proposing to solve the problem?

Implement the Happy Eyeballs algorithm.

What alternatives have you considered?

There is an user-land patch here: https://www.npmjs.com/package/happy-eyeballs by @zwhitchcox which might help as a workaround.

[aduh95]

I agree Node.js would benefit having this built-in. Is this something you'd be interested in working on @andreialecu?

[tniessen]

I am not sure what subsystem this would fit into. It seems that Happy Eyeballs not only applies to HTTP (http, http2, https), but also to other IPv4/IPv6 sockets, i.e., any TCP (net) or UDP socket (dgram).

[andreialecu]

There's a writeup on how this is implemented in OSX here by an Apple engineer:

https://mailarchive.ietf.org/arch/msg/v6ops/DYiI9v_O66RNbMJsx0NsatFkubQ

The updated implementation performs the following:

• Query the DNS resolver for A and AAAA.
  If the DNS records are not in the cache, the requests are sent back to back on the wire, AAAA first.
• If the first reply we get is AAAA, we send out the v6 SYN immediately
• If the first reply we get is A and we're expecting a AAAA, we start a 25ms timer
  • If the timer fires, we send out the v4 SYN
  • If we get the AAAA during that 25ms window, we move on to address selection
• When we have a list of IP addresses (either from the DNS cache or by receiving them close together with v4 before v6),
  we perform our own address selection algorithm to sort them. This algorithm uses historical RTT data to prefer addresses
  that have lower latency - but has a 25ms leeway: if the historical RTT of two compared address are within 25ms of each
  other, we use RFC3484 to pick the best one.
• Once the list is sorted, we send out the SYN for the first address and start timers based on average and variance of the
  historical TCP RTT. Roughly speaking, we start the second address around the same time we send out a SYN retransmission
  for the first address.
• The first address to reply with a SYN-ACK wins the race, we then cancel the other TCP connection attempts.

There's also an official RFC at: https://datatracker.ietf.org/doc/html/rfc8305

Since it deals with SYN packets, I'm not sure if this can be implemented in JS land or it needs to touch the C++ networking layer. I'm not familiar at all with nodejs networking internals.

[treysis]

Yes, I believe it should go into net. Doesn't the http module make use of net in the end as well?

Happy Eyeballs with UDP being connectionless is a bit more complicated.

[tniessen]

Doesn't the http module make use of net in the end as well?

Mostly, yes, at least by default.

As of 2017, IPv6 brokenness is now generally regarded as a non-problem. Wikipedia

I am wondering how much complexity we can justify just to work around broken IPv6 stacks. The algorithms seem useful even beyond IPv6 compatibility, but so far I have no idea how we could implement this. We probably cannot do this for all TCP connections, at least not by default. We could maybe add an option to socket.connect() to enable this behavior. However, an implementation would probably require changes within libuv first, which is responsible for establishing TCP connections.

[treysis]

We probably cannot do this for all TCP connections, at least not by default.

Why not? If I am not wrong this is the default for all Happy Eyeballs implementations. Though it uses some kind of cache.

[andreialecu]

As of 2017, IPv6 brokenness is now generally regarded as a non-problem.

That statement is probably not factual. At least not at scale. Even if 1% of users are affected, depending on the number of users, that can add up quite a bit.

Reddit has recently disabled IPv6 due to issues they were having:
https://reddit.com/r/ipv6/comments/r047ah/redditcom_has_deleted_aaaa_records_and_is_back_to

Also their Android app does not implement Happy Eyeballs, and here's a report specifically pinpointing this to be the problem:
https://reddit.com/r/ipv6/comments/qy70hv/new_phone_failing_to_load_any_ipv6_websites_on/hlebckp/?utm_source=reddit&utm_medium=web2x&context=3

There's also a huge thread here: aws-amplify/amplify-js#5539 about an Android only issue that I was also running into, and which was specifically about Happy Eyeballs.

Additionally, there are many other threads in other JS ecosystems related to Happy Eyeballs: facebook/react-native#29608 (comment)

There are tons of reports still popping up. And it goes unreported most of the time, because it's very random and pretty much unreproducible since it's hard to actually pin it down to IPv6 being the culprit.

[tniessen]

@andreialecu Don't get me wrong, I am not saying there are no issues. I frequently run into some myself. That was just me quoting Wikipedia. IPv6 is older than myself and I really wish we didn't have to worry about broken IPv6 stacks at this point 😄

Either way, as I said in my last comment, someone would probably have to implement this in libuv before we could consider adding it in Node.js.

[andreialecu]

Either way, as I said in my last comment, someone would probably have to implement this in libuv before we could consider adding it in Node.js.

Perhaps this might help with that: https://github.com/shtrom/happy-eyeballs-c

[bnoordhuis]

someone would probably have to implement this in libuv before we could consider adding it in Node.js

Not necessary, just call uv_tcp_connect() on two uv_tcp_t handles and keep the winner.