Replace deps/cares with pure-javascript alternative

[jbergstroem]

This has been brought up a time or two, so I just thought I'd get a discussion going whether we actually want to do this.

We've somewhat moved away from upstream c-ares. We're not abi compatible any longer which made us remove supporting building against a shared library. The way I see it; we could avoid complexity and c++ by replacing c-ares with a pure-javascript alternative.

The most likely contender for this spot would be node-dns by @tjfontaine which is meant to be a drop-in replacement. It has additional functionality such as a simple dns server which we may (or likely not) be interested in adopting. It has a few dependencies of it's own I'm unsure we'd want to have maintained externally. There's also been a few issues mentioned that needs to be resolved [todo: be more specific].

Before putting too much work in this, are there counter-arguments against such a transition -- or should we perhaps turn it around; why move at all? My main arguments would be that the benefits of c++ doesn't give us much here, as well as not having to stay close to c-ares (for good or bad). It would probably also allow more people to chip in in terms of maintenance.

[bnoordhuis]

I'm in favor of moving to (something like) node-dns.

It has a few dependencies of it's own I'm unsure we'd want to have maintained externally.

I'm guessing 'no'. The dependency chain is a few layers deep and some things seem to be more convenient than optimal.

There's also been a few issues mentioned that needs to be resolved [todo: be more specific].

node-dns doesn't support fallback to TCP, like c-ares does. Probably not an issue in everyday use for most.

It would probably also allow more people to chip in in terms of maintenance.

I'd say that's the best argument for moving away from c-ares.

[mscdex]

+1

[tellnes]

+1

[tjfontaine]

I will provide a better write up on if I think as it exists today if node-dns should be included. I do want to quickly clarify that it does do both tcp and udp lookups.

[silverwind]

I'd definitely be in favor of a javascript implementation that exposes more than resolve does right now. With node-dns, I'm able to access the socket through answer._socket which for example allows me to check the source address of the answer packet.

I have to caution you of dropping in node-dns as-is though. I encountered a throw in async code, which is a no-go, and I think there could be more surprises in there.

[rvagg]

Perf implications?

[tjfontaine]

For hysterical^Whistorical context, allow me to describe why I wrote node-dns/native-dns.

At the time I was fed up with a DNS server I was having to maintain that was written in twisted. Having recently fell in love with Node.js I embarked on writing a DNS application server in JavaScript. I started by re-using someone else's module, until I found a bug and realized that (despite what people were distributing it as on npm) the original source of the module was actually unlicensed. So I went about the work to write my own DNS parser and server, and it was evident the same code could be reusable as a client stack as well. At this point I decided I wanted to it to be a drop in replacement for the dns module found in Node.js, but written in pure JavaScript.

That's the lens you should look at native-dns with (now maintained by @taoeffect [though I owe him some code reviews]). A module that initially wanted to enable application server style workflows (like the http module, but for DNS), that grew a client implementation.

Now to ask the question if I, a maintainer of Node.js, would accept the module I wrote in core? As it stands today, no.

There are far too many places native-dns overreaches in scope for what I would consider to be included in core:

• native-dns.lookup
  • For a drop in replacement, it's important to have this -- but it does not actually implement the interface it claims to.
  • This should have fidelity with the actual system resolver, that is the one maintained by libc, and it's unlikely that you as a user program will be able to know enough about the plumbing of the system to be able to provide that interface. (i.e. it's harder than just reading /etc/nsswitch.conf)
  • It is attractive though to implement, because it takes pressure off the thread pool, but solving that concern should be left to non-core modules.
• DNS server
  • The way this works today in native-dns is too heavy handed for what I think is worthwhile for core.
  • The measuring stick for adding features/APIs to Node.js is often one that asks if there's another module that needs or would benefit from this API. And there's really not something that core needs for this.
  • Though enabling reuse of the code (like the parser etc) would be ideal such that it would be trivial for someone to write their own non-core module to do DNS serving.
• DNS caching
  • This is a trap, completely and utterly a trap, leave this to the forwarders they're going to be way better at it than you.
• Technical debt
  • Unfortunately for @taoeffect and as @silverwind mentioned, this module has warts that come from it growing with me as I grew in Node.js, much of the internals here should be redone.

Now on to the pieces of the API for native-dns that I think are worthwhile.

• native-dns.Request
  • A single shot API to a specific DNS server, you specify up front TCP/UDP (this may be what @bnoordhuis was referring to not having a fallback)
• native-dns.Resolve
  • Behaves like dns.resolve but you're able to pass in per question the list of DNS servers, timeouts, etc

Things that fall out of this implementation that I find neat.

• Full access to the resource records for every result, maybe you want to know TTLs or whatever else, you finally have access to that, it's not locked away in c-ares.
• Observable/Cancelable requests, the handle that's returned by dns.resolve lets you debug how far along you were with a request, and even "cancel" it (where cancel in DNS land mostly involves removing it from the hash table of valid response IDs)

Random thoughts I have on this:

• Moving to a pure JavaScript implementation means that the core can be reused and embedded in other platforms, and the DNS resolution behaves the same across implementations.
• Other platforms who embed the core JavaScript would probably prefer to have their own DNS stacks manage resolution so its consistent on their platform
• Does the core really need to be in the business of maintaining a DNS resolver?
• native-dns doesn't currently do DNSSec, if you're going to be maintaining DNS in core do you want to have to do that work as well?
• Aside from ABI compatibility concerns, what do you want to gain/enable by replacing the DNS implementation (regardless of who/what implements it).

Many of the things I like about native-dns could probably find ways of being achieved with other implementations, or even the current c-ares implementation. It's just a matter of understanding what it is you're trying to solve for by doing the work.

If you decide you want core to have a pure JavaScript implementation, I would advocate for working from an existing code base that can be morphed into the thing you're looking for. I do believe that native-dns with @taoeffect is a good place to coordinate that effort. That being said, I also know @indutny has a DNS parser that he wrote, and there are a myriad of others that can be found in our ecosystem.

Food for thought.

[piscisaureus]

In general, replacing cares by an all-js implementation is is an idea worth exploring IMO.

My preference would be to keep only GetAddrInfo/GetNameInfo (the APIs that net and http rely on) in core, and move the more advanced dns methods to userland. What we need is a transition path.

DNS caching
This is a trap, completely and utterly a trap, leave this to the forwarders they're going to be way better at it than you.

c-ares afaik doesn't do any caching. I've always seen this as problematic; using node-dns might be a way to resolve this long-standing issue.
But I will take @tjfontaine's word for it that it's challenging to get it right.

[tjfontaine]

I think the desire for caching mostly stems from use of dns.lookup and not dns.resolve. And the desire for caching that is to avoid saturating the thread pool with unnecessary work and reducing the amount of C/C++ roundtrips that core is doing.

If we contain the conversation to only dns.resolve the issue is pretty much moot, as most resolvers don't do client side caching. Indeed, c-ares does not do caching -- this is mostly left to the forwarders to handle.

If caching were to be considered in scope, there's almost certainly no way for the opinion of core to get it right for what the users needs are. You could do traditional TTL based caching, but the consumer may need more nuanced semantics.

All in all, caching is effectively wrapping http.request or net.connect with your own thing to cache:

// error handling, typos, etc all reserved -- this is a paraphrase.
var cache = {};
function lookup(name, cb) {
  if (name in cache)
    setImmediate(cb, cache[name]);
  else
    dns.lookup(name, function store(err, ip) {
      cache[name] = ip;
      cb(err, ip);
    });
}

function myconnect(port, name, cb) {
  lookup(name, function(err, ip) {
    net.connect(port, ip, cb);
  });
}

Where you can handle your own invalidation, on whatever semantics you want to achieve.

native-dns grew a concept about caching because it's related to wanting to provide the semantics of holding records that need to be resolved on the server side, which in some ways behaves similarly to what you might need for forwarder caching. But all in all, out of scope for what a pure resolver needs to function.

Splitting DNS resolution out of core is an interesting idea, and indeed fits more in line with the needs of core -- but in my opinion that ship has sailed, the API was included and for many people expected to be there and it (currently) is not too much of a burden to continue to maintain. However, shifting to something that core would be seeking to maintain, that burden could indeed feel too great and decide to not do that at all.

[piscisaureus]

If we contain the conversation to only dns.resolve the issue is pretty much moot, as most resolvers don't do client side caching. Indeed, c-ares does not do caching -- this is mostly left to the forwarders to handle.

But c-ares doesn't even expose the TTL that's reported by the DNS server, so implementing sensible custom caching is almost impossible.

Splitting DNS resolution out of core is an interesting idea, and indeed fits more in line with the needs of core -- but in my opinion that ship has sailed, the API was included and for many people expected to be there

That's true, we can't remove the dns module and break people's code. We need a creative solution for the core-to-userland problem however.
I noticed that recently people have begun to suggest semantic changes to many core modules. The most popular targets are util, assert, url and path.
I am reluctant to support modifications to these modules - even if the proposed changes are reasonable - because it'll break applications. I'd me much more comfortable if these modules were versioned independently.
If we were to find a good solution, changes to the dns module could use the same infrastructure.

[silverwind]

What other modules would be worth considering? I only know of @indutny's dns.js which could be a workable base, but it's far from being done last I tried it.

[tellnes]

But c-ares doesn't even expose the TTL that's reported by the DNS server, so implementing sensible custom caching is almost impossible.

Exposing the TTL in the api would be nice. That is something I've needed in the past.

[taoeffect]

On the topic of a DNS implementation, have any of you had a chance to evaluate/look at/consider getdns-node? It's node bindings for getdns.

Re: node-dns, it needs a good amount of simplification and cleanup IMO. I especially want to see fast packet relay that avoids parsing packets unless necessary (see issues tjfontaine/node-dns#67 and tjfontaine/node-dns#69).

If the C++ from getdns can be integrated into node, then it might be a reasonable alternative (I still need to evaluate it more though to be sure). While I don't like C++, the advantage is that it seems to be a fairly mature library that's being actively maintained by another group.

cc @silverwind @tjfontaine