Repository navigation
DNS Lookups Failing After Updating to Latest Node 20 and 18 for IBM i #53948
Description
Activity
(v22) $ node -e 'require("dns").resolve("smtp.office365.com", function() { console.log(arguments); })' [Arguments] { '0': null, '1': [ '52.96.184.50', '52.96.183.226', '52.96.165.210', '52.96.48.50' ] }(I also couldn't reproduce on v18, I got similar output to the one above)
I'm unable to reproduce, but I'm not on IBM i.
Have you tried on other computers/networks?
Additionally, is this reproducible on v22?
CC @nodejs/platform-ibmi
I should have clarified -- this is only happening on IBM i. The problem is not occurring on other platforms.
And only with Node 20.15.1 and Node 18.20.4. The problem started occurring after I updated those on my IBM i from the slightly older versions I mentioned above.
The problem also happens in Node 22.4.1 for IBM i.
The problem does not occur in Node 16 and Node 14 for IBM i, on the same system.
Based on this, and based on things working in the slightly older Node 18/20 versions I mentioned above, we suspect that something in the updated Node 18/20/22 versions for IBM i changes how it does DNS lookups.
I'm not sure if this is relevant or not, but on the affected system:
- The native IBM i CL command
NSLOOKUPworks:NSLOOKUP HOSTNAME(smtp.office365.com) - But, the PASE command
nslookup smtp.office365.comfails with this output:
;; connection timed out; no servers could be reachedWe're not sure when the PASE command stopped working, or if it ever worked on this system. We never tried it before today, while trying to troubleshoot this issue. We're thinking that it probably never worked, as the system DNS configuration is good, works fine with all other software on the system, and has not been changed.
- The native IBM i CL command
@DavidRusso what does Node think your configured servers are?
node -e 'console.log(require("dns").getServers())'When I run this it shows 127.0.0.1 on Node 14, 16, 18, and 20. Prior to Node 20, I get ECONNREFUSED when I try to run your example, but on Node 20 it's now ETIMEOUT. I don't have a DNS server running locally, so either of those errors makes sense.
Also, I'm not sure where you got an
nslookupPASE command as we don't ship one.@kadler ,
The PASE
nslookupcommand I mentioned is/usr/bin/nslookupwhich is a symlink to/QOpenSys/QIBM/ProdData/OS400/DNS/bin/nslookup. I assumed that was provided by IBM based on the location.- I also get
127.0.0.1fromdns.getServers()on all of those Node.js versions. - I also get the same behavior you noted where the older versions throw ECONNREFUSED and newer throw ETIMEOUT.
- This actually explains why things appeared to work for me on Node 14 and 16, because I'm using the
nodemailerpackage and it responds differently whendns.resolve()fails with ECONNREFUSED vs. ETIMEOUT. To be clear,dns.resolve()is actually not working for me with Node 14 or 16, either. - Also, if my system was configured to use localhost for DNS then both errors would also be expected, because no DNS server running on local host.
- But, the thing is... my IBM i system is not configured that way. There are 2 external DNS servers configured under the
CHGTCPDMNcommand. So, the question is... if Node doesn't get the DNS server addresses from there, where does it get them from?
- I also get
@kadler ,
Now that I understand more, it seems possible/likely that
dns.resolve()never worked on IBM i to begin with. As you pointed out, it thinks the DNS server IP is127.0.0.1, instead of using the DNS configuration from the OSCHGTCPDMNcommand.The difference in behavior in the updated Node.js versions is just that now it throws ETIMEOUT instead of ECONNREFUSED when it can't connect to a DNS service on localhost.
This confused us at first, because the software we were using (
nodemailer) reacts differently to the error message and for ECONNREFUSED it falls back todns.lookup()which works. Apparentlydns.lookup()calls an OS facility to do the lookup instead of trying to do it's own DNS comms over the network.So, the root problem or question here is... why does
dns.resolve()anddns.getServers()not use the system DNS config fromCHGTCPDMN? Is that a bug?The PASE
nslookupcommand I mentioned is/usr/bin/nslookupwhich is a symlink to/QOpenSys/QIBM/ProdData/OS400/DNS/bin/nslookup. I assumed that was provided by IBM based on the location.Ohhh, I didn't know the DNS server option includes that.
So, the root problem or question here is... why does
dns.resolve()anddns.getServers()not use the system DNS config fromCHGTCPDMN? Is that a bug?My guess is that it's just following the AIX code path and probably trying to read /etc/resolv.conf and falling back? @abmusse can you investigate further there?
Once we know where this is coming from, I have some code somewhere that calls the ILE APIs to get the configured DNS servers and we can try to fix it that way.
The difference in behavior in the updated Node.js versions is just that now it throws ETIMEOUT instead of ECONNREFUSED when it can't connect to a DNS service on localhost.
Potentially related is a c-ares change from 1.21.0+: #50743
Although for our own tests it was a change fromEBADRESPtoETIMEOUT.Reacted by Abdirahim MusseMy guess is that it's just following the AIX code path and probably trying to read /etc/resolv.conf and falling back? @abmusse can you investigate further there?
Once we know where this is coming from, I have some code somewhere that calls the ILE APIs to get the configured DNS servers and we can try to fix it that way.
@kadler, Yes I will need to investigate this further. I plan to look into this more next week and will provide an update of what I find.
Quick update, I haven't had an opportunity to investigate more this week. This is still on my radar to investigate.
Hey @abmusse, we have the same issue here at some of our customers on IBM I (7.4). We also updated Node (NodeJs 20.16.0) and since then encountered the problem using
nodemailer. We pasted the ip address manually to bypass this but this is just a temporary workaround of course.
We also tried to set the DNS manually like thisimport dns from 'dns'; dns.setServers(['8.8.8.8', '8.8.4.4', '1.1.1.1', '1.0.0.1']);without any success.
The same Node.js is working without any problems on our Mac OS with the same Node 20.16.0 version.Do you have any updates yet or are you still investigating?
Many thanks for your support
1 remaining item
Thanks for the additional reports. I have not forgot about this one. I've been busy putting out fires in other areas and this one is still on my radar.
Like @kadler mentioned in #53948 (comment)
I think we are following the AIX code path and probably trying to read /etc/resolv.conf but we need to add a patch to perform IBM i specific DNS changes to get things working properly.
@DavidRusso Did you ever make any progress on this on your end ?
We just reverted to using an older Node (14) for our process that was running
nodemaileruntil this is fixed in Node.js. That worked fine for the particular process we were having a problem with.Until this is fixed in Node.js, the only way I could see to get
nodemailerworking on current versions of Node.js for IBM i would be to open a PR with thenodemailerauthors to make it trydns.lookup()first, instead ofdns.resolve().That, or bypass DNS lookup altogether by using IP address. Obviously that could be problematic if the mail server IP is subject to change.
I found that if you add an
/etc/resolv.conffile to the system and populate with name servers, then Node for IBM will get the DNS servers from there and this allowsdns.resolve()to work w/them.For "100% super duper" integration with IBM i, I still would expect Node to get the name servers configured under the CHGTCPDMN command, as normally an IBM i system doesn't have the info in
/etc/resolv.conf.But, at least there is a way to get it working.
Reacted by Richard Schoen and Abdirahim MusseNice. Thanks for sharing that.
@DavidRusso That is an awesome find and it looks like @kadler suspicion was true! See #53948 (comment)
My status debugging this, I was following the call stack from the original JS error down in the cares code where the socket was getting an error. Spinning my tires figuring out why this was not working.
This update helps get us back on track to finding a solution for this issue. We will need to add a patch to cares to get the name servers on IBM i.
This issue has been marked as stale due to 210 days of inactivity.
It will be automatically closed in 30 days if no further activity occurs. If this is still relevant, please leave a comment or update it to keep it open.I'm adding debug instrumentation to the c-ares DNS library in Node.js v20.20.1 to trace the exact failure point on IBM i. The debug patch adds logging to
deps/cares/src/lib/ares_sysconfig.canddeps/cares/src/lib/ares_sysconfig_files.cto track platform detection, file access attempts (specifically/etc/resolv.conf), DNS server parsing, and final configuration. I expect this will confirm that IBM i uses the generic file-based path (like AIX),/etc/resolv.confdoesn't exist, no DNS servers are configured (defaulting to 127.0.0.1), and queries timeout. Once build is done and confirmed with debug output from IBM i, I'll try to implement a permanent fix.@sravani1510 you can use ILE API to retrieve DNS information from the system: https://www.ibm.com/docs/api/v1/content/ssw_ibm_i_74/apis/qtocrtvtcpa.htm#TCPA1400
I have some example code I wrote to do this a while back: get-dns.c. It would need quite some adjustments yet:
- looking up the the procedure pointer with _ILESYMX should be cached per-pid and made thread-safe
- retrieve domain name search list (needs conversion from EBCDIC)
- map DNS options (search order, retries, etc) to ares_sysconfig_t fields
Reacted by SRAVANI GUNDEPALLINode.js on IBM i follows the AIX code path in c-ares, which expects DNS configuration in
/etc/resolv.conf. However, IBM i configures DNS through theCHGTCPDMNILE command, not flat files. This causes DNS lookups to fail withETIMEDOUTin newer Node.js versions.I have created a new c-ares module (
ares_sysconfig_pase.c) that uses IBM i's nativeQtocRtvTCPAILE API to retrieve DNS configuration directly from the system. The implementation dynamically loads the ILE API, parses the returned data structures to extract DNS servers (IPv4 and IPv6), and converts EBCDIC domain search lists to ASCII. I am currently working on building this on IBM i to verify that my changes work as expected.I opened a PR in the c-ares repository to solve the issue - c-ares/c-ares#1166
This issue has been marked as stale due to 90 days of inactivity.
It will be automatically closed in 30 days if no further activity occurs. If this is still relevant, please leave a comment or update it to keep it open.
Version
v20.15.1
Platform
Subsystem
DNS
What steps will reproduce the bug?
Any DNS lookup fails with connection timeout. Can run this code to reproduce:
Fails with
ETIMEOUT, even though the system DNS configuration is good and other software can perform DNS lookups.How often does it reproduce? Is there a required condition?
Every time. This started happening when updating to Node 20.15.1 and Node 18.20.4 for IBM i. Everything was working fine before the update. Before, I was using Node 20.11.1 and Node 18.18.2.
What is the expected behavior? Why is that the expected behavior?
I expect a good DNS lookup, as the system DNS configuration is good and most other software can perform lookups.
What do you see instead?
ETIMEOUT on any DNS lookup.
Additional information
An interesting thing we noticed that may relate....
On the affected systems, the NSLOOKUP CL command can perform lookups with no problem, but the
nslookupcommand in PASE fails with ETIMEOUT for any lookup.