DNNSEC not working when stubby run as systemd service. Works fine run stubby run manually #106
Comments
|
What does stubby -i say? |
|
stubby -i output is as follows (Apologies, I tried formatting the following in the code, but it was not working very well as none of the line breaks worked and it looked very hard to read): [20:54:35.422324] STUBBY: Read config from file /etc/stubby.yml I’m guessing it’s something to do with DNSSEC stuff towards the top? My question would be why it would make a difference whether or not the daemon is started manually or via systemd how this works, when they apply the same config file? |
|
@eccgecko Zero configuration DNSSEC needs a writeable I managed to fix it by including a writeable
After doing a query, I noticed the AD bit in the result, and also that Zero configuration DNSSEC succeeded since it downloaded the root trust-anchor and root DNSKEY rrset to track in the @ArchangeGabriel I think it would be good to have that |
|
We are supposed to have DNSSEC working OOTB already on Arch (I build getdns with But @eccgecko is not on Arch anyway. |
|
@ArchangeGabriel acknowledged. Alternatively you could give the stubby user a writeable home directory... |
|
@ArchangeGabriel Oh yes... as a side note, having Zero configuration DNSSEC working would be more robust on systems that haven't been updated when the KSK rolls over. |
|
When adding appdata_dir: "/run/stubby" to /etc/stubby/stubby.yml and doing a sudo systemctl daemon-reload and restart of service I still have output: echo ~stubby although stubby -i shows "appdata_dir": <bindata of "/run/stubby/"> Any clue? |
|
That’s normal, you did not change the |
|
Thanks, learning every day here.. 🙂 |
|
@wtoorop Thanks. That makes sense. @ArchangeGabriel is correct, I am not running Arch but the Raspbian flavor of Debian, but I do, like you, also have the systemd service to execute as
Sorry if I am being dense here - what exactly is the correct method for inserting this line? |
|
Just adding something like this at the top [at line 24-25 for example] should work: # Include a writeable appdata_dir for Zero configuration DNSSEC. Watch out for spacing. |
|
@wtoorop @ArchangeGabriel After adding appdata_dir: "/var/run/stubby" it works fine now. I use a slightly little different configuration stubby.service with: [Unit] [Service] [Install] Since systemd 235 this DynamicUser configuration is possible. And it works very well. It is kind of magic to see the folder appear out of nothing creating root.key, root-anchors.p7s and root-anchors.xml. |
|
@wtoorop I managed to get it working in the end by adding it around line 24-25 like you said (I was adding it on at the end before). However, unfortunately it hasn't managed to fix my issues with DNSSEC zero-config. In fact, I don't know how excatly, but it even seems to have made it slightly worse, as now I added both |
|
@eccgecko Oh that's a pity. Could you do a |
|
Sure. Again, apologies that I can't seem to get the formatting right. [12:51:16.164408] STUBBY: Read config from file /etc/stubby.yml |
|
You’re missing a leading / in appdata_dir. |
|
Yes that's probably it... and also make sure |
|
Thanks, that's pretty much solved it. I think we're very close to completely solving it. Yes, the missing leading / was part of the problem, although that was a mistake I made when I changed it from However, the main issue is with permissions on the folder. I believed the folder permissions were already correct, as they had been when I had checked before. However, it seems that they aren't persisting through a reboot, and that's the problem I've been facing, as I hadn't checked again since the first time I checked. Changing the permissions of /var/run/stubby so that stubby group has read and write permissions solves the issue, and www.dnssec-failed.org no longer replies, and there are indeed trust anchor files created within the /var/run/stubby folder 👍 :) However, when I reboot, the permissions revert to only being read, write, execute for root, and just executable for the stubby group i.e. stubby user. How can I make permissions for /var/run/stubby persistent? |
|
@eccgecko What are the permissions before you change them? Do you know how this folder is created on your system? |
|
Acknowledged. This is due to the line |
|
@wtoorop That's it! 👍 great! DNSSEC now working and persisting through reboots. Thank you and @ArchangeGabriel for all your help with this :) |
|
My stubby.service example needs some attention. I was not completely sure about the use of RuntimeDirectory and StateDirectory. Although it works, only one of them is needed as it turns out. Sorry I had to scratch the information together. With only RuntimeDirectory you wil have a volatile directory /var/run/stubby. When the service quits it removes all, including the directory. Strong advice: if present first remove a leftover /var/run/stubby directory from another install, since it might have the wrong permissions set. Systemd will now take full care of folder and file creation and their permissions. You need to set appdata_dir: /var/run/stubby in stubby.yml. With only StateDirectory you will have a persistent directory, so after a reboot it keeps the Zero configuration DNSSEC files intact. You need to set appdata_dir: /var/lib/stubby in stubby.yml. More information of the benefits of a DynamicUser here: "http://0pointer.net/blog/dynamic-users-with-systemd.html" BTW my stubby.yml used for testing is very basic: (This one is for use with StateDirectory=stubby ) [I edited my previous stubby.service above] |
|
Prior to using DynamicUser it is important that any existing user stubby and group stubby have to be removed also. After removing user:stubby and group:stubby and a reboot all was fine again: |
|
@hanvinke Thanks for pointing out the DynamicUser configuration of systemd! I like it a lot! I believe the error you had could have been prevented when a I'll play with these settings a bit and will include it in the getdns 1.4.2-rc1 release candidate today (which will have a stubby 0.2.3-rc1 release candidate on board). |
To improve integration with system and service managers like systemd See also getdnsapi/stubby#106
|
Testing right now the new release candidate 😃 ! $ stubby -i So I used only the default enabled DNS recursive servers in stubby.yml, and that gave no errors. Buffer overflow of some kind? |
|
Ouch! That's really bad. I just tried with around 500000. It was really slow to parse, but it did. |
printing certain configuration. Thanks Han Vinke
|
Sorry to be the bearer of bad news, but unfortunately, after updating to latest getdns 1.4.2 with the latest commit e0e8576 for the stubby 0.2.3 submodule, this problem has resurfaced for me. As far as I can tell, I am now using the new default options regarding systemd and the working_app_dir. In my stubby.service file I have I also succesfully have the following in my /usr/lib/tmpfiles.d/stubby.conf file: However, it's exact same issue as before, with dnssec failing when run as the systemd service, but it's successful when the binary is run as sudo. Is it to do with it now using /var/cache/stubby? Your advice before was to use /var/run/stubby. I will probably change it back to this to see if that works, but ideally I wanted to use the defaults as much as possible. |
|
@eccgecko On your system what is |
|
Can you paste your full |
|
Also delete the contents of /var/cache/private/stubby when retesting Zero configuration DNSSEC, since the folder /var/cache/stubby is just a symlink (owned by root) to /var/cache/private/stubby. |
|
ls -ld /var/cache outputs the following: drwxr-xr-x 11 root root 4096 May 17 17:14 /var/cache My stubby.service file looks like this:
This is the default stubby.service file. The only change I made was to add -C /etc/stubby.yml to the ExecStart line. I did as you said @ArchangeGabriel and deleted both One thing I did notice is that WorkingDirectory=/var/cache/stubby and appdata_dir: "/var/cache/stubby" are different to what @wtoorop recommended before (/var/run/stubby). Could that be related? I want to keep it as close to default config as possible so haven't changed this yet. @hanvinke my system doesn't seem to have any /var/cache/private/ directory at all. Could that also be related? |
|
@eccgecko What systemd version? Doesn’t give any outputs in status on failure to start Stubby? |
|
@eccgecko |
|
@ArchangeGabriel ah...I guess that's the issue. I'm on the default Raspbian systemd package, which at present is 232, unfortunately. I see from the blog @hanvinke referenced, DynamicUser was introduced in 235. I did try upgrading my systemd package by downloading the 238 package from the buster repo, but unfortunately this broke my system and I had to restore from a backup. I guess it's just a case of removing DynamicUser from the stubby.service file? |
|
Indeed. Actually DynamicUser was introduced in 232, but Stubby uses related features introduced in 235. So yes, in this case you have to use the |
|
Thanks @ArchangeGabriel @hanvinke for your help. I removed the Not wanting to push my luck, so apologies if this is the wrong place, but just wanted to ask one additional question relating to stubby and DNSSEC. Is there a reason why, when running a DNSSEC algorithm test here https://rootcanary.org/test.html, ED25519 is not validated as an algorithm, but when using dnsmasq with DNSSEC, it is?
|
|
@eccgecko For the enthusiasts I have made the stubby service file optimized for security. This is because f.i. the ReadWritePaths= was not added to the original file. My stubby.service: |
|
Well I’m not sure… ED25519 is supported for the TLS connection, but maybe not for DS signing. |
|
@hanvinke If you can put a PR with comments on each added line, that would be very welcomed I think. |
|
@ArchangeGabriel I edited my previous file a little. I also changed Umask setting to 077, and ProtectHome to yes, which are much more restrictive. |
|
Edited a third time - ptrace: Already is blocked by dropping CAP_PTRACE under CapabilityBoundingSet AmbientCapabilities=CAP_NET_BIND_SERVICE can only be emitted when you do not online banking, otherwise f.i. payment transactions with Ideal will fail without it. For now I am just keeping SystemCallFilter= ~madvise |
|
Thank you all! @ArchangeGabriel you say systemd 232 will not start the service if it encounters an for it unkown (i.e. I suppose we have to provide two stubby.service files (one for systemd before 235 and one for systemd 235 and higher), but perhaps there was something else going on... @eccgecko I believe most of the groundwork for ED25519 has already been done. I'll see if I can enable the ED25519 and ED448 with newer OpenSSL and let you know (provide patch). |
|
@wtoorop No, I’m saying that it handled the |
|
@ArchangeGabriel Ok, so the |
|
No, I think a DynamicUser still doesn’t have the right to write to standard directory even if it owns it because the user is in fact not dynamic and the directory attributed to it. But I might be wrong, and this could also be a bug of some kind. In any case, problematic systems in this regard would be the one with |
|
@wtoorop it may be necessary to provide 2 different stubby.service files, as, with Stretch, the current stable dist of Debian / Raspbian, only ships systemd 232, so it may be necessary to make some allowances for that user-base. |
|
While testing logging with Stubby through 'stubby -v 7' I noticed that dns-tls.bitwiseshift.net has a problem currently, stubby nicely reporting: |
|
@hanvinke I've pinged the operator directly as they haven't made their contact details public. To double check for issues you can find monitoring of the servers here: |
Package changes: * PLIST adjustment; stubby no longer built by default Upstream changes: * 2018-12-21: Version 1.5.0 * RFE getdnsapi/stubby#121 log re-instantiating TLS upstreams (because they reached tls_backoff_time) at log level 4 (WARNING) * GETDNS_RESPSTATUS_NO_NAME for NODATA answers too * ZONEMD rr-type * getdns_query queries for addresses when a query name without a type is given. * RFE #408: Fetching of trust anchors will be retried after failure, after a certain backoff time. The time can be configured with getdns_context_set_trust_anchors_backoff_time(). * RFE #408: A "dnssec" extension that requires DNSSEC verification. When this extension is set, Indeterminate DNSSEC status will not be returned. * Issue #410: Unspecified ownership of get_api_information() * Fix for DNSSEC bug in finding most specific key when trust anchor proves non-existance of one of the labels along the authentication chain other than the non- existance of a DS record on a zonecut. * Enhancement getdnsapi/stubby#56 & getdnsapi/stubby#130: Configurable minimum and maximum TLS versions with getdns_context_set_tls_min_version() and getdns_context_set_tls_max_version() functions and tls_min_version and tls_max_version configuration parameters for upstreams. * Configurable TLS1.3 ciphersuites with the getdns_context_set_tls_ciphersuites() function and tls_ciphersuites config parameter for upstreams. * Bugfix in upstream string configurations: tls_cipher_list and tls_curve_list * Bugfix finding signer for validating NSEC and NSEC3s, which caused trouble with the partly tracing DNSSEC from the root up, introduced in 1.4.2. Thanks Philip Homburg * 2018-05-11: Version 1.4.2 * Bugfix getdnsapi/stubby#87: Detect and ignore duplicate certs in the Windows root CA store. * PR #397: No TCP sendto without TCP_FASTOPEN Thanks Emery Hemingway * Bugfix getdnsapi/stubby#106: Core dump when printing certain configuration. Thanks Han Vinke * Bugfix getdnsapi/stubby#99: Partly trace DNSSEC from the root up (for tld and sld), to find insecure delegations quicker. Thanks UniverseXXX * Bugfix: Allow NSEC spans starting from (unexpanded) wildcards Bug was introduced when dealing with CVE-2017-15105 * Bugfix getdnsapi/stubby#46: Don't assume trailing zero with string bindata's. Thanks Lonnie Abelbeck * Bugfix #394: Update src/compat/getentropy_linux.c in order to handle ENOSYS (not implemented) fallback. Thanks Brent Blood * Bugfix #395: Clarify that libidn2 dependency is for version 2.0.0 or higher. Thanks mire3212 * 2018-03-12: Version 1.4.1 * Bugfix #388: Prevent fallback to an earlier tries upstream within a single query. Thanks Robert Groenenberg * PR #387: Compile with OpenSSL with deprecated APIs disabled. Thanks Rosen Penev * PR #386: UDP failover improvements: - When all UDP upstreams fail, retry them (more or less) equally - Limit maximum UDP backoff (default to 1000) This is configurable with the --with-max-udp-backoff configure option. Thanks Robert Groenenberg * Bugfix: Find zonecut with DS queries (instead of SOA queries). Thanks Elmer Lastdrager * Bugfix #385: Verifying insecure NODATA answers (broken since 1.2.1). Thanks hanvinke * PR #384: Fix minor spelling and formatting. Thanks dkg. * Bugfix #382: Parallel install of getdns_query and getdns_server_mon * 2018-02-21: Version 1.4.0 * .so revision bump to please fedora packaging system. Thanks Paul Wouters * Specify the supported curves with getdns_context_set_tls_curves_list() An upstream specific list of supported curves may also be given with the tls_curves_list setting in the upstream dict with getdns_context_set_upstream_recursive_servers() * New tool getdns_server_mon for checking upstream recursive resolver's capabilities. * Improved handling of opportunistic back-off. If other transports are working, don't forcibly promote failed upstreams just wait for the re-try timer. * Hostname authentication with libressl Thanks Norbert Copones * Security bugfix in response to CVE-2017-15105. Although getdns was not vulnerable for this specific issue, as a precaution code has been adapted so that signatures of DNSKEYs, DSs, NSECs and NSEC3s can not be wildcard expansions when used with DNSSEC proofs. Only direct queries for those types are allowed to be wildcard expansions. * Bugfix PR#379: Miscelleneous double free or corruption, and corrupted memory double linked list detected issue, with serving functionality. Thanks maddie and Bruno Pagani * Security Bugfix PR#293: Check sha256 pinset's with OpenSSL native DANE functions for OpenSSL >= 1.1.0 with Viktor Dukhovni's danessl library for OpenSSL >= 1.0.0 don't allow for authentication exceptions (like self-signed certificates) otherwise. Thanks Viktor Dukhovni * libidn2 support. Thanks Paul Wouters * 2017-12-21: Version 1.3.0 * Bugfix #300: Detect dnsmasq and skip unit test that fails with it. Thanks Tim Rohsen and Konomi Kitten * Specify default available cipher suites for authenticated TLS upstreams with getdns_context_set_tls_ciphers_list() An upstream specific available cipher suite may also be given with the tls_cipher_list setting in the upstream dict with getdns_context_set_upstream_recursive_servers() * PR #366: Add support for TLS 1.3 and Chacha20-Poly1305 Thanks Pascal Ernster * Bugfix #356: Do Zero configuration DNSSEC meta queries over on the context configured upstreams. Thanks Andreas Schulze * Report default extension settings with getdns_context_get_api_information() * Specify locations at which CA certificates for verification purposes are located: getdns_context_set_tls_ca_path() getdns_context_set_tls_ca_file() * getdns_context_set_resolvconf() function to initialize a context upstreams and suffices with a resolv.conf file. getdns_context_get_resolvconf() to get the file used to initialize the context's upstreams and suffixes. getdns_context_set_hosts() function to initialize a context's LOCALNAMES namespace. getdns_context_get_hosts() function to get the file used to initialize the context's LOCALNAMES namespace. * get which version of OpenSSL was used at build time and at run time when available with getdns_context_get_api_information() * GETDNS_RETURN_IO_ERROR return error code * Bugfix #359: edns_client_subnet_private should set family Thanks Daniel Areiza & Andreas Schulze * Bugfix getdnsapi/stubby#34: Segfault issue with native DNSSEC validation. Thanks Bruno Pagani * 2017-11-11: Version 1.2.1 * Handle more I/O error cases. Also, when an I/O error does occur, never stop listening (with servers), and never exit (when running the built-in event loop). * Bugfix: Tolerate unsigned and unused RRsets in the authority section. Fixes DNSSEC with BIND upstream. * Bugfix: DNSSEC validation without support records * Bugfix: Validation of full recursive DNSKEY lookups * Bugfix: Retry to validate full recursion BOGUS replies with zero configuration DNSSEC only when DNSSEC was actually requested * Bugfix #348: Fix a linking issue in stubby when libbsd is present Thanks Remi Gacogne * More robust scheduling; Eliminating a segfault with long running applications. * Miscellaneous Windows portability fixes from Jim Hague. * Fix Makefile dependencies for parallel install. Thanks ilovezfs * 2017-09-29: Version 1.2.0 * Bugfix of rc1: authentication of first query with TLS Thanks Travis Burtrum * A function to set the location for library specific data, like trust-anchors: getdns_context_set_appdata(). * Zero configuration DNSSEC - build upon the scheme described in RFC7958. The URL from which to fetch the trust anchor, the verification CA and email can be set with the new getdns_context_set_trust_anchor_url(), getdns_context_set_trust_anchor_verify_CA() and getdns_context_set_trust_anchor_verify_email() functions. The default values are to fetch from IANA and to validate with the ICANN CA. * Update of Stubby with yaml configuration file and logging from a certain severity support. * Fix tpkg exit status on test failure. Thanks Jim Hague. * Refined logging levels for upstream statistics * Reuse (best behaving) backed-off TLS upstreams when non are usable. * Let TLS upstreams back-off a incremental amount of time. Back-off time starts with 1 second and is doubled each failure, but will not exceed the time given by getdns_context_set_tls_backoff_time() * Make TLS upstream management more resilient to temporary outages (like laptop sleeps) * 2017-09-04: Version 1.1.3 * Small bugfixes that came out of static analysis * No annotations with the output of getdns_query anymore, unless -V option is given to increase verbosity Thanks Ollivier Robert * getdns_query will now exit with failure status if replies are BOGUS * Bugfix: dnssec_return_validation_chain now also works when fallback to full recursion was needed with dnssec_roadblock_avoidance * More clear build instructions from Paul Hoffman. Thanks. * Bugfix #320.1: Eliminate multiple closing of file descriptors Thanks Neil Cook * Bugfix #320.2: Array bounds bug in upstream_select Thanks Neil Cook * Bugfix #318: getdnsapi/getdns/README.md links to nonexistent wiki pages. Thanks James Raftery * Bugfix #322: MacOS 10.10 (Yosemite) provides TCP fastopen interface but does not have it implemented. Thanks Joel Purra * Compile without Stubby by default. Stubby now has a git repository of its own. The new Stubby repository is added as a submodule. Stubby will still be build alongside getdns with the --with-stubby configure option. * 2017-07-03: Version 1.1.2 * Bugfix for parallel make install * Bugfix to trigger event callbacks on socket errors * A getdns_context_set_logfunc() function with which one may register a callback log function for certain library subsystems at certain levels. Currently this can only be used for upstream stastistics subsystem. * 2017-06-15: Version 1.1.1 * Bugfix #306 hanging/segfaulting on certain (IPv6) upstream failures * Spelling fix s/receive/receive. Thanks Andreas Schulze. * Added stubby-setdns-macos.sh script to support Homebrew formula * Include stubby.conf in the districution tarball * Bugfix #286 reschedule reused listening addresses * Bugfix #166 Allow parallel builds and unit-tests * NSAP-PTR, EID and NIMLOC, TALINK, AVC support * Bugfix of TA RR type * OPENPGPKEY and SMIMEA support * Bugfix TAG rdata type presentation format for CAA RR type * Bugfix Zero sized gateways with IPSECKEY gateway_type 0 * Guidance for integration with systemd * Also check for memory leaks with advances server capabilities. * Bugfix convert IP string to IP dict with getdns_str2dict() directly. ok'ed by root@zta.lk
|
When DNSSEC not working, also check system date. |
I have a strange issue that when I run the stubby daemon manually, DNSSEC seems to be working ok. For example the command
dig @127.0.2.2 -p 5353 www.dnssec-failed.orgreturns the following:; <<>> DiG 9.10.3-P4-Raspbian <<>> @127.0.2.2 -p 5353 +dnssec www.dnssec-failed.org ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 24774 ;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0 ;; WARNING: recursion requested but not available ;; QUESTION SECTION: ;www.dnssec-failed.org. IN A ;; Query time: 129 msec ;; SERVER: 127.0.2.2#5353(127.0.2.2) ;; WHEN: Sat Apr 28 12:22:10 CEST 2018 ;; MSG SIZE rcvd: 39so dnssec-failed.org doesn't resolve. However, once I quit the manual daemon, and start the systemd stubby.service I have, which starts up ok, I now get a reply from dnssec-failed.org:
; <<>> DiG 9.10.3-P4-Raspbian <<>> @127.0.2.2 -p 5353 www.dnssec-failed.org ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16532 ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags: do; udp: 1536 ; OPT=12: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 (".............................................................................................................................................................................................................") ;; QUESTION SECTION: ;www.dnssec-failed.org. IN A ;; ANSWER SECTION: www.dnssec-failed.org. 2325 IN A 68.87.109.242 www.dnssec-failed.org. 2325 IN A 69.252.193.191 www.dnssec-failed.org. 2325 IN RRSIG A 5 3 7200 20180430172414 20180423141914 44973 dnssec-failed.org. w7tdNJ/YrlNO30y2GuPSJ31388GnzrPrHgJw4vQijlsL5LgkTTg5hzJw Ox5Ra2xSjlLdR7JeA4ZXvKF9rzws+8ys+EFJyps0+KejonIELKuLIqEw b9QS4ITc3mii4hFqVOwMtxj7txv6lKngknqbxiFr2nCpyJX0SOo6UXye YsI= ;; Query time: 167 msec ;; SERVER: 127.0.2.2#5353(127.0.2.2) ;; WHEN: Sat Apr 28 12:29:53 CEST 2018 ;; MSG SIZE rcvd: 531This is strange, as when I run the daemon manually I am using the exact same options as the stubby.service file uses, so I can't work out why it would behave like this.
I have zero-configuration DNSSEC enabled in the stubby.yml config file
The text was updated successfully, but these errors were encountered: