Evaluating Local DNSSEC Validators

Domain Name System Security Extensions (DNSSEC) is a technology that uses cryptographic signatures to make the Domain Name System (DNS) tamper-proof, safeguarding against DNS hijacking. If your ISP or network operator cares about your online security, their DNS servers will validate DNSSEC signatures for you. DNSSEC is widely deployed: here in Scandinavia, about 80% of all DNS lookups are subject to DNSSEC validation (source). Wondering whether or not your DNS server validates DNSSEC signatures? www.dnssec-or-not.com will tell you.

However, even if your network operator’s DNS server claims to validate DNSSEC signatures, it is possible for that to be a lie; several ISPs hijack DNS on purpose. Some ISPs are ordered to do this by their respective governments in order to block certain types of content, while others simply want replace the web browser’s default «web site not found» error page with an ad-filled one of their own.

Another worry, especially on public wireless networks, is that the DNS server could be controlled by a malicious party, redirecting users to fake replicas of real web sites set up to steal login credentials and/or credit card details.

To alleviate concerns about the of the network-assigned DNS server being trustworthy, your local machine should independently verify the DNSSEC signatures. This is especially important if you are using DNSSEC to secure other protocols, like for example using SSHFP DNS records to validate SSH host key fingerprints.

In this article I will demonstrate how to install and configure six common local DNSSEC validators for Linux, and evaluate how well they perform.

About the Evaluation

In each Quick Start sections, I will demonstrate how to install and configure each validating resolver for strict DNSSEC validation, and how to configure the operating system to make use of it.

Unless otherwise noted, all the commands have been tested to work on both Fedora 30 and Ubuntu 19.04 live images. Most of them require root privilege.

I used each resolving validator for at least a day in order to increase my chances of discovering any unexpected behaviour.

Opportunistic and Strict Validation

Some of the validating resolvers can perform opportunistic DNSSEC validation. This means that if the upstream DNS server is incompatible with DNSSEC (e.g., if it refuses to answer queries for DNS records containing DNSSEC signatures), the resolver automatically falls back on non-validating mode.

In contrast, a strict validator will in this situation simply refuse to answer any queries, since it cannot guarantee their authenticity.

Opportunistic mode compromises on security by exposing the DNS lookups to a downgrade attack. While malicious parties can not make invalid data appear as valid by forging DNSSEC signatures, they can instead trick opportunistic validators into disabling DNSSEC by simply withholding the signatures.

If you care enough about DNS security to want a local validating resolver, you will also want to use strict mode. Each of the Quick Start sections will therefore ascertain that strict mode is active.

If you connect to a network with a DNSSEC-incompatible DNS server, strict mode will cause all DNS lookups to fail. To resolve this issue, you can disable the use of the DNS server address assigned by the network and instead use a public DNS service known to support DNSSEC. This can be done in the network settings, under the IPv4 and IPv6 tabs.

Well-known examples of public DNS services include Cloudflare’s 1.1.1.1, Google’s 8.8.8.8 and Quad9’s 9.9.9.9.

Captive Portals

Strict mode DNSSEC validation is incompatible with some (but not all) types of captive portals.

These captive portals typically provide a DNS server which answers all DNS queries with the IP address of the captive portal. After you have successfully logged in to the network, the DNS server starts answering queries normally.

This is of course exactly the kind of falsification of DNS data that DNSSEC is meant to protect against, but it does prevent you from accessing the captive portal and successfully logging in to the network.

To overcome this problem, you will probably need to temporarily disable DNSSEC validation in order to allow the captive portal to load. Once you have logged in, you can re-enable validation. The initial validation failures might also be cached, making it necessary to clear the validating resolver’s cache. I will demonstrate how this is done.

NetworkManager Integration

If available, NetworkManager integration will be enabled, so that the validating resolver automatically uses the DNS server assigned by the network operator.

If there is no ready-made NetworkManager integration, I will instead explicitly configure Cloudflare’s 1.1.1.1 as the upstream DNS server. I chose 1.1.1.1 because is the only public DNS service that I know of that has a node here in Norway. You may of course replace it with your preferred upstream DNS server.

Recursive Mode

If the validating resolver supports recursive mode, you can also opt not to configure any fixed upstream DNS server at all. The validating resolver will then perform recursive queries instead of the regular forwarded queries.

Recursive queries starts out with a query to one of the 13 root name servers and from there follow the delegation chain. Eventually it will locate the authoritative name server for the domain name being looked up, and will proceed to query it directly.

Recursive queries will normally cause more DNS traffic and take longer to complete than normal forwarded queries (which will usually be answered from cache), so you should be aware that recursive mode will most likely degrade the performance somewhat.

Private Domains and Negative Trust Anchors

Home gateways and companies sometimes use private domains like .lan or .company.internal. Since such domains do not officially exist, they will fail DNSSEC validation. That is inconvenient.

To make such private domains work, it is necessary to add Negative Trust Anchors (NTAs) for them. An NTA instructs the validating resolver to disable DNSSEC processing for the domain in question.

It is also necessary to ensure that lookups for the private domains are sent to correct DNS server, even though this server may not be the default upstream server, as the default upstream server might not be aware of the private domain.

In summary, my use case requires the following behaviour, which I will attempt to configure if possible:

  • When I am connected to my wireless network at home, DNS queries for the private .lan domain must be routed to my OpenWRT home gateway at 192.168.1.1.
  • The .lan domain should continue to work even though I log on to work via VPN (which changes NetworkManager’s default upstream DNS server to one reached through the VPN tunnel).
  • The .lan domain must be covered by an NTA.
  • Other connection-provided domains should not be covered by an NTA, for two reasons:
    1. I would like SSHFP validation to work for servers found within the connection-provided domains when I am at the office or connected to VPN.
    2. I do not want to be exposed to a downgrade attack where a rogue network operator can easily disable DNSSEC validation for all domains within .com, .net and so on by simply advertising them as search domains in DHCP.

DNSSEC Lookup Testing

Using the dig tool, we can determine the DNSSEC validation state for any given DNS lookup. The DNSSEC standard defines four possible states, but while using a correctly configured validating resolver you will only encounter three of them: Secure, Insecure and Bogus.

To confirm that the local validator determines the correct validation state for a given domain name, there are two very good test sites I can recommend: DNSSEC Analyzer and DNSViz.

Secure

The Secure state simply means that everything checks out. The owner of the domain being looked up has signed the data in DNS, and the validating resolver has verified that the signatures match. The answer given is thus guaranteed to be correct and not tampered with.

The Secure state is indicated by the presence of the ad flag (short for «authenticated data») in the DNS response packet. For example:

$ dig +dnssec toreanderson.no | grep -A1 HEADER
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48570
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

Insecure

The Insecure state means that the answer given is not covered by DNSSEC signatures, and that this situation was expected - typically a valid (Secure) signed proof was found at a higher level in the DNS hierarchy that said that this particular domain has not yet secured its DNS data with DNSSEC.

In the following example, the .org zone contains Secure statement telling the validating resolver to not expect the .gnu.org zone to contain any DNSSEC signatures. This is here indicated by the absence of the ad flag, in combination with a normal status code like NOERROR:

$ dig +dnssec gnu.org | grep -A1 HEADER
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2227
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

The data might have been tampered with, but due to the lack of DNSSEC signatures, there is simply no way of knowing whether or not that has happened.

Note that a private domain covered by an NTA is by definition Insecure.

Bogus

The Bogus state means that something is rotten in the state of Denmark. Somehow, the DNSSEC signatures failed to verify. This indicates either an attempt to tamper with DNS data, or that a domain is incorrectly signed.

In this example, the .org zone contains a Secure statement that the data in dnssec-failed.org. should be signed with certain key. However, it is not - the data is signed with a different key than expected. This failure is being indicated with the status code SERVFAIL:

$ dig +dnssec dnssec-failed.org | grep -A1 HEADER
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 1351
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

Note that SERVFAIL is an unspecified error code, so it could be caused by other error conditions unrelated to DNSSEC too. One way to determine that is to retry the query while setting the Checking Disabled (CD) flag, which instructs the resolver to not perform DNSSEC validation. If it succeeds, it is reasonable to assume that the previous SERVFAIL was caused by a DNSSEC validation failure. For example:

$ dig +cdflag dnssec-failed.org | grep -A1 HEADER
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21865
;; flags: qr rd ra cd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

In order to get more information about why a lookup resulted in a Bogus verdict, it is usually necessary to consult the validating resolver’s log file. My Quick Start sections will enable such logging, if it is not already on by default.

Integrity, not Confidentiality

DNSSEC provides integrity. In other words, if a DNS query results in a DNSSEC validation state of Secure, you can be certain that the answer is correct and has not been tampered with.

It is important to note, however, that DNSSEC does not provide confidentiality. The DNS queries and answers are transmitted in unencrypted form, so someone with the ability to eavesdrop on your network traffic (which, in the case of an open wireless network, is everyone in your vicinity) will be able to see which domain names you are looking up and thus deduce which web sites you are visiting.

Other DNS extensions provide confidentiality, including DNSCrypt, DNSCurve, DNS over HTTPS (DoH) and DNS over TLS (DoT). They can all co-exist with DNSSEC. I will not evaluate any of these extensions beyond noting whether or not the validating resolvers claim to support one or more of them.

Evaluation results

systemd-resolved

systemd-resolved is a stub resolver that is part of systemd. It was written primarily by Red Hat’s Lennart Poettering.

Key points:

  • It is installed by default on both Fedora and Ubuntu.
  • It does not perform strict DNSSEC validation by default.
  • It integrates with NetworkManager, so the network-assigned DNS servers will be used automatically.
  • It is a stub resolver, meaning it does not support recursive mode.
  • It supports DNS over TLS for confidentiality, but is susceptible to downgrade attacks as only opportunistic mode is supported.

Quick Start

# Enable systemd-resolved
systemctl enable systemd-resolved.service

# Enable strict DNSSEC validation
sed -Ei "s|.*(DNSSEC)=.*|\1=yes|" /etc/systemd/resolved.conf

# Example: Create NTA for private '.priv' domain
#mkdir /etc/dnssec-trust-anchors.d
#echo priv > /etc/dnssec-trust-anchors.d/priv.negative

# Start (or restart) systemd-resolved to activate changes
systemctl restart systemd-resolved.service

# Configure system to use the local systemd-resolved
ln -fsv /usr/lib/systemd/resolv.conf /etc

Operation

systemd-resolved comes with a command line tool called resolvectl. When run without arguments, it will display the state of systemd-resolved, including DNSSEC information like the operational mode, the list of NTAs, and whether or not it believes its upstream DNS servers are capable of DNSSEC and other advanced features.

resolvectl query example.org will make systemd-resolved look up example.org, and display which upstream DNS server was used and on which link, and whether or not the the result was Secure according to DNSSEC validation. This is an alternative to the dig tool.

The resolvectl tool can also be used to modify various parameters of systemd-resolved while it is running. Refer to the manual page for more information.

To inspect systemd-resolved logs, use journalctl -u systemd-resolved.

Private Domains and NTAs

systemd-resolved comes with a default hard-coded NTA list containing a collection of commonly used private domains. My private domain .lan was on this default list, so I did not have to configure anything to make it work.

(The Quick Start section above contains an commented out example of how to add an NTA for another private domain not included in the default NTA list.)

The NetworkManager integration ensures that queries for the network-assigned domains are explicitly routed to the correct upstream DNS server, meaning .lan continues to work even if I log on to VPN.

Toggling DNSSEC Validation

To temporarily turn local DNSSEC validation off, enabling you to log in to DNS-hijacking captive portals, run:

# Turn DNSSEC validation off by using upstream resolvers directly:
ln -fsv /run/systemd/resolve/resolv.conf /etc

# Turn it back on again and flush any cached Bogus validation verdicts:
ln -fsv /usr/lib/systemd/resolv.conf /etc
resolvectl flush-caches

Problematic Observations

  • systemd-resolved suffers from a fundamental design flaw that causes it to frequently flag its upstream DNS server as being incompatible with DNSSEC (even though it is not). When this happens, all DNS lookups will fail (until the flag is manually cleared with resolvectl reset-server-features). Different variations of this issue are reported in systemd bugs 6490, 8451, 9384 and 11171.
  • systemd-resolved will in some cases return an incorrect Bogus verdict for lookups that should have been Insecure. Some domains (e.g., savannah.gnu.org) give the wrong verdict 100% of the time, while others (e.g., ring.nlnog.net) just fails sometimes. See bugs 9867 and 12545 for more information.

Conclusion

The bugs described above do in my opinion make systemd-resolved unfit for the role of a local validating resolver. That is very unfortunate, because I think that it is otherwise a very nice piece of software.

While the first of the two bugs can be worked around, the second one can not and it simply gets too annoying to live with it in the long run.

When these bugs are fixed, systemd-resolved will likely be my validating resolver of choice. However, I am not holding my breath - some of these bugs have been open for over two years while only getting cursory attention from its maintainers.

Dnsmasq

Dnsmasq is a caching stub resolver written by Simon Kelley. Key points:

  • It is installed by default on both Fedora and Ubuntu.
  • It does not perform DNSSEC validation by default.
  • It integrates with NetworkManager, so the network-assigned DNS servers will be used automatically.
  • It is a stub resolver, meaning it does not support recursive mode.
  • It can also perform several non-DNS related tasks, like be a DHCP server.

Quick Start

# Enable strict DNSSEC validation
echo dnssec > /etc/NetworkManager/dnsmasq.d/dnssec.conf
echo conf-file = /usr/share/dnsmasq*/trust-anchors.conf >> /etc/NetworkManager/dnsmasq.d/dnssec.conf

# Revert Ubuntu DNSSEC-breaking patch that sets the default cache-size to 0 (Ubuntu only)
echo cache-size = 400 >> /etc/NetworkManager/dnsmasq.d/dnssec.conf

# Configure system to use the local Dnsmasq
echo -e "[main]\ndns=dnsmasq" > /etc/NetworkManager/conf.d/dns.conf
sudo systemctl reload NetworkManager.service

Operation

Dnsmasq is invoked directly by NetworkManager, and configured using D-Bus. This interface is not intended for direct use by humans, so I have not attempted to explore it.

To determine which upstream DNS servers are used, run pkill -USR1 -x dnsmasq and check its logs with journalctl -u NetworkManager _COMM=dnsmasq.

It is also possible to query NetworkManager directly using nmcli | less -p DNS (this seems to be the only way to determine which search domains are in use).

To make Dnsmasq clear its cache, use pkill -HUP -x dnsmasq.

I was not able to figure out a way of making Dnsmasq log DNSSEC validation failures, short of enabling logging of all queries with the log-queries option.

Private Domains and NTAs

The NetworkManager integration will automatically configure explicit forwarding of connection-provided search domains to the DNS server learned from the same connection. Dnsmasq will in turn automatically create NTAs for all of those domains.

This is useful when I am at home and the connection-provided search domain is the private .lan. However it is counter-productive when I am at the office or logged on to the VPN and the search domain is .redpill-linpro.com. I do want DNSSEC validation of the work domain, but the automatic NTA disables it.

The automatic addition of NTAs for connection-provided search domains is a also a security risk, as described in the introduction. Unfortunately, I found no way to easily disable this behaviour.

Toggling DNSSEC Validation

To temporarily turn local DNSSEC validation off, enabling you to log in to DNS-hijacking captive portals, run:

# Disable DNSSEC validation by using upstream DNS servers directly
cat /var/run/NetworkManager/no-stub-resolv.conf > /etc/resolv.conf

# Turn DNSSEC back on again and get rid of any cached *Bogus* responses:
systemctl reload NetworkManager.service

Problematic Observations

  • If the upstream DNS server omits some optional DNSSEC signatures in its answer, Dnsmasq would incorrectly yield a Bogus verdict instead of Insecure. It logged the error message Insecure DS reply received, do upstream DNS servers support DNSSEC?. I reported this bug upstream, and it was later fixed.
  • If the domain name queried for was an Insecure CNAME that pointed to another Secure domain name, Dnsmasq would in some circumstances incorrectly yield a Bogus verdict instead of Insecure. Bug report, fix.
  • During debugging the previous issue, I found a another bug that caused queries for Insecure domain names to fail after looping 50 upstream queries, all failing with the error message reply <domain> is no DS. Bug report, fix.
  • When running as non-root, Dnsmasq would refuse to forward TCP queries to name servers specified with an outgoing interface (which is how NetworkManager configures Dnsmasq). While this was not a DNSSEC issue per se, it caused DNSSEC-enabled queries to fail. This is because answer size was increased by the DNSSEC signatures, thus necessitating the use of TCP queries. Bug report, attempted fix, fixed fix.

  • NetworkManager invokes Dnsmasq with the --proxy-dnssec command line parameter. This is hard-coded. This parameter causes Dnsmasq to blindly trust the upstream resolver by proxying the ad flag from the upstream responses. Fortunately, it would appear that the dnssec setting in the configuration file takes precedence over the --proxy-dnssec argument from the command line.

  • NetworkManager does not add options edns0 to the /etc/resolv.conf file it generates (bug report).

    This prevents other software from determining whether or not a DNS response was Secure or Insecure, breaking OpenSSH’s ValidateHostKeysDNS feature. It can be worked around, e.g., by adding export RES_OPTIONS=edns0 to your ~/.bashrc file (or similar).

  • Ubuntu patches NetworkManager to invoke Dnsmasq with the --cache-size=0 command line parameter. Dnsmasq refuses to perform DNSSEC validation with a cache size of 0, so this patch breaks DNSSEC. Fortunately, setting cache-size in the configuration file appears to take precedence over the --cache-size command line argument.

Conclusion

The current Dnsmasq release at the time of writing (v2.80) contains far too many bugs to be usable with DNSSEC validation enabled. The good news is that all of the bugs I found are fixed in Git master.

The insecure coupling between the NTA list and the network-provided domain search list is also rather unfortunate. It breaks SSHFP validation for work servers and exposes me to downgrade attacks.

For these reasons I am unable to recommend using Dnsmasq as a validating resolver, unless you are willing to run a development snapshot.

Unbound/DNSSEC-Trigger

Unbound is a caching recursive resolver developed at NLNet Labs. Key points:

  • It performs strict DNSSEC validation by default.
  • The companion daemon DNSSEC-Trigger provides NetworkManager integration, ensuring that the network-assigned DNS servers are automatically used.
  • DNSSEC-Trigger also provides captive portal detection and a simple GUI front-end that allows for easy disabling of DNSSEC validation when necessary.
  • Unbound supports DNS over TLS for confidentiality (but DNSSEC-Trigger will not make use of it)

Quick Start

Unbound+DNSSEC-Trigger
# Install and enable Unbound and DNSSEC-Trigger
pkcon -y install unbound dnssec-trigger
systemctl enable unbound.service dnssec-triggerd.service

# Allow DNSSEC-Trigger to control Unbound (Ubuntu only)
dnssec-trigger-control-setup -i

# Prevent creation of automatic NTAs for network-provided domains
sed -Ei "s|^(validate_connection_provided_zones)=.*|\1=yes|" /etc/{dnssec-trigger,}/dnssec.conf
# Ensure network-provided domains are respected even on wireless networks
sed -Ei "s|^(add_wifi_provided_zones)=.*|\1=yes|" /etc/{dnssec-trigger,}/dnssec.conf

# Start (or restart) Unbound to activate changes
sudo systemctl restart unbound.service

# Create a NetworkManager dispatcher script to add an NTA for '.lan' and run it
echo "#!/bin/sh" > /etc/NetworkManager/dispatcher.d/02-unbound-nta
echo unbound-control insecure_add lan >> /etc/NetworkManager/dispatcher.d/02-unbound-nta
echo unbound-control flush_zone lan >> /etc/NetworkManager/dispatcher.d/02-unbound-nta
chmod a+x /etc/NetworkManager/dispatcher.d/02-unbound-nta
/etc/NetworkManager/dispatcher.d/02-unbound-nta

# Ensure NetworkManager does not manage /etc/resolv.conf (DNSSEC-Trigger does that)
echo -e "[main]\ndns=none" > /etc/NetworkManager/conf.d/dns.conf
systemctl reload NetworkManager.service
Unbound (Standalone)
# Install and enable Unbound
pkcon -y install unbound
systemctl enable unbound.service

# Enable forwarding of private '.lan' queries with NTA
echo -e "forward-zone:\n name: lan\n forward-addr: 192.168.1.1" >> /etc/unbound/unbound.conf
echo -e "server:\n domain-insecure: lan" >> /etc/unbound/unbound.conf

# Enable stub mode (forwarding of all queries)
echo -e "forward-zone:\n name: .\n forward-addr: 1.1.1.1" >> /etc/unbound/unbound.conf

# Enable logging of failed DNSSEC validations (Ubuntu only)
echo -e "server:\n val-log-level: 1" >> /etc/unbound/unbound.conf

# Start (or restart) Unbound to activate changes
sudo systemctl restart unbound.service

# Configure system to use the local Unbound
echo -e "[main]\ndns=none" > /etc/NetworkManager/conf.d/dns.conf
systemctl reload NetworkManager.service
find /etc/resolv.conf -type l -delete
echo -e "nameserver 127.0.0.1\noptions edns0" > /etc/resolv.conf

Operation

Unbound and DNSSEC-Trigger come with the CLI utilities unbound-control and dnssec-trigger-control that can be used to inspect/alter the state of the corresponding services. Useful invocations of those include:

dnssec-trigger-control status # show overall status
unbound-control list_forwards # show upstream DNS servers their associated domains
unbound-control list_insecure # show the NTA list

These utilities can do many other things too, of course. Make sure to check out their manual pages.

To inspect the logs from Unbound and DNSSEC-Trigger, use journalctl -u dnssec-triggerd -u unbound.

Private Domains and NTAs

By default, DNSSEC-Trigger will configure explicit forwarding for search domains learned from the network, except for search domains learned from wireless networks. It will by default ignore domains learned from wireless networks. On Fedora, it will also add NTAs for the non-wireless search domains by default.

The logic behind that behaviour is that wireless networks come with an unacceptable high risk of downgrade attacks, while this risk is considered acceptable on other network types like wired, mobile broadband and VPN.

While this thinking is not entirely unreasonable, it is incompatible with my use case for two reasons:

  1. The Fedora default disables DNSSEC validation for work domains when I am docked at the office or connected via VPN. This breaks SSHFP validation for work servers.
  2. Ignoring domains learned from wireless networks means I do not get explicit forwarding nor an NTA for my private .lan domain at home.

The first issue can be resolved by enabling validate_connection_provided_zones in dnssec.conf, the second by enabling add_wifi_provided_zones in the same file. One issue remains, though; my private .lan domain lacks an NTA.

My original idea was to simply hard-code domain-insecure: lan in unbound.conf (same as when running Unbound standalone). That did not work, however, because DNSSEC-Trigger would just override it by removing the .lan from the NTA list.

In the end I settled on writing using a short NetworkManager dispatcher script which runs after DNSSEC-Trigger, which re-adds the .lan NTA and ensures any cached Bogus validation results within that domain are cleared. This is a bit of a hack, but it works well enough.

If using Unbound standalone, on the other hand, forwarding for private domains and adding NTAs for them must be done manually, as described in the Quick Start section above.

Toggling DNSSEC Validation / Accessing Captive Portals

When Using DNSSEC-Trigger

To toggle DNSSEC validation:

# Disable local DNSSEC validation (use upstream DNS servers directly)
dnssec-trigger-control hotspot_signon

# Re-enable DNSSEC validation and flush caches
dnssec-trigger-control reprobe

As previously mentioned, DNSSEC-Trigger has a GUI front-end application. On Fedora, it is not installed as part of the dnssec-trigger package, so you need to install it separately:

pkcon -y install dnssec-trigger-panel

The application runs in the system tray, and can be clicked to show a little menu where you can toggle DNSSEC on/off and inspect the status of the DNSSEC-Trigger daemon. When a captive portal is detected, a pop-up shows up, asking if you want to (temporarily) disable DNSSEC to allow you to log in to the captive portal. There are some screenshots posted on the NLNet Labs web site that show how it works.

Note that GNOME (the default graphical user interface on both Fedora and Ubuntu) does not have system tray functionality, so the GUI icon will not be displayed. However, the captive portal pop-up still works fine.

When Using Unbound Standalone
# Disable local DNSSEC validation (use upstream DNS servers directly)
cat /var/run/NetworkManager/resolv.conf > /etc/resolv.conf

# Re-enable DNSSEC validation and flush caches
echo -e "nameserver 127.0.0.1\noptions edns0" > /etc/resolv.conf
unbound-control flush_zone .

Problematic Observations

DNSSEC-Trigger does not add options edns0 to the /etc/resolv.conf file it generates (bug report). This prevents other software from determining whether or not a DNS response was Secure or Insecure, breaking OpenSSH’s ValidateHostKeysDNS feature. It can be worked around, e.g., by adding export RES_OPTIONS=edns0 to your ~/.bashrc file (or similar).

Conclusion

The DNSSEC support in Unbound itself appears solid. Highly recommended.

I am more ambivalent about DNSSEC-Trigger. While the NetworkManager integration and captive portal detection are nice features, the all-or-nothing approach to NTAs did not fit well with my use case. It works well enough to work around it with a NetworkManager dispatcher script, though.

Knot Resolver

Knot Resolver is a caching recursive resolver developed at CZ.NIC. Highlights:

  • Performs strict DNSSEC validation out of the box.
  • Designed for socket activation.
  • Very flexible due to use of Lua for configuration.
  • Support DNS over TLS for confidentiality.
  • No NetworkManager integration.

Quick Start

# Install and enable Knot Resolver
pkcon -y install knot-resolver
systemctl enable --now kresd.socket

# Enable forwarding of private '.lan' queries with NTA
echo "trust_anchors.set_insecure({'lan'})" >> /etc/knot-resolver/kresd.conf
echo "policy.add(policy.suffix(policy.FORWARD('192.168.1.1'),{todname('lan')}))" >> /etc/knot-resolver/kresd.conf

# Enable stub mode (forwarding of all queries)
echo "policy.add(policy.all(policy.FORWARD('1.1.1.1')))" >> /etc/knot-resolver/kresd.conf

# Enable logging of failed DNSSEC validations
echo "modules.load('bogus_log')" >> /etc/knot-resolver/kresd.conf

# Start (or restart) Knot Resolver to activate changes
sudo systemctl restart kresd@1.service

# Configure system to use the local Knot Recursor
echo -e "[main]\ndns=none" > /etc/NetworkManager/conf.d/dns.conf
systemctl reload NetworkManager.service
find /etc/resolv.conf -type l -delete
echo -e "nameserver 127.0.0.1\noptions edns0" > /etc/resolv.conf

Operation

You will need to speak Lua, as that is the only language Knot Resolver understands. The control interface is a UNIX domain socket you can connect to using either nc or socat, ideally within rlwrap:

rlwrap nc -U /var/run/knot-resolver/control@1
rlwrap socat - UNIX-CONNECT:/var/run/knot-resolver/control@1

(rlwrap adds standard shell behaviour like command line history and editing shortcuts like ^W to delete a word, and so on. It is completely optional, but I recommend it as makes the interactive CLI much more pleasant to work with. You might need to install it separately with pkcon -y install rlwrap).

You can also issue commands in a non-interactive way, for example:

$ echo "trust_anchors.summary()" | nc -U /var/run/knot-resolver/control@1
> lan. is negative trust anchor
.                       9200    DNSKEY  257 3 8 AwEAAaz/tAm8yTn4Mfeh5eyI96WSVexTBAvkMgJzkKTOiW1vkIbzxeF3+/4RgWOq7HrxRixHlFlExOLAJr5emLvN7SWXgnLh4+B5xQlNVz8Og8kvArMtNROxVQuCaSnIDdD5LKyWbRd2n9WGe2R8PzgCmr3EgVLrjyBxWezF0jLHwVN8efS3rCj/EWgvIWgb9tarpVUDK/b58Da+sqqls3eNbuv7pr+eoZG+SrDK6nWeL3c6H5Apxz7LjVc1uTIdsIXxuOLYA4/ilBmSVIzuDWfdRUfhHdY6+cn8HFRm+2hM8AnXGXws9555KrUB5qihylGa8subX2Nn6UwNR1AkUTV74bU= ; Valid: ; KeyTag:20326

The documentation and the help() command are good places to start out learning to operate Knot Resolver using the Lua interface.

To read the log messages from Knot Resolver, use journalctl -u kresd@1.

Private Domains and NTAs

As there is no NetworkManager integration, it is necessary to handle routing of private domains and addition of NTAs manually. It is pretty straight forward, as shown in the Quick Start section. The commands can either be added to the configuration file, or sent as commands to the control socket.

Toggling DNSSEC Validation / Accessing Captive Portals

To toggle DNSSEC validation, use:

# Disable local DNSSEC validation (use upstream DNS servers directly)
cat /var/run/NetworkManager/resolv.conf > /etc/resolv.conf

# Re-enable DNSSEC validation and flush caches
echo -e "nameserver 127.0.0.1\noptions edns0" > /etc/resolv.conf
echo "cache.clear_everything()" | nc -U /var/run/knot-resolver/control@1

Problematic Observations

None.

Conclusion

Knot Resolver is a solid DNSSEC validator. Highly recommended.

One possible drawback is the lack of NetworkManager integration. The upstream DNS server, domain forwarding rules and NTAs must all be configured manually.

That Knot Resolver is configured entirely using Lua is something I personally think is awesome, but it does come with a steeper learning curve compared to simpler and less flexible key=value-style configuration files.

PowerDNS Recursor

PowerDNS Recursor is a caching recursive resolver developed by PowerDNS.COM BV.

  • Does not perform strict DNSSEC validation out of the box.
  • Very flexible due to use of Lua for configuration.
  • No NetworkManager integration.

Quick Start

# Install and enable PowerDNS Recursor
pkcon -y install pdns-recursor
systemctl enable pdns-recursor.service

# Handle diverging config directories on Fedora and Ubuntu
cd /etc/powerdns || cd /etc/pdns-recursor

# Enable strict DNSSEC validation
sed -Ei "s|.*(dnssec)=.*|\1=validate|" recursor.conf

# Enable forwarding of private '.lan' queries with NTA (see note below)
sed -Ei "s|.*(forward-zones-recurse=.*)|\1,lan=192.168.1.1|" recursor.conf
sed -Ei "s|.*(lua-config-file)=.*|\1=$PWD/recursor.lua|" recursor.conf
echo "addNTA('lan')" > recursor.lua

# Enable stub mode (forwarding of all queries)
sed -Ei "s|.*(forward-zones-recurse=.*)|\1,.=1.1.1.1|" recursor.conf

# Log failed DNSSEC validations to system journal
sed -Ei "s|.*(dnssec-log-bogus)=.*|\1=yes|" recursor.conf

# Start (or restart) PowerDNS Recursor to activate changes
sudo systemctl restart pdns-recursor.service

# Configure system to use the local PowerDNS Recursor
echo -e "[main]\ndns=none" > /etc/NetworkManager/conf.d/dns.conf
systemctl reload NetworkManager.service
find /etc/resolv.conf -type l -delete
echo -e "nameserver 127.0.0.1\noptions edns0" > /etc/resolv.conf

(Note: I had to use forward-zones-recurse instead of the more natural forward-zones to configure forwarding of the private .lan domain. This works around a bug in Dnsmasq, which is embedded in my OpenWRT home gateway.)

Operation

PowerDNS Recursor is controlled with the utility rec_control. For example:

$ rec_control get-ntas
Configured Negative Trust Anchors:
lan

The rec_control tool can produce lots of interesting statistics, like for example rec_control top-pub-queries which shows which domains is queried for the most. Run rec_control help to see the list of available commands.

To inspect PowerDNS Recursor’s logs, use journalctl -u pdns-recursor.

Private Domains and NTAs

As there is no NetworkManager integration, it is necessary to handle routing of private domains and addition of NTAs manually. It is pretty straight forward, as shown in the Quick Start section.

It is also possible to add NTAs runtime using rec_control add-nta <domain>.

Toggling DNSSEC Validation / Accessing Captive Portals

To toggle DNSSEC validation, use:

# Disable local DNSSEC validation (use upstream DNS servers directly)
cat /var/run/NetworkManager/resolv.conf > /etc/resolv.conf

# Re-enable DNSSEC validation and flush caches
echo -e "nameserver 127.0.0.1\noptions edns0" > /etc/resolv.conf
systemctl restart pdns-recursor.service

Note that while it is possible to use rec_control wipe-cache example.com to clear the cache for example.com specifically, it does not clear subdomains (like www.example.com). I found no way to completely clear the cache, short of restarting the service.

Problematic Observations

Version 4.1 of PowerDNS Recursor appears to have a bug that causes reverse look-ups of IPv6 addresses (PTR lookups in the ip6.arpa zone) to fail with a SERVFAIL error. The bug only occurs when in stub mode and with DNSSEC validation enabled. The failures are not logged as DNSSEC validation failures, however. Both Fedora and Ubuntu include v4.1.

The bug appears to be fixed upstream. I could not reproducible it with PowerDNS Recursor v4.2, which was released last month.

Conclusion

PowerDNS Recursor version v4.2 seems to work very well for DNSSEC validation. Recommended.

The v4.1 version included with Fedora and Ubuntu unfortunately have a bug that breaks validation of reverse IPv6 lookups, so I would avoid v4.1.

The lack of NetworkManager integration is a disadvantage, meaning that the upstream DNS server, domain forwarding rules and NTAs must all be configured manually.

There is a Lua configuration interface, which means you can probably do lots of cool stuff with it.

BIND

You could perhaps call BIND by Internet Systems Consortium the «original DNS server». Compared to most of the other resolvers in my test, it has been around for a very long time. Some highlights:

  • Performs strict DNSSEC validation out of the box.
  • No NetworkManager integration.
  • Can also serve as an authoritative name server.

Quick Start

# Install and enable BIND
pkcon -y install bind bind9
systemctl enable named.service # Fedora
systemctl enable bind9.service # Ubuntu

# Handle diverging config directories on Fedora and Ubuntu
cd /etc/bind || cd /etc

# Enable forwarding of private '.lan' queries with NTA (see note below)
echo "zone \"lan\" IN { type forward; forwarders { 192.168.1.1; }; forward only; };" >> named.conf
# Add permanent NTA for '.lan' (this requires BIND v9.14 or newer)
#sed -Ei "s|(options \{)|\1 validate-except { \"lan\"; };|" named.conf*

# Enable stub mode (forwarding of all queries)
sed -Ei "s|(options \{)|\1 forwarders { 1.1.1.1; }; forward only;|" named.conf*

# Start (or restart) BIND to activate changes
sudo systemctl restart named.service bind9.service

# Add temporary NTA for '.lan' (use with BIND older than v9.14)
rndc nta -lifetime 604800 lan

# Configure system to use the local BIND
echo -e "[main]\ndns=none" > /etc/NetworkManager/conf.d/dns.conf
systemctl reload NetworkManager.service
find /etc/resolv.conf -type l -delete
echo -e "nameserver 127.0.0.1\noptions edns0" > /etc/resolv.conf

Note that the BIND installation vary significantly between Fedora and Ubuntu. The above commands take those differences into account, but will produce some harmless error messages about packages/files/services not being found.

Operation

BIND is operated using the rndc tool:

$ rndc status
version: BIND 9.11.9-RedHat-9.11.9-1.fc30 (Extended Support Version) <id:80a845e>
running on sloth.fud.no: Linux x86_64 5.2.9-200.fc30.x86_64 #1 SMP Fri Aug 16 21:37:45 UTC 2019
[...]

rndc help shows the list of available commands.

To inspect BIND’s logs, use journalctl -u bind9 -u named.

Private Domains and NTAs

Since there is no NetworkManager integration, it is necessary to manually configure forwarding for private domains and their associated NTAs.

One big caveat is that the validate-except configuration directive that is used to configure permanent NTAs requires BIND v9.14 or newer. Fedora and Ubuntu ship older v9.11 versions.

With BIND v9.11, it appears that the only way to add NTAs is to use rndc, like so:

$ rndc nta -lifetime 604800 lan
Negative trust anchor added: lan/_default, expires 03-Sep-2019 13:38:06.000

This adds a temporary NTA which will eventually expire. The maximum lifetime it will accept is one week (604800 seconds), so in order to make this NTA “permanent” you will need to create a timed system service or a cron job that repeats the above command at least once per week.

To inspect the list of active NTAs, use rndc nta -dump.

Toggling DNSSEC Validation / Accessing Captive Portals

To toggle DNSSEC validation, use:

# Disable local DNSSEC validation (use upstream DNS servers directly)
cat /var/run/NetworkManager/resolv.conf > /etc/resolv.conf

# Re-enable DNSSEC validation and flush caches
echo -e "nameserver 127.0.0.1\noptions edns0" > /etc/resolv.conf
rndc flush

Problematic Observations

None.

Conclusion

BIND’s DNSSEC validation engine works very well. Recommended.

Apart from the fact that there is no NetworkManager integration, the biggest drawback for me was that you require a newer version than what is available in the Linux distributions in order to add permanent NTAs.

Summary

All in all, my two favourites were Knot Resolver and Unbound. They both appear to be mature pieces of software with rock solid DNSSEC implementations. They both support DNS over TLS, another plus for security-conscious users.

Choosing between them depends on the specifics of the use case. If NetworkManager integration is important, then Unbound+DNSSEC-Trigger is the obvious choice, even though DNSSEC-Trigger’s NTA handling is not exactly ideal.

If NetworkManager integration is not important, then Knot Resolver and Unbound (standalone) will both do a great job. Knot Resolver does appear to have a little bit more modern architecture than Unbound, considering its support for Lua and socket activation, but this is unlikely to make any significant difference for the simple use case described in this post.

PowerDNS Recursor and BIND also have solid DNSSEC implementations, but only if you run newer versions than those available in the Fedora and Ubuntu repositories. Neither PowerDNS Recursor or BIND seem to offer any unique functionality that sets them apart from Knot Resolver and Unbound, so to me it does not seem worth the trouble to upgrade.

systemd-resolved and Dnsmasq did unfortunately disappoint - their DNSSEC implementations were too buggy to be considered usable for me. This is a shame, especially when it comes to systemd-resolved, because it would otherwise have been the perfect tool for the job.

Tore Anderson

Senior Systems Consultant at Redpill Linpro

Tore works with infrastructure at Redpill Linpro. Joining us more than a decade ago as a trainee, Tore is now responsible for our network architecture and operations.

More Posts

Comparison of different compression tools

Working with various compressed files on a daily basis, I found I didn’t actually know how the different tools performed compared to each other. I know different compression will best fit different types of data, but I wanted to compare using a large generic file.

The setup

The file I chose was a 4194304000 byte (4.0 GB) Ubuntu installation disk image.

The machine tasked with doing of the bit-mashing was an Ubuntu with a AMD Ryzen 9 5900X 12-Core ... [continue reading]

Why TCP keepalive may be important

Published on December 17, 2024

The irony of insecure security software

Published on December 09, 2024