This post appeared originally in our sysadvent series and has been moved here following the discontinuation of the sysadvent microsite
While dynamic DNS is a wonderful tool for automation and orchestration, tools for easy cleaning up and logging changes are needed. This post describes a couple of scripts that may help.
A good thing: Dynamic DNS in automation
In a world of automation, dynamic DNS is a wonderful tool. It lets automatic provisioning and orchestration tools create, update, and delete DNS records all by itself. But from time to time, strange things happens. Scripts fail or get killed. Automatic systems may be used in ways nobody had thought of, or even, users do things they shouldn’t. So we might need tools to handle that. Here are two helper scripts that may help in taming Dynamic DNS. The scripts may be downloaded from GitHub at https://github.com/ingvarha/dyndns_helper_scripts
Note that these scripts may be filled with errors, and do all kind of crazy stuff. Always test thoroughly before using anything in production environments. Please report any bugs or strange behaviour on the GitHub page.
Remove lingering records
One of the most common problems with dynamic DNS in our environments is that old records are lingering. So the first script is a cleaning script:
Say, you just found an old record in DNS, for a name that is no longer in use. You would like to remove it, including any A, AAAA, and TXT records that might be added. And by the way, also remove matching PTR (reverse) records.
The script delete_host_from_dns does all this in one go. It even has an “oh sh*t function” that makes you reinsert the records in case you removed something that should have been left untouched.
The script uses nsupdate(1) from the bind-utils package to send updates to DNS. For auth, you also need a TSIG_authentication_key matching the zones you want to update. Read the nsupdate man page for details.
The usage of the script should be quite self-explanatory
Usage: delete_host_from_dns [-k keyfile] [-d] [-nr] [-f] [--dry-run] name
Removes A, AAAA, TXT, and optionally, PTR records for a name from (dynamic) DNS
Options:
-k --keyfile DNS keyfile, default is /home/myuser/.cert/Kmykey.key
-nr --noreverse Do not remove PTR (reverse) records
-nt --notxt Do not remove TXT records
-f --force Force removal, no warnings, no dry-run
--dry-run Just show me what will be done (default on)
Note that the script defaults to dry-run mode, so that you may check
what nsupdate commands that will be ran. To override dry-run mode, and
execute immediately, add the -f
option.
If a PTR record is found that does not point back to the original name, dry-run mode is turned on anyhow.
Here is an example run.
delete_host_from_dns test45.example.com
Looking for nsupdate ... /usr/bin/nsupdate
Warning: No reverse lookup for fe80::7ae3:b5ff:fe9f:1337
DRY RUN MODE
Will run these commands against DNS:
update delete test45.example.com. 60 IN A 10.20.149.220
send
answer
update delete test45.example.com. 60 IN AAAA fe80::7ae3:b5ff:fe9f:1337
send
answer
update delete 220.149.20.10.in-addr.arpa. 60 IN PTR test45.example.com.
send
answer
THIS IS DRY RUN MODE
To apply these changes for real, do
nsupdate -v -k /home/myuser/.cert/Kmykey.key < /tmp/tmp.uWf8jDhxia
You may remove the tempfile or edit it and run the above command by hand
Revert file is /tmp/tmp.W6j6RDgMPN
To revert all changes, try
nsupdate -v -k /home/myuser/.cert/Kmykey.key < /tmp/tmp.W6j6RDgMPN
Note: tempfiles are left behind. Clean up after yourself.
Your mother does not work here.
Who/When/What broke my zone?
The next problem is about state. Say something or somebody did a change. We do not know when the change happened, nor who did it, but it surely breaks stuff. It was not like this yesterday morning. So what did that DNS zone look like yesterday?
Or more precisely, who did the change, what was the change, and when was the change executed?
This second script followdns is a simple Perl log tailer, pushing zone changes to a Git repositories. Installed on the DNS master server, it tails the server log and looks for updates. When an update is found, it downloads the new zone, and adds the change to a Git repositories, as well as the the IP address and the key used by the client changing the zone. The script has been tested against bind version 9.
While the script is quite rude, it works, at least as a proof of concept in our environment. You should probably try running it in a screen for a few days before putting it in full production.
The script needs access to do download (axfr) on any zone it monitors. It also requires a local checked-out Git repositories, with a remote master, and must be able to push changes using SSH. For Git to pick up a non-standard SSH key, you may want to use the wrapper script ssh_wrap which points to the location of the key, and call the script like this:
export GIT_SSH=/path/to/ssh_wrap
/path/to/named-log-parser.pl >> /var/log/named-log-parser.log
A crude sysv initrc script is tossed in for good measure.
All configurations are hard-coded in the scripts. Edit it at your leisure.
When all the parts are patched up and sewn together, you get a lovely Git repositories with all dynamic changes, and can watch your dynamic zones change using your favorite Git visualization tool.
Thoughts on the CrowdStrike Outage
Unless you’ve been living under a rock, you probably know that last Friday a global crash of computer systems caused by ‘CrowdStrike’ led to widespread chaos and mayhem: flights were cancelled, shops closed their doors, even some hospitals and pharmacies were affected. When things like this happen, I first have a smug feeling “this would never happen at our place”, then I start thinking. Could it?
Broken Software Updates
Our department do take responsibility for keeping quite a lot ... [continue reading]