systemd comforts

This post appeared originally in our sysadvent series and has been moved here following the discontinuation of the sysadvent microsite

One common complaint about systemd is that it does «too much», where the threshold for the appropriate amount of action is left unspecified. Some of the stuff it can do is hold your hand and offer some comfort functions.

The result is that you don’t need to know where your unit files are. You can, and systemd will tell you, but you don’t need to know the path a priori. So here are some functions of systemctl that you don’t need to know about, but which can make your life a little easier.

systemctl cat

This will print your service file, with the path in a comment at the top:

$ systemctl cat systemd-journald.service
# /usr/lib/systemd/system/systemd-journald.service
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
## blah blah blah…

The first line with the path is not actually included in the service file, and there’s a good chance it’ll be printed in another color on your machine. This will print the unit file that systemd will actually use for that unit. In this case it’s running the vendor-supplied unit file, which you can tell by the fact that it’s in /usr/lib/systemd; your files are supposed to go in /etc/systemd/. Unless it’s a temporary file, they go in /run/systemd, or it might be a user file, in yet another set of paths that I don’t actually care to remember or think about. Hence this blog post.

systemctl edit

Now, if you for some reason need to change a unit file, use systemctl edit. I prefer systemctl edit --full which will give you the full file to edit, rather than expecting you to write a replacement snippet blind. systemctl edit will give you the appropriate file to edit, and save it in the appropriate place, and it will run systemctl daemon-reload. And of course it relies on ed, the standard editor.¹

By default (without --full) it will create an appropriate foo.d/override.conf; with --full it will create a foo.$unit. My intuition tells me that if I make no changes, no new file will be written, and that’s correct for the default version, but with --full you’re in effect creating a copy of the vendor unit file as it exists at that time.

If you need to delete something from a unit file, use --full and delete the line there.

It has some extra features:

• --force is handy if you want to start a brand new unit file: it’ll create a unit file with that name if one doesn’t already exist. Requiring this to create new unit files means you won’t inadvertently create a new unit file if you do a typo or are confused about what the service name is (like me, I always expect «postgres», not «postgresql» (yes, I know about auto-completion)).
• --runtime makes a temporary edit that will be lost on the next reboot.

systemd-delta

Once you’ve edited a unit file, it can start diverging even more from the vendor-supplied one as upstream thinks up new and exciting ways to do blinkenlights. systemd-delta shows you how your system is different from a vanilla system, with diffs for overrides, which file supersedes what, and so on. Sadly, the output from systemd-delta isn’t sorted or stable, and there’s no systemd-delta-delta.

If you’ve used edit to create a foo.d/override.conf, it’ll show up as [EXTENDED]. If you used --full and made a copy, it’ll show up as [OVERRIDDEN]. The naming makes sense if you think about it, although it would be OK too if I didn’t have to think and the override.conf version matched with [OVERRIDDEN].

systemctl is-system-running

This is a very simple indicator of whether systemd thinks everything is running properly. systemctl is-system-running outputs the same thing as the «State» line in systemctl status.

• What you want is «running» and exit code 0.
• What you might get is «degraded» and exit code > 0. In that case, do systemctl list-units --state=failed to see what it’s unhappy about. If you don’t want to fix something, you can comfort systemd by telling it systemctl reset-failed $foo. This will not restart services or anything, it will just convince systemd that things are fine (morbid comic).
• There are other options, which are listed in table form in systemctl(1).

So by adding systemctl is-system-running to your login script or monitoring system, you can get a quick feel for whether something is off.

¹ Blatant lie. It uses $EDITOR.

Update

• 2025-09-03: Format