Allow backup sysadmins to gain access through a "Break the Glass"-solution

This post appeared originally in our sysadvent series and has been moved here following the discontinuation of the sysadvent microsite

I want backup sysadmins to have login access to some systems, with said access rarely (if ever) used. To prevent abuse I’d like strong audit logging, logging that stands out from the rest of all the logging, logging that cannot be tampered with, and that can easily be followed up in case of abuse. I’ve been fixing a working setup through PAM, sending emails or instant messaging through third-parties. Jump down to the last section (“Specific complete examples”) to find a working setup.

Motivation

Consider this: The probability that something goes wrong due to malice, accident or incompetence increases with the number of people having login/sudo-access (it may even increase overlinearly). At the other hand, if something goes wrong, the more people having access to the system, the higher probability that the problem may be fixed properly within a reasonable time frame. How to strike the balance?

In some situations, a login account or a sudo access on some server can also be considered to be for “emergency-only” usage; daily maintenance is typically done by one or two sysadmins, but there are a dozen others that may access the server in case of emergency. On your personal computers where you store your SSH keys and other important passwords, you may not want to allow anyone else than yourself to log in - and still, there are situations when it may be convenient to ask some colleague or family member things like: “my X session seems to have frozen, can you log into my workstation and do a sudo chvt 3 for me?” I also have personal servers doing various stuff such as receiving my emails - on those systems I’d also like to have the possibility to yield access to others if I’m offline and there are problems - or if I lack the competence to fix the problems.

In a “break-glass”-solution the probability of abuse of emergency equipment is significantly lowered by increasing the usage threshold. Nobody would hesitate to break a glass in a real emergency, but the probability that someone would use the equipment or press the button “just for fun” is lowered a lot - even if some people enjoy breaking glass. Wouldn’t it be nice to make some “glass breaking”-provisions to reduce the likelihood that some classes of users would log in or gain sudo-access?

“Break the glass”-design

For a “break-the-glass”-solution to be efficient, those things applies:

• The user should be made aware of the “glass”. Some ideas:

  • Warnings and possibilities to bail out during the login stage (difficult to accomplish in the context of SSH and key-based login)

  • Simply a well-known company-policy: “you are not supposed to log into those systems except in case of emergency”.

  • Some convention i.e. to use the suffix “_eo” for “emergency only” on any user name that aren’t supposed to log in.

  • Some provisions making it more difficult to get access. Say, one should write an email to some specific address requesting access, and access is automatically granted - but the sysadmins responsible for the box and/or managers would know that you’re trying to get in.

  • “Need-to-know”-obscurity - create the accounts, and inform that they exist only in the case of an emergency.

• There should be strong and persistent audit trails; the user should not be able to simply overwrite the logs after logging in.

• The audit trails should be followed up, i.e. some manager should be following up, questioning “why did you do log in on system A at Saturday 02:00AM, and what did you need root for?”

With those three in place, people will most likely not log in unless they have a good reason for it, and the likelihood of wrongdoings is reduced a lot.

“Break the glass”-implementation

PAM

Probably the Linux PAM (Pluggable Authentication Modules) is the right place to throw in logic; on almost any Linux computer as of 2017 all entry points into a host and most privilege escalation routes goes through PAM.

On modern distros the PAM configuration is located under /etc/pam.d/. There is one file for each program utilizing PAM, in addition there are include-files for common logic. The most appropriate approach here is arguably to put your changes into those include-files to make sure the logic applies to all programs - but make sure you won’t get locked out while experimenting:

• Keep a terminal window open with root access on the host you’re working on

• Try out changes on a harmless program first - you probably use sudo for gaining root access on the system, so experimentation with su is probably harmless. (caveat: I excluded myself from the wheel group while experimenting, this caused sudo to fail). Similarly, if you have both SSH and console access, playing with the console login should be harmless.

• Eventually, use an expendable VM that can be easily rebuilt.

• It’s a good idea to throw some revision control over the files while editing them.

To get an overview of the include files, you may:

grep -r include /etc/pam.d/

On Arch Linux, all the include files are prefixed with “system-“. There is system-auth for privilege-escalating utilities. There is system-remote-login and system-local-login, two identical files that does nothing but include the general system-login configuration.

Understanding and debugging PAM

An introduction to the PAM configuration is outside the scope of this article; most of what you need is in the system administrators guide (“sag”). All of the sag should be mirrored perfectly in your local man-pages. Except for the sag, there doesn’t seem to be a whole lot of online resources - though there may be some relevant pointers on sites like Stackoverflow, various online forums, bug trackers, etc. The rest of this article assumes you have a basic knowledge of how PAM works, but you should be able to follow the examples even without this knowledge.

For debugging the single most useful module is pam_echo. It doesn’t say explicitly in the man-page, but this is the simple usage:

session optional pam_echo.so some text here

There is also a module pam_debug.so. It can be used as a temporary placeholder, mocking up return codes for verifying that the configuration works as intended in scenarios that aren’t easily reproducible. Actually, pam_debug.so seems to be the more generic variant of the modules pam_permit.so and pam_deny.so.

Pre-login warning messages through PAM

One can use the module pam_echo for writing messages to the user during the login. To make sure it’s written before the user enters a password, it has to be at the top of the PAM configuration file and with the auth or account type, i.e. try to insert this line at the top of your /etc/pam.d/su file:

auth   required  pam_echo.so  file=/etc/preloginmsg

Then populate the /etc/preloginmsg with an appropriate warning message:

# echo<<EOF>/etc/preloginmsg
Login/superuser access granted for emergencies only.
All logins will be logged and flagged
Please abort, unless it is an emergency

Now, verify that when doing “su”, the message is printed out first, and then probably you’ll be asked for password.

Obviously, this cannot be made water-tight on systems where one can get access without typing a password. So far I haven’t found any simple way to use PAM to prompt the user “please disconnect or press enter to break the glass”; it should be possible (pam_exec to the rescue, in worst case), but then again this is probably a bad idea, such a scheme may break non-interactive use-cases for SSH and sudo. The simplest approach seems to expect the users to know in advance where they are supposed to log in and where they are not supposed to log in - and/or to assume it’s no disaster if the glass gets broken accidentally. For interactive SSH sessions, the message can be delivered post-login:

session   required  pam_echo.so  file=/etc/postloginmsg

Populate it with some appropriate message:

# cat<<EOF>/etc/postloginmsgecho
Access granted for emergency handling
Please log out immediately unless it's an emergency
EOF

Securing the log

One approach is to use rsyslog to ensure important syslog messages are delivered to some remote host, and taken action on - but this may be non-trivial, for several reasons - like those three:

• There is quite some complexity in designing and configuring a good logging solution; you will need to set up a node receiving logs, and ensure said host cannot be compromised. This may be costly, time-consuming and non-trivial.

• By default log messages are delivered by UDP, but UDP is unreliable by design. TCP is risky, as whole system may stall if the remote logging server is unavailable.

• Making sure the right log messages are actually read and taken action on is yet another problem.

Sending information from Pam to the syslog is easy, i.e. one can use the warn.so module:

session required pam_warn.so

I was thinking, the simplest way to dispatch logs in a tamper-proof way would be to use some third-party cloud provider - i.e. GitHub or google. There is the exec module for running any kind of logic, by executing a script - like this:

session required pam_exec /usr/local/bin/auth_warn

Information from PAM is passed as environment variables - PAM_RHOST, PAM_RUSER, PAM_SERVICE, PAM_TTY, PAM_USER and PAM_TYPE

auth_warn can send an email through gmail:

# cat<<EOF>/usr/local/bin/auth_warn
echo "Glass broken:\
    \$PAM_RUSER@\$PAM_RHOST -> \$PAM_USER@\$HOSTNAME, service \$PAM_SERVICE" |
    mail -S "smtp=gmail-smtp-in.l.google.com" \
         -S"from=yourself@example.com" \
         -s "broken glass at \$HOSTNAME" yourself@gmail.com
EOF
# chmod a+x /usr/local/bin/auth_warn

The command above will deliberately wait until the message is delivered before it exits. This will cause some seconds delay before access is granted. One may need to tweak the mail setup or the gmail account settings a bit, the message may be likely to end up in some spam folder at gmail.

Or one can send a message on some instant messaging network, for instance Telegram - there is a command line utility available at https://github.com/vysheng/tg - in Arch Linux telegram-cli-git can be installe through AUR - but it seems like it is only designed to be run interactively as a terminal program, i.e. under screen. Nevertheless, this seems to work:

# cat<<EOF>/usr/local/bin/auth_warn
{
    echo "contact_search my_user_name"
    sleep 1
    echo "msg @my_user_name Glass broken:\
        \$PAM_RUSER@\$PAM_RHOST -> \$PAM_USER@\$HOSTNAME, service \$PAM_SERVICE"
} |
telegram-cli > /dev/null 2>&1
EOF
# chmod a+x /usr/local/bin/auth_warn

With some more experimentation, one can probably get the message to go to a group chat rather than a personal account.

Ensuring “break-the-glass”-logic won’t apply to the daily legitimate users of the system

Separating users into classes

First off, it’s probably best to use the UNIX groups to sort out those who aren’t supposed to log in - say, anyone in the group emergencylogin is not supposed to log in, and users in the emergencysu group is not supposed to gain root privileges:

# groupadd emergencylogin
# for user in joe_eo ole_eo brian_eo knuth_eo ; do usermod -aG emergencylogin $user ; done
# groupadd emergencysu
# for user in joe_eo ole_eo pointy_hairy_boss katie susan  ; do usermod -aG emergencysu $user ; done

Making PAM aware of the user classes

It’s important to understand one crucial difference here: for the logins, the logic should filter on the target users - while for the sudo it’s the source users that are important.

There are four PAM modules that may help us:

• pam_access - allows for specifying access details in /etc/security/access.conf
• pam_wheel - allows for users of the wheel group to gain su access
• pam_succeed_if - allows for various logic to be specified directly in the PAM configuration
• pam_exec - run anything

pam_access may work for checking the target user (login), but apparently it does not support checks on the source user (sudo access). Access definitions should be in a separate file (typically /etc/security/access.conf).

pam_wheel may work for checking the source user (sudo), but apparently it does not support checks on the target user (login). It also doesn’t work in the session layer.

pam_succeed_if may work for checking the target user (login), and it does also support checks on the source user (sudo access).

pam_exec is the most flexible - it can do whatever - but again, the modules above seem to be simpler, leaner and cheaper.

So apparently pam_wheel should be good for privilege escalation and pam_succeed_if can be used anywhere. While experimenting I found that sudo logins without password will skip all layers except the session layer, hence pam_wheel could not be used there. We’ll stick to pam_succeed_if.

Stacking the PAM modules

We can instruct PAM to skip one line of the configuration file if some pam module yields a specific return code, like this:

auth [some_return_code=1 default=ignore] pam_some_module.so module_options=1

pam_succeed_if will return success if a user is a member of a specific group. Since we’re interested in doing things on success, we turn it around - we’ll skip one line by default and do nothing on success. The syntax will be like this for su/sudo etc:

auth [default=1 success=ignore] pam_succeed_if.so ruser ingroup emergencysu
auth optional pam_exec.so /usr/bin/local/auth_warn

and this for login methods:

auth [default=1 success=ignore] pam_succeed_if.so user ingroup emergencylogin
auth optional pam_exec.so /usr/local/bin/auth_warn

Above I’ve assumed that the default should be to allow people in if there are problems with the “security glass”. If anything goes wrong, either in pam_succeed or in auth_warn, user will be let in without any fuzz. A malicious user that somehow can disable the auth_warn logic - i.e. disconnect the computer from the network - will be able to get in without triggering the audit logging. We might want to reverse this by replacing “optional” with “required”:

auth [default=1 success=ignore] pam_succeed_if.so ruser ingroup emergencysu
auth required pam_exec.so /usr/local/bin/auth_warn

and similarly:

auth [default=ignore auth_err=1] pam_succeed_if.so user ingroup emergencylogin
auth required pam_exec.so /usr/local/bin/auth_warn

Stitching the PAM configuration together

There are two approaches here - I’d say the proper approach is to add the logic into include-files. However, in many setups as of 2017, the only way an emergency sysadmin would log in is through SSH, and the only way privileges would be gained is through sudo, so it should suffice to add the logic into those two. Also, exact configuration may be dependent on the program - like, sudo may skip both the account and auth steps (forcing all logic into the session layer), while SSHD may skip the session layer.

Specific complete examples

First of all, fix the /usr/local/bin/auth_warn to send logs securely through some third party provider (YMMV on this one).

# MYSRCEMAIL=yourself@example.com ; MYGMAIL=yourself@gmail.com ; MYTELEGRAMUSER=your_username_here
# cat<<EOF>/usr/local/bin/auth_warn
#!/usr/bin/bash
## Send email through gmail
echo "Glass broken:        \$PAM_RUSER@\$PAM_RHOST -> \$PAM_USER@\$HOSTNAME, service \$PAM_SERVICE" |
    mail -S "smtp=gmail-smtp-in.l.google.com"              -S"from=$MYSRCEMAIL" \
         -s "broken glass at \$HOSTNAME" $MYGMAIL

## Send message through telegram
{
    echo "contact_search $"
    sleep 1
    echo "msg @$MYTELEGRAMUSER Glass broken:\
        \$PAM_RUSER@\$PAM_RHOST -> \$PAM_USER@\$HOSTNAME, service \$PAM_SERVICE"
} |
sudo -i -u $LOGNAME telegram-cli > /dev/null 2>&1
EOF
# chmod a+x /usr/local/bin/auth_warn

Messages to the logged-in user

# echo<<EOF>/etc/preloginmsg
Login/superuser access granted for emergencies only.
All logins will be logged and flagged
Please abort, unless it's an emergency
EOF
# cat<<EOF>/etc/postloginmsgecho
Access granted for emergency handling
Please log out immediately unless it's an emergency
EOF

Mark up the users who aren’t supposed to log in / gain su

# emergency_users="alice bob charlie douglas eve"
# groupadd emergencylogin
# groupadd emergencysu
# for user in $emergency_users ; do usermod -aG emergencylogin $user ; usermod -aG emergencylogin $user ; done

This is an example of a working su file, without include files. With this setup, we presume the person to gain access knows the root password and that he’s not member of the wheel group. In general, su is deprecated in favor of sudo nowadays. It is nice with an emergency backdoor though, in case someone messes up the sudoers config file(s).

# cat>/etc/pam.d/su<<EOF
#%PAM-1.0
account         required        pam_unix.so

## root should always have access to su, no questions asked
auth           sufficient        pam_rootok.so

## wheel members too, should be able to su easily
auth           sufficient        pam_wheel.so trust use_uid

## give a pre-login warning to members of the emergencysu group
auth [default=1 success=ignore]  pam_succeed_if.so ruser ingroup emergencysu
auth           optional          pam_echo.so file=/etc/preloginmsg

## ask for a password
auth           required          pam_unix.so

## strong audit logging on members from the emergencysu group
auth [default=1 success=ignore]  pam_succeed_if.so ruser ingroup emergencysu
auth           optional          pam_exec.so /usr/local/bin/auth_warn

session        required          pam_unix.so
EOF

This is an example of a working /etc/pam.d/sshd file on my arch Linux. It’s tested only with “PasswordAuthentication no”. In this mode SSHD does most of the auth logic itself, it rejects failed logins before the PAM layers. This is arguably bad, as PAM probably should be the one-stop-goto for intrusion attempt avoidance, including rate limiting logic - but for us it means we can safely put things into account without getting lots of audit logs on failed logins. The includes are likely to be arch-specific, use your favorite editor to just add the first and last section.

#%PAM-1.0

## "strong" audit logging of login attempts
account [default=1 success=ignore] pam_succeed_if.so user ingroup emergencylogin
account optional pam_exec.so /usr/local/bin/auth_warn

## ArchLinux include logic (untouched)
auth      include   system-remote-login
account   include   system-remote-login
password  include   system-remote-login
session   include   system-remote-login

## Give the user a hint that the login will be logged.
## Logic below only applies to interactive login sessions.
session [default=1 success=ignore] pam_succeed_if.so user ingroup emergencylogin
session   optional  pam_echo.so file=/etc/postloginmsg

This is an example of working sudo file on my arch Linux. Sudo also skips quite some of the PAM logic, it seems like the session logic is the only one we can be sure will be run.

#%PAM-1.0

## ArchLinux include logic (untouched)
auth      include   system-remote-login
account   include   system-remote-login
password  include   system-remote-login
session   include   system-remote-login

## "strong" audit logging of login attempts.
session [ignore=ignore default=1] pam_succeed_if.so ruser ingroup emergencylogin
session optional pam_exec.so /usr/local/bin/auth_warn

## Give the user a hint that the login will be logged.
## Logic below only applies to interactive login sessions.
session [ignore=ignore default=1] pam_succeed_if.so ruser ingroup emergencylogin
session   optional  pam_echo.so file=/etc/postloginmsg