This post appeared originally in our sysadvent series and has been moved here following the discontinuation of the sysadvent microsite

If you rely on SSL/TLS certificates and you have a slew of services to maintain online, things can quickly get out of hand. If you don’t have the time or the resources to keep up to speed with what ciphers to disable or what techniques to employ server-side, you might quickly fall prey to the next “Exploit with a Logo”. Heartbleed, Beast, Poodle and friends come to mind.

The guys at Mozilla have taken measures to give all of us sysadmins more free time by maintaining lists of recommended steps to take server-side.

In addition they’ve also created Cipherscan, which is a great little tool to check if your sites comply with the latest security advisories.

Verify new certificates and ciphersuites on a test instance

One of my favorite features is that I can leverage SNI to check a new certificate on an arbitrary IP. This takes all the tension out of rolling out updated certificates every year or two, because I can verify that the new certificate has a sane certificate chain and does what it’s supposed to before I expose it to production traffic.

$ ./cipherscan --servername web.acme.customer.com 10.10.10.10:4435
.......................................
Target: 10.10.10.10:443

prio  ciphersuite                  protocols              pfs                 curves
1     ECDHE-RSA-AES128-GCM-SHA256  TLSv1.2                ECDH,P-256,256bits  prime256v1
2     ECDHE-RSA-AES256-GCM-SHA384  TLSv1.2                ECDH,P-256,256bits  prime256v1
3     DHE-RSA-AES128-GCM-SHA256    TLSv1.2                DH,2048bits         None
4     DHE-RSA-AES256-GCM-SHA384    TLSv1.2                DH,2048bits         None
5     ECDHE-RSA-AES128-SHA256      TLSv1.2                ECDH,P-256,256bits  prime256v1
6     ECDHE-RSA-AES256-SHA384      TLSv1.2                ECDH,P-256,256bits  prime256v1
7     ECDHE-RSA-AES128-SHA         TLSv1,TLSv1.1,TLSv1.2  ECDH,P-256,256bits  prime256v1
8     ECDHE-RSA-AES256-SHA         TLSv1,TLSv1.1,TLSv1.2  ECDH,P-256,256bits  prime256v1
9     DHE-RSA-AES128-SHA256        TLSv1.2                DH,2048bits         None
10    DHE-RSA-AES128-SHA           TLSv1,TLSv1.1,TLSv1.2  DH,2048bits         None
11    DHE-RSA-AES256-SHA256        TLSv1.2                DH,2048bits         None
12    DHE-RSA-AES256-SHA           TLSv1,TLSv1.1,TLSv1.2  DH,2048bits         None
13    ECDHE-RSA-DES-CBC3-SHA       TLSv1,TLSv1.1,TLSv1.2  ECDH,P-256,256bits  prime256v1
14    EDH-RSA-DES-CBC3-SHA         TLSv1,TLSv1.1,TLSv1.2  DH,2048bits         None
15    AES128-GCM-SHA256            TLSv1.2                None                None
16    AES256-GCM-SHA384            TLSv1.2                None                None
17    AES128-SHA256                TLSv1.2                None                None
18    AES256-SHA256                TLSv1.2                None                None
19    AES128-SHA                   TLSv1,TLSv1.1,TLSv1.2  None                None
20    AES256-SHA                   TLSv1,TLSv1.1,TLSv1.2  None                None
21    DES-CBC3-SHA                 TLSv1,TLSv1.1,TLSv1.2  None                None

Certificate: <span style="color:#00cd00;">trusted</span>, <span style="color:#00cd00;">2048</span> bits, <span style="color:#00cd00;">sha256WithRSAEncryption</span> signature
TLS ticket lifetime hint: 300
NPN protocols: None
OCSP stapling: <span style="color:#00cd00;">supported</span>
Cipher ordering: <span style="color:#00cd00;">server</span>
Curves ordering: <span style="color:#00cd00;">server</span> - fallback: <span style="color:#cd0000;">no</span>
Server <span style="color:#00cd00;">supports</span> secure renegotiation
Server supported compression methods: <span style="color:#00cd00;">NONE</span>
TLS Tolerance: <span style="color:#00cd00;">yes</span>

This checks the certificate and ciphers for web.acme.customer.com on a test instance at 10.10.10.10 (of course the test server needs to have the same web server setup as production).

Testing and fixing a bad setup

We have internal-service.example.com serving highly important pictures of lolcats to all employees, and we want to check if it’s up to scratch:

$ ./cipherscan</span> internal-service.example.com:443
.......................................
Target: internal-service.example.com:443

prio  ciphersuite                  protocols              pfs                 curves
1     ECDHE-RSA-AES256-GCM-SHA384  TLSv1.2                ECDH,P-256,256bits  prime256v1
2     ECDHE-RSA-AES256-SHA384      TLSv1.2                ECDH,P-256,256bits  prime256v1
3     ECDHE-RSA-AES256-SHA         TLSv1,TLSv1.1,TLSv1.2  ECDH,P-256,256bits  prime256v1
4     DHE-RSA-AES256-GCM-SHA384    TLSv1.2                DH,2048bits         None
5     DHE-RSA-AES256-SHA256        TLSv1.2                DH,2048bits         None
6     DHE-RSA-AES256-SHA           TLSv1,TLSv1.1,TLSv1.2  DH,2048bits         None
7     DHE-RSA-CAMELLIA256-SHA      TLSv1,TLSv1.1,TLSv1.2  DH,2048bits         None
8     AES256-GCM-SHA384            TLSv1.2                None                None
9     AES256-SHA256                TLSv1.2                None                None
10    AES256-SHA                   TLSv1,TLSv1.1,TLSv1.2  None                None
11    CAMELLIA256-SHA              TLSv1,TLSv1.1,TLSv1.2  None                None
12    ECDHE-RSA-AES128-GCM-SHA256  TLSv1.2                ECDH,P-256,256bits  prime256v1
13    ECDHE-RSA-AES128-SHA256      TLSv1.2                ECDH,P-256,256bits  prime256v1
14    ECDHE-RSA-AES128-SHA         TLSv1,TLSv1.1,TLSv1.2  ECDH,P-256,256bits  prime256v1
15    DHE-RSA-AES128-GCM-SHA256    TLSv1.2                DH,2048bits         None
16    DHE-RSA-AES128-SHA256        TLSv1.2                DH,2048bits         None
17    DHE-RSA-AES128-SHA           TLSv1,TLSv1.1,TLSv1.2  DH,2048bits         None
18    DHE-RSA-SEED-SHA             TLSv1,TLSv1.1,TLSv1.2  DH,2048bits         None
19    DHE-RSA-CAMELLIA128-SHA      TLSv1,TLSv1.1,TLSv1.2  DH,2048bits         None
20    AES128-GCM-SHA256            TLSv1.2                None                None
21    AES128-SHA256                TLSv1.2                None                None
22    AES128-SHA                   TLSv1,TLSv1.1,TLSv1.2  None                None
23    SEED-SHA                     TLSv1,TLSv1.1,TLSv1.2  None                None
24    CAMELLIA128-SHA              TLSv1,TLSv1.1,TLSv1.2  None                None
25    ECDHE-RSA-RC4-SHA            TLSv1,TLSv1.1,TLSv1.2  ECDH,P-256,256bits  prime256v1
26    RC4-SHA                      TLSv1,TLSv1.1,TLSv1.2  None                None
27    ECDHE-RSA-DES-CBC3-SHA       TLSv1,TLSv1.1,TLSv1.2  ECDH,P-256,256bits  prime256v1
28    EDH-RSA-DES-CBC3-SHA         TLSv1,TLSv1.1,TLSv1.2  DH,2048bits         None
29    DES-CBC3-SHA                 TLSv1,TLSv1.1,TLSv1.2  None                None

Certificate: <span style="color:#00cd00;">trusted</span>, <span style="color:#00cd00;">2048</span> bits, <span style="color:#00cd00;">sha256WithRSAEncryption</span> signature
TLS ticket lifetime hint: 300
NPN protocols: None
OCSP stapling: <span style="color:#cd0000;">not supported</span>
Cipher ordering: <span style="color:#cd0000;">client</span>
Curves ordering: <span style="color:#00cd00;">server</span> - fallback: <span style="color:#cd0000;">no</span>
Server <span style="color:#00cd00;">supports</span> secure renegotiation
Server supported compression methods: <span style="color:#00cd00;">NONE</span>
TLS Tolerance: <span style="color:#00cd00;">yes</span>

It’s clear that we are not doing everything right here, but what to do about it? Enter analyze.py , a small utility included with Cipherscan that can tell you which knobs to turn. It lets you check your setup against the three defined levels of compliance that Mozilla lays out: Modern, Intermediate and Old. For our specific case, intermediate will do the trick. To use it, first you need to output the results from Cipherscan as JSON to a file, and run analyze on it:

$ ./cipherscan -j internal-service.example.com:443 > foo
$ ./analyze.py foo -l intermediate
internal-service.example.com:443 has bad ssl/tls
and DOES NOT comply with the 'intermediate' level

Things that are bad:
* remove cipher ECDHE-RSA-RC4-SHA
* remove cipher RC4-SHA

Changes needed to match the intermediate level:
* remove cipher DHE-RSA-CAMELLIA256-SHA
* remove cipher CAMELLIA256-SHA
* remove cipher DHE-RSA-SEED-SHA
* remove cipher DHE-RSA-CAMELLIA128-SHA
* remove cipher SEED-SHA
* remove cipher CAMELLIA128-SHA
* remove cipher ECDHE-RSA-RC4-SHA
* remove cipher RC4-SHA
* consider enabling OCSP Stapling
* enforce server side ordering

Well then! We are using blacklisted ciphers, and a few other weak ciphers. We are not using OCSP stapling, and the client gets to dictate which ciphers we prefer. Let’s get rid of those blacklisted ciphers. We are using Nginx for this, so set it up with the ciphers found here

Also, we should use server side ordering and OCSP stapling, resulting in this configuration (generated using this nifty generator):

# intermediate configuration. tweak to your needs.
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS';
ssl_prefer_server_ciphers on;

# OCSP Stapling
ssl_stapling on;
ssl_stapling_verify on;

After reloading (i would restart) the Nginx, lets see how we perform:

$ ./cipherscan</span> -servername web.acme.customer.com 10.10.10.10:443
.......................................
Target: 10.10.10.10:443

prio  ciphersuite                  protocols              pfs                 curves
1     ECDHE-RSA-AES128-GCM-SHA256  TLSv1.2                ECDH,P-256,256bits  prime256v1
2     ECDHE-RSA-AES256-GCM-SHA384  TLSv1.2                ECDH,P-256,256bits  prime256v1
3     DHE-RSA-AES128-GCM-SHA256    TLSv1.2                DH,2048bits         None
4     DHE-RSA-AES256-GCM-SHA384    TLSv1.2                DH,2048bits         None
5     ECDHE-RSA-AES128-SHA256      TLSv1.2                ECDH,P-256,256bits  prime256v1
6     ECDHE-RSA-AES256-SHA384      TLSv1.2                ECDH,P-256,256bits  prime256v1
7     ECDHE-RSA-AES128-SHA         TLSv1,TLSv1.1,TLSv1.2  ECDH,P-256,256bits  prime256v1
8     ECDHE-RSA-AES256-SHA         TLSv1,TLSv1.1,TLSv1.2  ECDH,P-256,256bits  prime256v1
9     DHE-RSA-AES128-SHA256        TLSv1.2                DH,2048bits         None
10    DHE-RSA-AES128-SHA           TLSv1,TLSv1.1,TLSv1.2  DH,2048bits         None
11    DHE-RSA-AES256-SHA256        TLSv1.2                DH,2048bits         None
12    DHE-RSA-AES256-SHA           TLSv1,TLSv1.1,TLSv1.2  DH,2048bits         None
13    ECDHE-RSA-DES-CBC3-SHA       TLSv1,TLSv1.1,TLSv1.2  ECDH,P-256,256bits  prime256v1
14    EDH-RSA-DES-CBC3-SHA         TLSv1,TLSv1.1,TLSv1.2  DH,2048bits         None
15    AES128-GCM-SHA256            TLSv1.2                None                None
16    AES256-GCM-SHA384            TLSv1.2                None                None
17    AES128-SHA256                TLSv1.2                None                None
18    AES256-SHA256                TLSv1.2                None                None
19    AES128-SHA                   TLSv1,TLSv1.1,TLSv1.2  None                None
20    AES256-SHA                   TLSv1,TLSv1.1,TLSv1.2  None                None
21    DES-CBC3-SHA                 TLSv1,TLSv1.1,TLSv1.2  None                None

Certificate: <span style="color:#00cd00;">trusted</span>, <span style="color:#00cd00;">2048</span> bits, <span style="color:#00cd00;">sha256WithRSAEncryption</span> signature
TLS ticket lifetime hint: 300
NPN protocols: None
OCSP stapling: <span style="color:#00cd00;">supported</span>
Cipher ordering: <span style="color:#00cd00;">server</span>
Curves ordering: <span style="color:#00cd00;">server</span> - fallback: <span style="color:#cd0000;">no</span>
Server <span style="color:#00cd00;">supports</span> secure renegotiation
Server supported compression methods: <span style="color:#00cd00;">NONE</span>
TLS Tolerance: <span style="color:#00cd00;">yes</span>

Much better! Let’s see what analyze says:

$ ./cipherscan -j internal-service.example.com:443 > foo && ./analyze.py foo -l intermediate
internal-service.example.com:443 has intermediate ssl/tls
and complies with the 'intermediate' level

Compliant!

Using Nagios to verify your certificates

I like to let someone else do my work for me and Nagios (Icinga in this case) is pretty good for that. So I want to set up tests of all my SSL/TLS endpoints everywhere. For Nagios, things are pretty easy. analyze.py even has a --nagios-mode that produces the right exit codes. Here’s the script I use:

#!/bin/bash

set -euo pipefail

_fail(){
        echo $@
        exit 2
}

LEVEL=$1
SERVERNAME=$2
HOST=$3
PORT=$4

tmpfile=$(mktemp) || _fail "error creating temp file"

LOCK=/tmp/$(basename $0)-${SERVERNAME}_${HOST}.lock

trap "rm -f $LOCK $tmpfile" INT HUP TERM EXIT

test -f $LOCK && _fail "lockfile found: $LOCK"

/opt/cipherscan/cipherscan -j -servername ${SERVERNAME} ${HOST}:${PORT} > $tmpfile || \
  _fail unable to scan ${SERVERNAME} at ${HOST}:${PORT}
/opt/cipherscan/analyze.py $tmpfile -l ${LEVEL} --nagios

exit $?

I call it through NRPE like this:

command[check_cipherscan_someservice_example_com]=/usr/local/lib/monitoring/plugins/nagios_cipherscan intermediate some-service.example.com 10.10.10.10 443

That’s it!

Ørjan Ommundsen

Former Senior Systems Consultant at Redpill Linpro

Thoughts on the CrowdStrike Outage

Unless you’ve been living under a rock, you probably know that last Friday a global crash of computer systems caused by ‘CrowdStrike’ led to widespread chaos and mayhem: flights were cancelled, shops closed their doors, even some hospitals and pharmacies were affected. When things like this happen, I first have a smug feeling “this would never happen at our place”, then I start thinking. Could it?

Broken Software Updates

Our department do take responsibility for keeping quite a lot ... [continue reading]

Alarms made right

Published on June 27, 2024

Just-Make-toolbox

Published on March 22, 2024