This post appeared originally in our sysadvent series and has been moved here following the discontinuation of the sysadvent microsite
If you rely on SSL/TLS certificates and you have a slew of services to maintain online, things can quickly get out of hand. If you don’t have the time or the resources to keep up to speed with what ciphers to disable or what techniques to employ server-side, you might quickly fall prey to the next “Exploit with a Logo”. Heartbleed, Beast, Poodle and friends come to mind.
The guys at Mozilla have taken measures to give all of us sysadmins more free time by maintaining lists of recommended steps to take server-side.
In addition they’ve also created Cipherscan, which is a great little tool to check if your sites comply with the latest security advisories.
Verify new certificates and ciphersuites on a test instance
One of my favorite features is that I can leverage SNI to check a new certificate on an arbitrary IP. This takes all the tension out of rolling out updated certificates every year or two, because I can verify that the new certificate has a sane certificate chain and does what it’s supposed to before I expose it to production traffic.
$ ./cipherscan --servername web.acme.customer.com 10.10.10.10:4435
.......................................
Target: 10.10.10.10:443
prio ciphersuite protocols pfs curves
1 ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 ECDH,P-256,256bits prime256v1
2 ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 ECDH,P-256,256bits prime256v1
3 DHE-RSA-AES128-GCM-SHA256 TLSv1.2 DH,2048bits None
4 DHE-RSA-AES256-GCM-SHA384 TLSv1.2 DH,2048bits None
5 ECDHE-RSA-AES128-SHA256 TLSv1.2 ECDH,P-256,256bits prime256v1
6 ECDHE-RSA-AES256-SHA384 TLSv1.2 ECDH,P-256,256bits prime256v1
7 ECDHE-RSA-AES128-SHA TLSv1,TLSv1.1,TLSv1.2 ECDH,P-256,256bits prime256v1
8 ECDHE-RSA-AES256-SHA TLSv1,TLSv1.1,TLSv1.2 ECDH,P-256,256bits prime256v1
9 DHE-RSA-AES128-SHA256 TLSv1.2 DH,2048bits None
10 DHE-RSA-AES128-SHA TLSv1,TLSv1.1,TLSv1.2 DH,2048bits None
11 DHE-RSA-AES256-SHA256 TLSv1.2 DH,2048bits None
12 DHE-RSA-AES256-SHA TLSv1,TLSv1.1,TLSv1.2 DH,2048bits None
13 ECDHE-RSA-DES-CBC3-SHA TLSv1,TLSv1.1,TLSv1.2 ECDH,P-256,256bits prime256v1
14 EDH-RSA-DES-CBC3-SHA TLSv1,TLSv1.1,TLSv1.2 DH,2048bits None
15 AES128-GCM-SHA256 TLSv1.2 None None
16 AES256-GCM-SHA384 TLSv1.2 None None
17 AES128-SHA256 TLSv1.2 None None
18 AES256-SHA256 TLSv1.2 None None
19 AES128-SHA TLSv1,TLSv1.1,TLSv1.2 None None
20 AES256-SHA TLSv1,TLSv1.1,TLSv1.2 None None
21 DES-CBC3-SHA TLSv1,TLSv1.1,TLSv1.2 None None
Certificate: <span style="color:#00cd00;">trusted</span>, <span style="color:#00cd00;">2048</span> bits, <span style="color:#00cd00;">sha256WithRSAEncryption</span> signature
TLS ticket lifetime hint: 300
NPN protocols: None
OCSP stapling: <span style="color:#00cd00;">supported</span>
Cipher ordering: <span style="color:#00cd00;">server</span>
Curves ordering: <span style="color:#00cd00;">server</span> - fallback: <span style="color:#cd0000;">no</span>
Server <span style="color:#00cd00;">supports</span> secure renegotiation
Server supported compression methods: <span style="color:#00cd00;">NONE</span>
TLS Tolerance: <span style="color:#00cd00;">yes</span>
This checks the certificate and ciphers for web.acme.customer.com on a test instance at 10.10.10.10 (of course the test server needs to have the same web server setup as production).
Testing and fixing a bad setup
We have internal-service.example.com serving highly important pictures of lolcats to all employees, and we want to check if it’s up to scratch:
$ ./cipherscan</span> internal-service.example.com:443
.......................................
Target: internal-service.example.com:443
prio ciphersuite protocols pfs curves
1 ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 ECDH,P-256,256bits prime256v1
2 ECDHE-RSA-AES256-SHA384 TLSv1.2 ECDH,P-256,256bits prime256v1
3 ECDHE-RSA-AES256-SHA TLSv1,TLSv1.1,TLSv1.2 ECDH,P-256,256bits prime256v1
4 DHE-RSA-AES256-GCM-SHA384 TLSv1.2 DH,2048bits None
5 DHE-RSA-AES256-SHA256 TLSv1.2 DH,2048bits None
6 DHE-RSA-AES256-SHA TLSv1,TLSv1.1,TLSv1.2 DH,2048bits None
7 DHE-RSA-CAMELLIA256-SHA TLSv1,TLSv1.1,TLSv1.2 DH,2048bits None
8 AES256-GCM-SHA384 TLSv1.2 None None
9 AES256-SHA256 TLSv1.2 None None
10 AES256-SHA TLSv1,TLSv1.1,TLSv1.2 None None
11 CAMELLIA256-SHA TLSv1,TLSv1.1,TLSv1.2 None None
12 ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 ECDH,P-256,256bits prime256v1
13 ECDHE-RSA-AES128-SHA256 TLSv1.2 ECDH,P-256,256bits prime256v1
14 ECDHE-RSA-AES128-SHA TLSv1,TLSv1.1,TLSv1.2 ECDH,P-256,256bits prime256v1
15 DHE-RSA-AES128-GCM-SHA256 TLSv1.2 DH,2048bits None
16 DHE-RSA-AES128-SHA256 TLSv1.2 DH,2048bits None
17 DHE-RSA-AES128-SHA TLSv1,TLSv1.1,TLSv1.2 DH,2048bits None
18 DHE-RSA-SEED-SHA TLSv1,TLSv1.1,TLSv1.2 DH,2048bits None
19 DHE-RSA-CAMELLIA128-SHA TLSv1,TLSv1.1,TLSv1.2 DH,2048bits None
20 AES128-GCM-SHA256 TLSv1.2 None None
21 AES128-SHA256 TLSv1.2 None None
22 AES128-SHA TLSv1,TLSv1.1,TLSv1.2 None None
23 SEED-SHA TLSv1,TLSv1.1,TLSv1.2 None None
24 CAMELLIA128-SHA TLSv1,TLSv1.1,TLSv1.2 None None
25 ECDHE-RSA-RC4-SHA TLSv1,TLSv1.1,TLSv1.2 ECDH,P-256,256bits prime256v1
26 RC4-SHA TLSv1,TLSv1.1,TLSv1.2 None None
27 ECDHE-RSA-DES-CBC3-SHA TLSv1,TLSv1.1,TLSv1.2 ECDH,P-256,256bits prime256v1
28 EDH-RSA-DES-CBC3-SHA TLSv1,TLSv1.1,TLSv1.2 DH,2048bits None
29 DES-CBC3-SHA TLSv1,TLSv1.1,TLSv1.2 None None
Certificate: <span style="color:#00cd00;">trusted</span>, <span style="color:#00cd00;">2048</span> bits, <span style="color:#00cd00;">sha256WithRSAEncryption</span> signature
TLS ticket lifetime hint: 300
NPN protocols: None
OCSP stapling: <span style="color:#cd0000;">not supported</span>
Cipher ordering: <span style="color:#cd0000;">client</span>
Curves ordering: <span style="color:#00cd00;">server</span> - fallback: <span style="color:#cd0000;">no</span>
Server <span style="color:#00cd00;">supports</span> secure renegotiation
Server supported compression methods: <span style="color:#00cd00;">NONE</span>
TLS Tolerance: <span style="color:#00cd00;">yes</span>
It’s clear that we are not doing everything right here, but what to do about it? Enter analyze.py , a small utility included with Cipherscan that can tell you which knobs to turn. It lets you check your setup against the three defined levels of compliance that Mozilla lays out: Modern, Intermediate and Old. For our specific case, intermediate will do the trick. To use it, first you need to output the results from Cipherscan as JSON to a file, and run analyze on it:
$ ./cipherscan -j internal-service.example.com:443 > foo
$ ./analyze.py foo -l intermediate
internal-service.example.com:443 has bad ssl/tls
and DOES NOT comply with the 'intermediate' level
Things that are bad:
* remove cipher ECDHE-RSA-RC4-SHA
* remove cipher RC4-SHA
Changes needed to match the intermediate level:
* remove cipher DHE-RSA-CAMELLIA256-SHA
* remove cipher CAMELLIA256-SHA
* remove cipher DHE-RSA-SEED-SHA
* remove cipher DHE-RSA-CAMELLIA128-SHA
* remove cipher SEED-SHA
* remove cipher CAMELLIA128-SHA
* remove cipher ECDHE-RSA-RC4-SHA
* remove cipher RC4-SHA
* consider enabling OCSP Stapling
* enforce server side ordering
Well then! We are using blacklisted ciphers, and a few other weak ciphers. We are not using OCSP stapling, and the client gets to dictate which ciphers we prefer. Let’s get rid of those blacklisted ciphers. We are using Nginx for this, so set it up with the ciphers found here
Also, we should use server side ordering and OCSP stapling, resulting in this configuration (generated using this nifty generator):
# intermediate configuration. tweak to your needs.
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS';
ssl_prefer_server_ciphers on;
# OCSP Stapling
ssl_stapling on;
ssl_stapling_verify on;
After reloading (i would restart) the Nginx, lets see how we perform:
$ ./cipherscan</span> -servername web.acme.customer.com 10.10.10.10:443
.......................................
Target: 10.10.10.10:443
prio ciphersuite protocols pfs curves
1 ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 ECDH,P-256,256bits prime256v1
2 ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 ECDH,P-256,256bits prime256v1
3 DHE-RSA-AES128-GCM-SHA256 TLSv1.2 DH,2048bits None
4 DHE-RSA-AES256-GCM-SHA384 TLSv1.2 DH,2048bits None
5 ECDHE-RSA-AES128-SHA256 TLSv1.2 ECDH,P-256,256bits prime256v1
6 ECDHE-RSA-AES256-SHA384 TLSv1.2 ECDH,P-256,256bits prime256v1
7 ECDHE-RSA-AES128-SHA TLSv1,TLSv1.1,TLSv1.2 ECDH,P-256,256bits prime256v1
8 ECDHE-RSA-AES256-SHA TLSv1,TLSv1.1,TLSv1.2 ECDH,P-256,256bits prime256v1
9 DHE-RSA-AES128-SHA256 TLSv1.2 DH,2048bits None
10 DHE-RSA-AES128-SHA TLSv1,TLSv1.1,TLSv1.2 DH,2048bits None
11 DHE-RSA-AES256-SHA256 TLSv1.2 DH,2048bits None
12 DHE-RSA-AES256-SHA TLSv1,TLSv1.1,TLSv1.2 DH,2048bits None
13 ECDHE-RSA-DES-CBC3-SHA TLSv1,TLSv1.1,TLSv1.2 ECDH,P-256,256bits prime256v1
14 EDH-RSA-DES-CBC3-SHA TLSv1,TLSv1.1,TLSv1.2 DH,2048bits None
15 AES128-GCM-SHA256 TLSv1.2 None None
16 AES256-GCM-SHA384 TLSv1.2 None None
17 AES128-SHA256 TLSv1.2 None None
18 AES256-SHA256 TLSv1.2 None None
19 AES128-SHA TLSv1,TLSv1.1,TLSv1.2 None None
20 AES256-SHA TLSv1,TLSv1.1,TLSv1.2 None None
21 DES-CBC3-SHA TLSv1,TLSv1.1,TLSv1.2 None None
Certificate: <span style="color:#00cd00;">trusted</span>, <span style="color:#00cd00;">2048</span> bits, <span style="color:#00cd00;">sha256WithRSAEncryption</span> signature
TLS ticket lifetime hint: 300
NPN protocols: None
OCSP stapling: <span style="color:#00cd00;">supported</span>
Cipher ordering: <span style="color:#00cd00;">server</span>
Curves ordering: <span style="color:#00cd00;">server</span> - fallback: <span style="color:#cd0000;">no</span>
Server <span style="color:#00cd00;">supports</span> secure renegotiation
Server supported compression methods: <span style="color:#00cd00;">NONE</span>
TLS Tolerance: <span style="color:#00cd00;">yes</span>
Much better! Let’s see what analyze says:
$ ./cipherscan -j internal-service.example.com:443 > foo && ./analyze.py foo -l intermediate
internal-service.example.com:443 has intermediate ssl/tls
and complies with the 'intermediate' level
Compliant!
Using Nagios to verify your certificates
I like to let someone else do my work for me and Nagios (Icinga in this case) is pretty good for that. So I want to set up tests of all my SSL/TLS endpoints everywhere. For Nagios, things are pretty easy. analyze.py even has a --nagios-mode that produces the right exit codes. Here’s the script I use:
#!/bin/bash
set -euo pipefail
_fail(){
echo $@
exit 2
}
LEVEL=$1
SERVERNAME=$2
HOST=$3
PORT=$4
tmpfile=$(mktemp) || _fail "error creating temp file"
LOCK=/tmp/$(basename $0)-${SERVERNAME}_${HOST}.lock
trap "rm -f $LOCK $tmpfile" INT HUP TERM EXIT
test -f $LOCK && _fail "lockfile found: $LOCK"
/opt/cipherscan/cipherscan -j -servername ${SERVERNAME} ${HOST}:${PORT} > $tmpfile || \
_fail unable to scan ${SERVERNAME} at ${HOST}:${PORT}
/opt/cipherscan/analyze.py $tmpfile -l ${LEVEL} --nagios
exit $?
I call it through NRPE like this:
command[check_cipherscan_someservice_example_com]=/usr/local/lib/monitoring/plugins/nagios_cipherscan intermediate some-service.example.com 10.10.10.10 443
That’s it!
Thoughts on the CrowdStrike Outage
Unless you’ve been living under a rock, you probably know that last Friday a global crash of computer systems caused by ‘CrowdStrike’ led to widespread chaos and mayhem: flights were cancelled, shops closed their doors, even some hospitals and pharmacies were affected. When things like this happen, I first have a smug feeling “this would never happen at our place”, then I start thinking. Could it?
Broken Software Updates
Our department do take responsibility for keeping quite a lot ... [continue reading]