Using Let's Encrypt with OpenShift

This post appeared originally in our sysadvent series and has been moved here following the discontinuation of the sysadvent microsite

When installing OpenShift, the default certificates that are being installed are self-certified. Although this gives you functional encryption, this is in no way best practice and is especially annoying for the route being exposed for the Hawkular metrics, which is integrated within the Web console.

Luckily there is a relatively easy (and did I mention it is free?) solution: use Let’s Encrypt as your certificate authority.

Installing Let’s Encrypt

Let’s Encrypt is an automated and open certificate authority (CA), run for the public’s benefit. The goal is to create a more secure and privacy-respecting web. Let’s Encrypt achieves automation by using software that uses the ACME protocol, which typically runs on your web host.

In this example we will use the Certbot ACME client. We have it installed from EPEL on our loadbalancer, which is running HAProxy, with:

yum -y install certbot

NOTE On RHEL you need to install python-pyOpenSSL from the OpenStack Cloud SIG to upgrade to a version of pyOpenSSL which is supported by Certbot.

Configure HAProxy for Let’s Encrypt

Adjust the configuration of HAProxy on the frontend loadbalancer by adding the following:

frontend atomic-openshift-http
  bind *:80
  acl letsencrypt-request path_beg -i /.well-known/acme-challenge/
  use_backend letsencrypt if letsencrypt-request
  default_backend atomic-openshift-http

backend letsencrypt
  server letsencrypt 127.0.0.1:8080

And reload the updated configuration:

systemctl reload haproxy.service

This enables Certbot to answer the HTTP-01 challenges from Let’s Encrypt, thereby confirming our identity.

Requesting a certificate

We can now request a certificate from Let’s Encrypt by using:

certbot certonly --standalone --preferred-challenges http --http-01-port 8080 --http-01-address 127.0.0.1 -m admin@example.com -d ocp.example.com

If everything is configured correctly, you end up with a private key and the certificates under /etc/letsencrypt/live/ocp.example.com/:

/etc/letsencrypt/live/ocp.example.com/cert.pem
/etc/letsencrypt/live/ocp.example.com/privkey.pem
/etc/letsencrypt/live/ocp.example.com/chain.pem
/etc/letsencrypt/live/ocp.example.com/fullchain.pem

Certificate used by OpenShift API

We can now proceed to install and use the certificate from Let’s Encrypt for the OpenShift API. This is done by adding the following to your Ansible inventory file for OpenShift:

openshift_master_named_certificates=[{"certfile":"/etc/letsencrypt/live/ocp.example.com/fullchain.pem","keyfile":"/etc/letsencrypt/live/ocp.example.com/privkey.pem"}]
openshift_master_overwrite_named_certificates=true

and (re-)running the Ansible playbook to configure OpenShift, thereby installing the named certificate on your masters:

ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/config.yml

After this, you should no longer receive any warnings about an unknown certificate authority within your web-browser or from the oc command line interface.

Fixing the certificate for Hawkular metrics

Request the certificate for Hawkular metrics with:

certbot certonly --standalone --preferred-challenges http --http-01-port 8080 -m admin@example.com -d hawkular-metrics.cloudapps.example.com

Installation of the private key and the certificate chain from Let’s Encrypt into the hawkular-metrics deployment is done by using the following script:

#!/bin/bash -e

LINEAGE=${1:-/etc/letsencrypt/live/cloudapps.example.com}
PROJECT=${2:-openshift-infra}
ROUTE=${3:-hawkular-metrics}

update_tls() {
  type=${1}
  case ${type} in
    key)
      pem="${2}/privkey.pem"
    ;;
    certificate)
      pem="${2}/cert.pem"
    ;;
    caCertificate)
      pem="${2}/chain.pem"
    ;;
  esac
  oc="oc -n ${3}"
  res="routes/${4}"
  if ! diff -q ${pem} <(${oc} get ${res} --template '{{.spec.tls.'${type}'}}')
  then
    echo "Updating key from ${pem}..."
    ${oc} patch ${res} --patch "{\"spec\":{\"tls\":{\"${type}\":\"$(awk '{printf "%s\\n",$0}' ${pem})\"}}}"
  fi
}

update_tls key ${LINEAGE} ${PROJECT} ${ROUTE}
update_tls certificate ${LINEAGE} ${PROJECT} ${ROUTE}
update_tls caCertificate ${LINEAGE} ${PROJECT} ${ROUTE}

exit 0

Save this script as /usr/local/sbin/update-letsencrypt-certificate and make it executable:

chmod +x /usr/local/sbin/update-letsencrypt-certificate

Keeping the certificates fresh

Since Let’s Encrypt only offers certificates with a limited lifetime of ninety days, it is very important to automate the renewal of the certificates to prevent the use of expired (and therefore invalid) certificates.

To make sure renewals of certificates are requested from Let’s Encrypt, a systemd timer-unit is available. To enable this feature use:

systemctl enable --now certbot-renew.timer

Also make sure you add the endpoints using those certificates to your monitoring system to get notified in case there is a problem with the automated update procedure.

Updating the OpenShift certificates

Once certificates are renewed from Let’s Encrypt, Certbot has a deployment hook available to update a certificate within the components that use them. To utilize this hook for the certificates used in this example, install the following script as /etc/letsencrypt/renewal-hooks/deploy/openshift:

#!/bin/bash -e

for domain in ${RENEWED_DOMAINS}
do
  case ${domain} in
    ocp.example.com)
      ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/redeploy-master-certificates.yml
    ;;
    hawkular-metrics.cloudapps.example.com)
      /usr/local/sbin/update-letsencrypt-certificate ${RENEWED_LINEAGE}
    ;;
  esac
done

exit 0

Remember to set the eXecutable bit on this script:

chmod +x /etc/letsencrypt/renewal-hooks/deploy/openshift

The wilcard certificate

As of this writing, support for wildcard certificates is not provided by Let’s Encrypt. This means that support for routes established using the DNS wildcard entry for OpenShift is currently not possible by following this procedure.

However, Let’s Encrypt has announced that they will include support for wildcard certificates from January 2018!

Finally

The above is just one way to fix the challenges of having proper certificates installed on your main OpenShift infrastructure, and is easily extended to include multiple components of an OpenShift installation. This is left as an exercise for the reader.