iPXE and automated provisioning

This post appeared originally in our sysadvent series and has been moved here following the discontinuation of the sysadvent microsite

Provisioning of new servers can be a daunting experience. Back in days it meant booting the machine with a CD or a DVD and doing manual choices. Automation of the installation process makes the process faster and less prone to human errors.

Network installation helps the process, but you still need to know the hardware to be able to automate provisioning.

When dealing with Virtual Machines, you decide the parameters for the hardware so the machines can be defined by recognizable parameters for the installation and configuration systems.

Our solution consists of many different parts that combined make a whole.

• DNS server
• DHCP server
• iPXE
• Apache
• KVM

iPXE

The iPXE project, ipxe.org gives us a lot of flexibility. What we have done to make it fit our needs is compile it with a small change to src/config/general.h to enable IPv6 support:

#undef NET_PROTO_IP

to

#define NET_PROTO_IP

With this myscript.ipxe in the src folder

#!ipxe

ifconf --configurator dhcp net0 && goto ipv4boot || goto ipv6boot


:ipv4boot
 echo ========================================================
 echo UUID: ${uuid}
 echo Manufacturer: ${manufacturer}
 echo Product name: ${product}
 echo Hostname: ${hostname}
 echo
 echo MAC address: ${net0/mac}
 echo IP address: ${net0/ip}
 echo IPv6 address: ${net0.ndp.0/ip6:ipv6}
 echo Netmask: ${net0/netmask}
 echo
 echo Gateway: ${gateway}
 echo DNS: ${dns}
 echo IPv6 DNS: ${dns6}
 echo Domain: ${domain}
 echo ========================================================
 chain --replace --autofree http://boot.example.com/mac/${mac:hexhyp}/ipxe ||
 chain --replace --autofree http://boot.example.com/pxedust || goto notfound

:ipv6boot
 clear ip6
 ifconf --configurator ipv6
 show ip6
 isset ${net0.ndp.0/ip6:ipv6} || goto ipv6boot
 echo ========================================================
 echo UUID: ${uuid}
 echo Manufacturer: ${manufacturer}
 echo Product name: ${product}
 echo Hostname: ${hostname}
 echo
 echo MAC address: ${net0/mac}
 echo IP address: ${net0/ip}
 echo IPv6 address: ${net0.ndp.0/ip6:ipv6}
 echo Netmask: ${net0/netmask}
 echo
 echo Gateway: ${gateway}
 echo DNS: ${dns}
 echo IPv6 DNS: ${dns6}
 echo Domain: ${domain}
 echo ========================================================
 chain --replace --autofree http://boot.example.com/pxedust || goto notfound

:notfound
  echo
  echo No netboot configuration was found for this machine. Please go to
  echo http://boot-master.example.com/mac/${mac:hexraw} to configure it if
  echo needed.
  echo
  echo Skipping to the next boot device according to the BIOS Boot Order.
  exit

we then build the 1af41000 module, with the script embedded by running:

make bin/1af41000.rom EMBED=myscript.ipxe

Copy that bin/1af41000.rom blob to /usr/share/gpxe/virtio-net.rom your EL6 KVM server, or /usr/share/ipxe/1af41000.rom on EL7.

This script tries IPv4 DHCP first, and falls back to IPv6 SLAAC if there is no DHCP answer.

When defining the network card for the Virtual Machine like this

<devices>
...
  <interface type='bridge'>
    <mac address='02:04:06:08:73:ac'/>
    <source bridge='br135'/>
    <model type='virtio'/>
  </interface>
...
</devices>

then you will be able to boot it into the code you wrote in myscript.ipxe

IPv4

MAC address for IPv4 DHCP

When using IPv4 we need to decide the IP address first, and map it to DNS.

And using IPv4 means we need DHCP. For DHCP, we need predicable MAC address. We base our MAC address on a the last 2 octets of the IP address and the VLAN number. With the assumption that no VLAN is bigger than /23 we can calculate an unique MAC for every IPv4 address in our net.

With the freely chosen prefix 02:04:06, the python function for creating the mac is as follows:

def genStrictMAC(vlan, ip):
    first        = int(vlan) >> 4
    firstsecond  = int(vlan) & 0xf
    secondsecond = (int(ip.split('.')[2])) & 0xf
    third        = int(ip.split('.')[3])
    mac          = "02:04:06:%02x:%x%x:%02x" % (first,
                                                firstsecond,
                                                secondsecond,
                                                third)
    return mac

DHCP configuration for predetermined MAC addresses

For all VLANs that are assigned to have provisioned virtual machines through this system, we have a cron job that does following:

for every IPv4 address in the range:
  check revers DNS for address:
  if there is a reverse record for the address and that resolves back to the address:
    create DHCP configuration for the address, and point to iPXE
  else
    next

IPv6

MAC address for the IPv6

For IPv6 we use SLAAC https://en.wikipedia.org/wiki/IPv6_address#Stateless_address_autoconfiguration

So there we make a random MAC and calculate the IPv6 address from that, then add that address to DNS, with corresponding reverse entry.

Assuming that the IPv6 prefix is from a /64 net then the python function to create the IPv6 suffix is as follows (with randomly chosen 2a:4a:6a as prefix of the MAC address)

import re
import random
def genMAC():
    randomNumber = random.getrandbits(24)
    randomBits   = re.sub(r'(?<=..)(..)', r':\1',format(randomNumber, '06X')).lower()
    mac          = "%s%s" % ("2a:4a:6a:", randomBits )
    randomSlaac  = "%s%s" % ("284a6afffe", format(randomNumber, '06X').lower())
    slaacSuffix  = re.sub(r'(?<=....)(....)', r':\1', randomSlaac)
    return mac, slaacSuffix

Note the change from 2a to 28. See https://en.wikipedia.org/wiki/IPv6_address#Modified_EUI-64

Virtual Machine definition.

In the XML for the VM there is one thing worth nothing

<os>
  <type arch='x86_64' machine='pc'>hvm</type>
  <bios useserial='yes'/>
  <boot dev='hd'/>
  <boot dev='network'/>
</os>

The boot lines mean that the VM first tries to boot from its hard disk, if that fails, boot from the network. That will invoke the installation routine.

If you have a Virtual Machine on VLAN 135 with the IP of 192.168.35.172, the configuration of the network card is something like this:

<devices>
...
  <interface type='bridge'>
    <mac address='02:04:06:08:73:ac'/>
    <source bridge='br135'/>
    <model type='virtio'/>
  </interface>
...
</devices>

Apache and the boot configurations

Traditionally we had a web server filled with folders named after the the MAC address with hyphens as separator, that is for the MAC 2a:4a:6a:74:55:06 we had the 2a-4a-6a-74-55-06 folder with an iPXE script, residing in that folder, pointing to the installation medium.

From the embedded iPXE script above, the line

chain --replace --autofree http://boot.example.com/mac/${mac:hexhyp}/ipxe ||

chain --replace --autofree http://boot.example.com/pxedust || goto notfound

Here the chain command fetches the code from the URL, and runs it, keeping the variables it has defined so far.

The pxedust script on the web server checks if the IP address accessing it resolves, and checks if that name resolves back to the address. And then it checks if the name has an TXT field with the text “filename=script.ipxe”. That (plus some more information from the host-name) decides which iPXE script is sent from the Apache server. That script can set parameters for other iPXE scripts that get chain loaded.

Here from an actual node:

jonas@test-centos7:~> dig +short txt test-centos7.example.com
"filename=test.ipxe"
"distribution=core"

Our pxedust uses that:

jonas@test-centos7:~> GET http://boot.example.com/pxedust
#!ipxe

set distribution core
chain --replace --autofree http://boot.example.com/bootstrap/example/test.ipxe

The test.ipxe contains local configuration, and then a call for the final iPXE file.

jonas@test-centos7:~> GET http://boot.example.com/bootstrap/example/test.ipxe
#!ipxe

set puppet_server puppet.example.com
set puppet_environment ipxe_test_master

chain --replace --autofree http://boot.example.com/bootstrap/install/default.ipxe

And then the final file, here is an excerpt of ours

#!ipxe
#
isset ${distribution} || goto no_distribution

isset ${console}      || set console console=ttyS0,115200
isset ${arch}         || set arch x86_64
isset ${ifnames}      || set ifnames 0

iseq ${distribution} core   && goto centos ||
iseq ${distribution} xenial && goto ubuntu ||
goto unsupported_distribution

:centos
set distribution centos/7
isset ${pxeboot}   || set pxeboot http://repo.example.com/${distribution}/os/${arch}/images/pxeboot
isset ${kickstart} || set kickstart http://boot.example.com/bootstrap/install/${distribution}/${arch}/ks.cfg

set vmlinuz ${pxeboot}/vmlinuz
set initrd  initrd.img
set cmdline ks=${kickstart}

goto boot

:ubuntu
isset ${pxeboot}   || set pxeboot http://repo.example.com/ubuntu-installer/${distribution}/amd64
isset ${preseed}   || set preseed http://boot.example.com/bootstrap/install/${distribution}/${arch}/preseed.cfg
set vmlinuz ${pxeboot}/linux
set initrd  initrd.gz
set cmdline DEBIAN_FRONTEND=text auto interface=auto url=${preseed} hostname=${hostname} domain=${domain} locale=en_US console-setup/ask_detect=false keyboard-configuration/layoutcode=no recommends=false tasks=

goto boot

:boot
isset ${firstboot} || set firstboot http://boot.example.com/bootstrap/install/${distribution}/firstboot
set cmdline ${cmdline} net.ifnames=${ifnames} firstboot=${firstboot} ${console}
isset ${puppet_server}      && set cmdline ${cmdline} puppet_server=${puppet_server} ||
isset ${puppet_environment} && set cmdline ${cmdline} puppet_environment=${puppet_environment} ||
isset ${extra}              && set cmdline ${cmdline} ${extra} ||

kernel ${vmlinuz} ${cmdline}
initrd ${pxeboot}/${initrd}

boot

:no_distribution
echo No distribution specified, aborting...
exit

:unsupported_distribution
echo Unsupported distribution specified, aborting...
exit

The firstboot script that is mentioned in this file uses the fact that the content of ${cmdline} is found in /proc/cmdline after boot. That script can then install Puppet and configure Puppet for its initial run from the values from iPXE.

Installing the Virtual Machine

Now we have covered the parts, then it comes together when installing a Virtual Machine. We do not cover here the provisioning of storage.

We need a name for the machine. And predefined iPXE script for installing the distribution of choice.

If we want IPv6 only then the only thing else that we need is the VLAN it should reside in. From the VLAN we get the IPv6 prefix. Our scripts will then:

1. create MAC address
2. from that, determine IPv6 address
3. update DNS with that IPv6 address
4. update the DNS with TXT field pointing to the desired installation iPXE
5. create the XML for the VM, with defined VLAN bridge and MAC address
6. boot the VM

For IPv4 we need to know the IP address reserved for the machine. Our scripts then:

1. from the IP finds the VLAN.
2. from the IP and the VLAN creates MAC address
3. update DNS with name and IP
4. update the DNS with TXT field pointing to the desired installation iPXE
5. create the XML for the VM, with defined VLAN bridge and MAC address
6. boot the VM