This post appeared originally in our sysadvent series and has been moved here following the discontinuation of the sysadvent microsite
The network is a very proprietary place. When you buy an IP router or an Ethernet switch, what you’re really buying is a tightly integrated bundle of hardware and software.
Mixing and matching software and hardware components in order to design a network infrastructure tailored to your precise set of requirements is something that has previously been completely unheard of.
For example, imagine that you’ve found a Cisco Systems switch that has the exact hardware specifications you’re after. However, also imagine your use case requires a software feature unique to Juniper Networks’s Junos NOS. If you told the sales representatives of those two companies that you would like combine their two products, odds are they would walk out of the meeting.
This would certainly never be accepted in other parts of the data centre. Imagine if a server vendor such as Dell told you that sure, you’re free to buy their servers - but with the condition that you could only use with Dell operating system, that could only run a Dell web server or database software, and so on. In that sales meeting, the techies would be the ones leaving.
It’s all just merchant silicon, silly
It is not really a secret that most Ethernet switches being sold today are built using only commodity off-the-shelf components. That is, the NOS runs in an general-purpose computer equipped with a standard ARM or x86 CPUs that controls the ASIC that handles the actual packet forwarding. The ASIC is what is called «merchant silicon»; a commodity chipset manufactured by companies such as Broadcom or Marvell.
The ASIC is what determines the performance and capabilities of the switch. Therefore, if the ASIC is the same, the biggest hardware difference between two switch models from competing vendors might be the mix of interfaces or simply the company logos printed on the front!
In this case, the capabilities and features of the NOS are the only real distinguishing factors. The vendors know very well that if you need a feature in their software to begin with, they essentially have a monopoly on the accompanying hardware - and they will most certainly charge you accordingly!
The emergence of bare-metal switches and open networking
The large data centre operators were far from happy with the status quo. They needed more open, flexible and cost-effective solutions, and the market responded. Today, even big-name vendors such as Dell and HPE offer bare-metal switches to any customer who knows to ask for them.
HPE Norway has in fact graciously lent us a HPE Altoline 6920 switch:
This Altoline has a nice spec of 48x 10GB SFP+ ports with 6x 40GB QSFP+ up-link ports. It has now been complemented by the HPE Altoline 6921, which contains the Broadcom Trident II ASIC.
ONIE
A bare-metal switch is essentially just a switch built using merchant silicon that is delivered without a pre-installed network operating system. Instead, the only piece of software it comes with is typically the Open Network Install Environment (ONIE). This is essentially just a boot-loader that lets you easily install your NOS of choice onto the switch:
There are already several different network operating systems available, both commercial and open-source. The choice is finally ours! Here’s a non-exhaustive list:
NOS installation in practice
This demonstration shows how easy it is to install a NOS onto a bare-metal switch. It is a serial console recording that shows OpenSwitch being installed onto a HPE Altoline 6920 we’ve got sitting in a lab.
We went for the simplest network installation mode out of the many supported by ONIE:
- First, we configured the
eth1interface one of our lab nodes with the IPv4 address192.168.0.1/24and started a simple DHCP service (dnsmasq -di eth1 -F 192.168.0.2,192.168.0.254). We then connectedeth1directly to the management interface of the switch. - Second, we installed a simple web server (
apt-get install apache2) and downloaded the appropriate OpenSwitch ONIE installation image to a file named simplyonie-installerin its root directory.
With those simple preparations out of the way, all that remained was to power up the switch and observe the automatic installation taking place. About two minutes later, OpenSwitch was installed and ready to use:
(You can click the recording to see it from the beginning.)
Summary
It is adamantly clear that the future of our network is open.
Using bare-metal switches in combination with a freely chosen NOS, we gain the freedom to automate all aspects of the network using our Puppet framework, install our own monitoring software Munin directly on the network equipment, and use our favourite routing software suite BIRD instead of being forced to use the vendor-provided one.
These are just a few examples of possibilities any Linux server administrator will take for granted, but which for a network administrator is totally revolutionary.
Redpill Linpro was originally founded on the ideas of freedom and flexibility offered by open-source software. Over twenty years later, we are finally in a position to bring those exact same liberties to our network infrastructure. It is certainly an exciting time to be a network architect!
Update
- Updated Links to HPE switches.
Thoughts on the CrowdStrike Outage
Unless you’ve been living under a rock, you probably know that last Friday a global crash of computer systems caused by ‘CrowdStrike’ led to widespread chaos and mayhem: flights were cancelled, shops closed their doors, even some hospitals and pharmacies were affected. When things like this happen, I first have a smug feeling “this would never happen at our place”, then I start thinking. Could it?
Broken Software Updates
Our department do take responsibility for keeping quite a lot ... [continue reading]