As mentioned in my previous post, we have some interesting projects involving Hyper-V. One of them being a way to automatically verify that our Windows server backups are functional.
This post is about how we’ve set that up.
TLDR: check the script on GitHub
Windows backup on OpenStack
Our main virtualization platform is OpenStack, which is awesome. And while we try to mostly utilize open-source solutions, there are times when that’s just not the best way to go. Certain software need to run on windows server, and windows is stateful code. Which means we can’t just base ourselves on file backups for anything running on windows, we need full image backups to be able to do reasonably quick restores.
So, to back up our windows servers, we use Veeam Backup and Replication. While Veeam has very good integration with VMware, Hyper-V and Nutanix, it still doesn’t have any integration into OpenStack. Which is why we’re using Veeam Agent for Windows to backup both our virtual and our physical windows servers.
Now, if we had our VMs running on VMware or Hyper-V, Veeam has a built-in backup verification solution they call SureBackup. Where you can actually start up groups of VMs with cross-dependencies to check that whole applications work as they should after a restore. This option is not available for machines backed up with Veeam agent.
Something that is available, is Instant Restore to Hyper-V. Meaning you can do a Physical-to-Virtual conversion with the use of a backup. Or in our case, Virtual-to-Virtual (OpenStack to Hyper-V)
Instant Restore to Hyper-V
Since we were able to do instant restores of the backups into Hyper-V, we first needed a Hyper-V server to serve as a target host. This is how my previous post about how to run Hyper-V inside of KVM came to be.
Once we had a target server, it was a straight forward procedure to do an Instant Restore through the Veeam GUI. But that was a manual process, and we want this to be automatic.
Enter PowerShell.
Searching the Veeam PowerShell documentation I could only find the commands for Instant restoring VMware VMs into VMware: Start-VBRInstantRecovery and for restoring Hyper-V VMs into Hyper-V: Start-VBRHvInstantRecovery Neither one of these worked for machines backed up with Veeam agent.
But knowing that you can restore through the GUI, I ended up contacting Veeam support, and they were super helpful finding the command for me: Start-VBREpInstantRecovery
With this command I was finally able to do a scripted instant recovery of a machine backed up from OpenStack into Hyper-V
$restoreHost = ***name of Hyper-V server you want to restore to***
$restorePath = ***Path accessible to the Hyper-V server where you want the VMfiles and empty VHDX files to be stored***
$restorepoint = Get-VBRRestorePoint -Name ***name of machine you want to restore***
#will give you an array of restorepoints for the machine you want to restore
Start-VBREpInstantRecovery -Server $restoreHost -RestorePoint $restorepoint[$restorepoint.Length - 1] -Path $restorePath -PowerUp $true
### Start instant restore of machines latest restorepoint and power up after mounting is done.
Verifying the backups
The next step was to find a way to verify that the machines were actually functional.
Because the coolest features from SureBackup aren’t available for machines backed up with Veeam Agent, we can’t change the network configuration of the restored VMs other than deciding which VM-Network they will have their NICs in on the Hyper-V Server. This makes it difficult to use network access as an easy option to check that the VM is functional.
Newer Windows servers
If you’re running a Hyper-V server newer than Windows server 2016, and the guest is Windows server 2016/Windows 10 or newer, you can actually do a remote PowerShell login from the Hyper-V server with no network access to the VM
with New-Pssession -vmname #name-of-vm or just Enter-Pssession -vmname #name-of-vm
Doing this you can actually check anything you can access through PowerShell on the VM. Basically almost anything you can think of if you put your mind to it. You need to use the correct credentials for each VM, which is a bit annoying to deal with when you have a multi-tenant environment. Also, if you restore a Domain Controller, you can’t use regular domain credentials to verify the restore, you have to use the Directory Services Restore Mode (DSRM) password.
You can check that the VM actually boots properly with basically any credentials, even completely bogus ones.
Because if you try to remote access a VM and the authentication services are running, you’ll get an error message telling you that your credentials are invalid.
enter-pssession : The credential is invalid.
While if they are not running, you’ll get a different error:
enter-pssession : An error has occurred which Windows PowerShell cannot handle. A remote session might have ended.
Which means a quick way to see that the VM is actually up and running is to fail an authentication check:
try {
New-PSSession -VMName (get-vm).Name -Credential $credential -ErrorAction stop
} catch {
$ErrorMessage = $_.Exception.Message;
if ($ErrorMessage -eq "The credential is invalid.") {
write-host "Testing logon to VM: " $ErrorMessage " Authentication works, the VM is up and running"
} else {
write-host "login testing failed, trying again. error: " $ErrorMessage
}
}
Older (or Linux) servers
Sadly, remote PowerShell without network access doesn’t work if the VM is older than Windows server 2016. Since we still have some servers on 2012r2, we need to have a way to check that they are functional as well. Without network access, we’re not able to check much inside the VMs. But if integration services are running, the Hyper-V host is getting VM heartbeats.
We can check these Heartbeats with PowerShell from the Hyper-V server.
$result = Get-VMIntegrationService -VMName (get-vm).name -Name Heartbeat
if ($result.PrimaryStatusDescription -eq "OK") {
write-host "heartbeat found. Success"
}
This will even work on Linux servers backed up with Veeam agent, as long as they have the settings for Hyper-V integration configured. Here’s how to do that
The downside with this check is that it takes longer for the heartbeat service to start, than the authentication service. So, it can take a long time to verify these machines.
Full script
Check out the full script on our GitHub