Migrating to ESXi 5
Back in March I decided that my brief stint running Xen Hypervisor wasn't working out and I needed to get back into the rich environment that VMWare offers through ESXi 5.
My reluctance of using ESXi is how limited the host layer is. For instance, my server runs software RAID - something that is not possible to implement in ESXi. I toyed with building two datastores and running software RAID in the guest machines - this works quite well especially if you thin provision and use LVM in the guests. You can then chop and change where you host your data without having to worry rebuilding your guest.
Still, with a reliance on numerous customised Linux services, over a terrabyte of data and little spare time I am hesitant of a complete rebuild currently.
In comes RDMs - or, Raw Device Maps. These are shortcut/pointer files on the ESXi host that allow you to attach a local physical hard drive directly to a VM. What is even better, you can boot from them to. This has allowed me to boot ESXi off a USB thumb drive and then launch a Virtual Machine that directly runs what is already installed on the physical disks of the server. Unfortunately I have yet to find a way of getting ESXi to allow VMX files on a USB key so another hard drive is required to host the minimal configuration files. If I pull the USB key from my server and boot it, the system will boot as normal - giving me a fall back to direct iron if required.
Getting this all working though, is a little easier said than done. Luckily there is always somebody else that has done it and written it up...
Step one: ESXi onto a USB key
I did this using VMWare Workstation and a copy of ESXi (already customised with my odd network drivers) and just installed directly onto the USB key.
Step two: RDMs
Dave Mishchenko has a detailed guide on Creating RDMs on SATA drives. It's fairly straight forward to follow and you can either use Physical or Virtual mappings (I'm not sure of the pros and cons). He didn't seem to be able to boot a Physical mappings but I had no such problem.
Step three: Setup a VM
Just setup a VM as normal, and attach your newly created RDM vmdk files as disks. The trick here is to make sure you plug them in (assign SCSI ports) in the correct order.
I spent a lot of time trying to figure out why my system wouldn't boot at all, and it was because although ESXi claimed it was checking both hard drives for a boot disk it didn't seem to actually boot off the second hard drive. Switching the boot order or the disk order around solved this issue.
Step four: Reverting to physical and fixing all the drive mappings
This bit took me a while to figure out. Whilst I got GRUB running fine, when I attempted to boot the OS I couldn't access hard drives - instead I got presented with "Waiting on /dev/xxx device to appear...". On inspecting /dev, it was clear that udev (Linux device manager) was not finding any drives.
I fiddled at length with /etc/fstab switching how drives were mounted to no avail. I finally stumbled upon a blog post Resolving Linux boot issues after P2V with VMware Converter. Whilst I wasn't doing a conversion the same issues arose.
The issue is that the disk controller changes type from whatever is natively on your host to a virtual one. If the kernel is not set to load the related modules it won't be able to find anything attached. Rebooting directly onto the physical host I added the modules piix and mptspi to my INITRD_MODULES in the Linux Kernel sysconfig.
Restarting under ESXi got me a bit further but I still could not boot. The last step was to fix the drive mapping for my /boot partition. In /etc/fstab this was mapped by device. The change from physical to virtual meant that the device was no longer listed at an ATA connection but instead a SCSI connection. I remapped this using a label to avoid problems when reverting directly to physical and rebooted the VM.
Everything came online - woohoo!
It really is as simple (hah) as that. I still have problems with getting my virtualised server to talk to my UPS - something that I'm not entirely sure if I will be able to fix or not. I will post here if I do.