Multi Headed Hypervisor Server

Submitted by Matt on Tue, 04/14/2015 - 20:36

So, being in the server tech business I have access to special high performance hardware. This gives me a special opportunity to fool around with enterprise level software and solutions.

Some cool things I have been playing with:

  • Virtualization (SR-IOV, VT-D, VT-X)
  • Hardware RAID

These capabilities have been around for a while but mainly with servers. And they now arent even necessarily limited to servers since some high end desktops are now hitting 6-8 cores.

My Initial Problem:

Because I like to tinker around with a lot of different things, I pretty much needed a computer to fill each individual specialized task. This began to get pretty rediculous after a while. So many power cables, network cables, monitors, spare parts lying around everywhere. It became very unsightly. And not even to mention that each computer could consume somewhere between 20-100W idle. There had to be a better way!

Sure enough there is. It is called VMware ESXi, and it is awesome. I really wish I would have found it earlier. Vmware ESXi (Do they call it Vsphere now) is a bare metal Virtual Machine Hypervisor. It allows you to take a system''s hardware and divide its hardware resources up among one or more "virtual" machines. There are other products that do similar things, such as VMWare workstation and Virtualbox, but these are not bare metal hypervisors. They run on top of a given OS and tend to not be as flexible or efficient (but still very useful in certain circumstances).

So my server is HP ProLiant DL380 Gen9 with two processors with tens of gigabytes of RAM. I have multiple virtual machines set up off all different kinds of OSs depending on its purpose. Most are headless (I VNC/RDP into them) bit 2 are "headed".

Server Hypervisor

Plugged into the server PCIe risers, I have 2 graphics cards and 2 USB controller cards. I can pass a graphics card and PCIe card each through to the 2 virtual machine. So I can use my server as two high performance gaming rigs simultaneously, each independently controllable. Since the USB controlers are passed through, each OS has its own USB keyboard/mouse plugged in among other things. 

Virtual Machine Loadout

The server has 4 port Gigabit LOM (Lan on Motherboard), so I decided to replace my router (that in hindsight looks very crappy) with a linux distro, not named here for security purposes. This helps me in a number of ways. My "router" VM is completely flexible. It can host a webserver, dyndns and VPN client. Way more useful than any router software i have ever used. Also, it is potentially more secure. Most routers out there do not get FW updates very often. Maybe every 6 months tops, for about 2 years if that. Since most of them are actually embedded linux installs, they are riddled with known security exploits galore.

For storage on the server I have four 5 TB drives in RAID 1+0 for a total of 10 GB in spinny disk storage. Once you format the array with VMFS the VMs are files. This makes backing up super easy; no more imaging disks, no more downtime ever just copy the file. You can even take snapshots of a disk for later restore. I use the spinny array for any OSs that have tons of data that does not need to be that fast (though it is actually very, very fast). I have a 1 TB SSD that I subdivide among all of the VMs to be super fast.

I can do lots of stuff at once on all the OSs with no lag. Everything is hardware accelerated. The only gotcha so far is that blu rays dont work well due to HDCP. Stupid stupid copy protection keeps me from playing my own legal Blu Rays. Imagine that. I will have to rant about this some other time. But I did find a workaround, cant list it here. Just google for "Decrypt Blu Ray" or something like this.

All of this is remotely manageable with the vsphere client. Systems reboot super fast, they can auto start. It is a dream.

If you want to do a similar build please feel free to ask questions. Be careful, not all graphics cards support passthrough. Most high end ATI do, but only the highest end Nvidia ones do at the time of writing.