Current Virtual machine applications, like VirtualBox and VMware, can virtualize the IOMMU (VT-x or AMD-V). Which means we can create nested virtualization. In layman's terms, we can create a VM within a VM within a VM. The question is, with an i7 and 32GB of RAM, how deep can we go?
Caution: Do not try this at home. It's not dangerous, or anything. Just a waste of time.
To satisfy your scientific curiosity, we have documented all our attempts, including the failures. So, if you were dreaming of nested virtualization, then you have weird dreams, maybe you should talk to a professional. But you will get the answers to what's possible.
- 1 The test rig
- 2 Nested virtualization with VirtualBox
- 3 Nested virtualization with VMware Workstation
- 4 The results
The test rig
To conduct this experiment in a proper way, we should have used a Multi-CPU server.
120 physical cores and 12.0 TB of RAM would have helped a lot. Like, immensely.
Unfortunately, we forgot the $100,000 that it costs on our other pair of pants. Don't you just hate when this happens?
So, we had to do with a simple, consumer-grade PC. It has an i7 4790K, running at the stock clock speed of 4GHz. It also has 32GB of RAM, but our guess is we will run out of CPU power way before we run out of RAM.
Still, eight threads is nothing to look down at. It should give us at least a two-level nested virtualization, and maybe more if we're lucky.
Our main OS, and our nested virtualization experiment starting point is Ubuntu Linux 14.10 x64.
Nested virtualization with VirtualBox
VirtualBox is a great piece of software. It's free, most of it is open source, it's easy to use, and it does the job well. And we already saw how to install VirtualBox in Linux Mint and Ubuntu.
The first VirtualBox VM
Since we started with Ubuntu Linux, it made sense to install Ubuntu on all the nested VMs. This way, the hardware requirements would have been evenly split up.
The first VM should have lots of RAM. So, we turned the slider all the way to the end of the green zone, which is the "safe zone" according to VirtualBox.
We also gave four CPU cores, which again is the healthy amount according to VirtualBox.
Selecting and Ubuntu 14.10 ISO as a virtual disk, we fired up our first VM. So far, so good.
No matter what settings we tried, we couldn't even get the live Ubuntu environment to load. The best we managed was to get a cryptic message.
So, it was time for plan B.
Instead of Ubuntu, we used a Lubuntu 14.10 ISO.
Since Lubuntu is a lightweight Linux Distribution, this would only help our nested virtualization experiment.
We are starting to suspect that the 14.10 version of Ubuntu just doesn't like VirtualBox. We guess that if we have kept at it, we would have managed to install at least the first VM.
However, the nested virtualization experiment didn't call for a particular distribution.
We could probably have tried Windows, since creating a Windows virtual machine in VirtualBox is a matter of minutes.
Instead, we gave Linux Mint 17.1 a try, which is based on Ubuntu 14.04 LTS.
Lo and behold, we 're live. 14.04 apparently hit the sweet spot with VirtualBox.
It wouldn't recognize video hardware acceleration, but that's a small price to pay.
The first nested VirtualBox VM
It wasn't long before we run into trouble. For the first nested virtualization VM, we were only allowed to create a 32bit system.
That meant that we could only allocate 4GB of RAM to this VM.
Not to be deterred, we created Linux Mint 2.
We could also allocate a single core, even though the VM had four to spare.
Things weren't going well for our nested virtualization experiment. And, sure enough...
Even though we double checked that we had Hardware Virtualization activated...
...it wouldn't work on the first nested VM.
The first VirtualBox VM (redux)
So, back to square zero. We thought that maybe it was Linux Mint's fault, so we scrapped the previous VM, and created a Windows 7 one. Having the 30-day trial ISOs from Digital River sure did help.
We didn't have any problems with the first VM.
However, we still couldn't install a 64bit VM inside the first...
...and VT-x virtualization still didn't work.
All in all, nested virtualization with VirtualBox was nothing sort of a failure. And a miserable one, at that.
So, did this mean that the nested virtualization experiment was over? Hardly.
Nested virtualization with VMware Workstation
VMware Workstation is a professional solution to create virtual machines, and it costs 250$.
Here at PCsteps we don't think that "paid" necessarily equals "better" when talking about software. But, after we strung out with VirtualBox, VMware was worth a try.
The first VMware Workstation VM
For the first VM, we allocated 26GB of RAM.
We also gave the full eight cores and made sure we had virtualized IOMMU.
Ubuntu 14.10 installation went on without a hitch. Also, the "Easy install" feature of VMware came handy; it went through the installation completely unattended.
Not before long, we had our first virtual machine.
The first nested VMware Workstation VM
For the first nested VM, we allocated around 19GB of RAM.
This time, we gave it six cores and virtualized VT-x, of course.
That was as far as we went with VirtualBox. Fearing that it wouldn't work, and this whole article would be a bust, we started up the nested VM.
We got a warning that a VM within a VM will result in degraded performance. A much better warning than VirtualBox's error message.
Well, what do you know. It works.
The CPU usage is peaking during the installation.
We even had VMware Workstation black-out at some point.
But then it got unstuck, and our first nested VM was live.
Officially, the nested virtualization experiment was a successful one. But now it's time to ask the serious question. How deep can we go?
The second nested VMware Workstation VM
For the second nested VM, we allocated 15GB of RAM and four CPU cores out of the six of the first nested VM.
Not only did it work, but it went more smoothly than the previous VM. We didn't have any black-outs.
Not before long, the second nested VM installation was live.
Of course, the deeper we went, the slower everything became. But it worked, no doubt about that.
The thing is, even the simplest task, such us copying a file, would make the CPU light up like a Christmas tree.
It's abundantly clear by now that RAM will have nothing to do with the limit of the VMs. It's all a matter of CPU cores and the quality of the VT-x virtualization.
The third nested VMware Workstation VM
Things didn't look up for the third nested VM. VMware's logo, which usually flies by too fast to enter the BIOS with F2, now took whole minutes to load.
The CPU activity was also erratic. A single core would be at 100%, and then change places with another core.
After all, the VM managed to load beyond the POST screen, but it got stuck at the ISOLINUX screen and never recovered.
We tried to recreate the scenario with Windows 7 x64, in case it was Ubuntu's fault.
In fact, the results were worse. We could only install Windows on the first nested virtual machine, and the second would get stuck before even starting the installation.
In the end, it wasn't the RAM or the CPU cores. It was probably the VT-x that couldn't handle the progressive re-virtualization.
Could an even lighter OS, such as a stripped down Arch Linux, give us a third nested VM? Possibly.
For the time being, though, a VM within a VM within a VM is impressive enough. And utterly useless, as opposed to having three separate VMs working at the same time.
Well, now you know. And knowing is half the battle.