vSphere 5 doesn’t allow nesting of 64-bit hypervisors by default. This is a change from version 4. With the following tweaks you can have a virtual Hyper-V or Xen Server up and running in an ESXi 5 environment.
Task 1: Configure ESXi Hosts for Nesting
Tools Needed: Putty, vSphere Client
For every ESXi host that may present the VM, edit the /etc/vmware/config file.
1. Confirm your host has the SSL service running and allowed through the host’s firewall using the VI Client.
- If needed, turn on the SSH service on in the host through the Configuration tab àSoftwareàSecurity Settings.
2. Connect to the host using Putty. Choose the default SSH connection, login to access the command prompt.
3. At the prompt enter the following command:
echo ‘vhv.allow = “TRUE” ‘ >> /etc/vmware/config
This writes changes directly to the host’s config file.
4. To confirm the changes, type:
Cat /etc/vmware/config
You should see the following:
Task 2: Build 64-bit Windows Server 2008 R2 Virtual Machine
Keep in mind that you’ll need to be on Virtual Machine Version 8. Customize the VM with the following:
Add whatever resources you think is appropriate, then complete the creation.
Task 3: Edit the Virtual Machine Configuration File
This change tricks Hyper-V into thinking it’s running on physical hardware, or more specifically, assure that it isn’t running in a virtualized environment even though it is.
- Use the vSphere client to download the VM’s .vmx file from the datastore it was created in. Save it to the desktop and copy it in case a bad change corrupts the VM.
- Add the following line to the end of the .vmx: hypervisor.cpuid.v0 = “FALSE”
- An example of the very end of the changed .vmx file:
- Save the changed .vmx and upload it back to the datastore of the virtual machine. The vSphere Client will overwrite the existing file.
Task 4: Change the Guest Operating System Setting
Edit the settings of the new virtual machine. In the options tab, change the Guest Operating System to VMware ESX1 5.x. This allows the ESXi host to recognize the guest as a potential hypervisor.
You’re now ready to install the Hyper-V role on the virtual Windows Server 2008 R2 machine. You will be able to add nested Hyper-V virtual machines, assign storage and networking. Performance will be slower as you’re now traversing two virtual infrastructures.
I’ve found this to be very effective in V2V conversions with vhd files. Instead of trying to convert a vhd to a vmdk, I’ve been building Hyper-V VM’s and converting them using VMware Converter 5.
Resources: Thanks to these v-Bloggers.
http://www.virtuallyghetto.com/2011/07/how-to-enable-support-for-nested-64bit.html
http://tsmith.co/2011/creating-a-nested-esxi-5-environment/
wfcastle33.wordpress.com 
Thanks for the link back! Great write up. This was my highest hit post for quite a while, glad more people are sharing how to run nested.