none
Nested Virtualization Inside Azure

    Question

  • Hi,

    This may be a dumb question but is there a way to run hypervisor or ESXI as an Azure instance.

    To my understanding the CPU requires the vmx flag in order to support nesting.

    I guess the real question is can you get access to that part of the CPU to enable nesting. I'm assuming you can't due to security.

    Thanks in advance.

    Wednesday, June 22, 2016 9:48 PM

Answers

  • Hello,

    Thank you for using the Microsoft Azure forums!

    Normally you could install a VMware ESXi host using tools like Intel-VT but for this you would need to have access to the underlying CPU hardware. Since you do not have control over that in an Azure environment you will not be able to install a VMware ESXi host in a Azure VM.

    As for installing a Microsoft hypervisor i.e. Hyper-V on a Azure VM, lets say running a Windows Server will also give you errors since the Hyper-V is designed to be installed on the underlying hardware. When this is tried on a Azure VM which is already running on a Hyper-V, the setup will identify that the underlying hardware is virtualized (i.e. a Hyper-V) and will not install all the modules although you may manage to install it.

    There is a unsupported workaround for this though but even I this case you will not be able to utilize the functionality of Hyper-V since the VM will probably not boot up. You may refer to How to Install Hyper-V on a Virtual Machine in Hyper-V for this.

    So in summary you do not have control of the Azure (Hosted environment) to make the changes to servers and so you will not be able to configure Nested Hypervisors or to allow Intel-VT passthrough,, and start a VM. So this would apply to any hypervisor as well. 

    Hope this helps.

    Regards,

    Loydon

    ________________________________________________________________________________________________________________
    When you see answers and helpful posts, please click Vote As Helpful, Propose As Answer, and/or Mark As Answer so that other customers can benefit from it.

    Thursday, June 23, 2016 6:56 AM

All replies

  • Hello,

    Thank you for using the Microsoft Azure forums!

    Normally you could install a VMware ESXi host using tools like Intel-VT but for this you would need to have access to the underlying CPU hardware. Since you do not have control over that in an Azure environment you will not be able to install a VMware ESXi host in a Azure VM.

    As for installing a Microsoft hypervisor i.e. Hyper-V on a Azure VM, lets say running a Windows Server will also give you errors since the Hyper-V is designed to be installed on the underlying hardware. When this is tried on a Azure VM which is already running on a Hyper-V, the setup will identify that the underlying hardware is virtualized (i.e. a Hyper-V) and will not install all the modules although you may manage to install it.

    There is a unsupported workaround for this though but even I this case you will not be able to utilize the functionality of Hyper-V since the VM will probably not boot up. You may refer to How to Install Hyper-V on a Virtual Machine in Hyper-V for this.

    So in summary you do not have control of the Azure (Hosted environment) to make the changes to servers and so you will not be able to configure Nested Hypervisors or to allow Intel-VT passthrough,, and start a VM. So this would apply to any hypervisor as well. 

    Hope this helps.

    Regards,

    Loydon

    ________________________________________________________________________________________________________________
    When you see answers and helpful posts, please click Vote As Helpful, Propose As Answer, and/or Mark As Answer so that other customers can benefit from it.

    Thursday, June 23, 2016 6:56 AM
  • Loydon,

    Thank you for the clarification. 

    I did come across a company called Ravello that has nesting capabilities in AWS/Google Cloud.

    How is that possible? Could they possibly have full access to the hardware?

    Thank you.


    Tuesday, June 28, 2016 10:00 PM
  • Hello,

    Thank you for reverting!

    As I mentioned earlier, installing an Intel-VT extension on a Azure hypervisor is not supported. That is why if you were to install another Hypervisor on top of an existing VM then the nested hypervisor would not be able to emulate the underlying virtualized hardware that the primary hypervisor is using.

    As per the technology used by Ravello, it uses a different approach towards nested virtualization. Ravello is not a Cloud service provider but it provides its product called HVX which enables virtualization extensions such as Intel-VT. As a result a nested virtualization environment can be constructed on top of an existing Cloud based hypervisor.

    Please have a go through the documentation for HVX here. It will give you more details. As of now this is supported on AWS and Google Cloud but not on Azure. If required you may get in touch with them to explore the possibilities (if any) with Azure.

    In case you want to give a suggestion to our Azure team please feel free to visit https://feedback.azure.com/forums/216843-virtual-machines and post your feedback.

    Hope this helps.

    Regards,

    Loydon

    Disclaimer: This response contains a reference to a third party World Wide Web site. Microsoft is providing this information as a convenience to you. Microsoft does not control these sites and has not tested any software or information found on these sites; therefore, Microsoft cannot make any representations regarding the quality, safety, or suitability of any software or information found there. There are inherent dangers in the use of any software found on the Internet, and Microsoft cautions you to make sure that you completely understand the risk before retrieving any software from the Internet.

    ________________________________________________________________________________________________________________
    When you see answers and helpful posts, please click Vote As Helpful, Propose As Answer, and/or Mark As Answer so that other customers can benefit from it.

    Wednesday, June 29, 2016 3:22 AM
  • Now with nested virtualization comming to Windows server 2016, will this feature be available to Azure VMs in the near future?
    You mention that a switch need to be turned on and in Azure you dont have the possibility for this, maybe will there be such a possibility in the near future?
    Wednesday, August 10, 2016 7:29 AM
  • Hello,

    Thank you for posting here.

    With the launch of Windows Server 2016 Microsoft has integrated Docker containers feature in this version of Windows Server. So as per the open source usage of Docker it should allow you to run any Linux distros or Windows images on top of any underlying infrastructure. It is clear that Microsoft does not support Nested Virtualization in Azure using hyper-V containers. However, you can check http://searchwindowsserver.techtarget.com/tutorial/How-to-deploy-Windows-containers-in-Microsoft-Azure which describes a way to utilize the Docker containers feature introduced in Windows Azure. The Windows Server 2016 is still in TP4 so we do not have a confirmation about how this will behave. So if you are looking to utilize this and give it a try it would be interesting to know your feedback.

    Regards,

    Loydon


    Disclaimer: This response contains a reference to a third party World Wide Web site. Microsoft is providing this information as a convenience to you. Microsoft does not control these sites and has not tested any software or information found on these sites; therefore, Microsoft cannot make any representations regarding the quality, safety, or suitability of any software or information found there. There are inherent dangers in the use of any software found on the Internet, and Microsoft cautions you to make sure that you completely understand the risk before retrieving any software from the Internet.
    • Edited by Loydon Mendonca Wednesday, August 10, 2016 10:10 AM Added disclaimer
    Wednesday, August 10, 2016 10:09 AM
  • Hi Loydon,

    Is there anyway to run a Windows 2003 32 bit server version on an Azure VM?  The problem that we face with my clients is that their not going to upgrade their legacy ERP anytime soon.  We, therefore, have to continue running Tuxedo 6.5, which does not boot up when running on a Windows 2008 64 bit machine.  Any suggestions?  Thanks.

    Ahmed Al-Hujazi

    ahmed.al-hujazi@cgifederal.com

    Monday, October 10, 2016 2:16 PM
  • Hello Alhujazi,

    Thank you for posting on the Azure forums.

    Since this would be a new topic I have gone ahead and opened a new thread for this. Kindly refer to https://social.msdn.microsoft.com/Forums/en-US/7ce4b933-45c1-4864-bedf-69ee3e47bc1a/is-there-anyway-to-run-a-windows-2003-32-bit-server-version-on-an-azure-vm?forum=WAVirtualMachinesforWindows where we can discuss this.

    I have responded in the link mentioned above. Kindly check and post your response or queries in the above thread from hereon.

    Best regards,

    Loydon


    • Edited by Loydon Mendonca Monday, October 10, 2016 6:30 PM added reference.
    Monday, October 10, 2016 3:07 PM
  • This could be quite frustrating.  I'm setting up a Visual Studio development environment on an Azure Virtual Machine (Windows 10).  This development environment won't be fully featured because the Windows Mobile emulator used within Visual Studio is based on Hyper-V.  It won't be possible for this emulator to function without nested virtualisation.

    I'm aware that the following command should activate nesting on Windows 10, if run against a stopped VM:

         Set-VMProcessor -VMName <VMName> -ExposeVirtualizationExtensions $true

    Is there a way of running this or achieving the same effect? It would really disappointing to be prevented from running an effective cloud-based development environment using a 100% Microsoft branded toolset :(

    Thanks,

    Simon

    Thursday, October 13, 2016 1:53 AM
  • I echo Simons sentiments. I use the cloud for all my infrastructure development work. Not being able to use nested virtualisation within Azure really hampers the Azure IaaS offering for release/dev work, especially when other virtualisation vendors like VMware have supported nested virtualisation for years. I'm hoping this is addressed really soon.
    Saturday, October 29, 2016 11:03 PM
  • Hello Simon / Shawn,

    This has been an ask from our customers for a while now and we are working on getting this addressed. However, we don not have a timeline so I request you to get this through to us on the feedback forums as well. Please visit https://feedback.azure.com/forums/34192--general-feedback for writing your feedback about nested virtualization or any other improvements you would want with Azure. Your opinions and views are much appreciated and valued.

    Regards,

    Loydon

    Sunday, October 30, 2016 8:58 AM
  • It has been months since this forum has been updated and Server 2016 is in its full production version. Is nested virtualization in Azure when running Server 2016 now possible?

    Wednesday, June 14, 2017 4:13 PM
  • This is still not possible in the Azure environment. 

    -------------------------------------------------------------------------------------------------------------------------
    Do click on "Mark as Answer" on the post that helps you, this can be beneficial to other community members.

    Thursday, June 15, 2017 2:31 AM
  • Here you go, much awaited announcement is here.

    Nested Virtualization in Azure


    Please use Mark as Answer if my post solved your problem and use Vote As Helpful if a post was useful.

    Friday, July 14, 2017 4:36 AM
  • Yes, you can with the new Dv3 and Ev3 VM sizes

    The Dv3 and Ev3 sizes are also some of the first VM’s to be running on Windows Server 2016 hosts.  Windows 2016 hosts enable Nested Virtualization and Hyper-V Containers for these new VM sizes.  Nested virtualization allows you to run a Hyper-V server on an Azure virtual machine. With nested virtualization you can run a Hyper-V Container in a virtualized container host, set-up a Hyper-V lab in a virtualized environment, or to test multi-machine scenarios

    For details: https://azure.microsoft.com/en-us/blog/introducing-the-new-dv3-and-ev3-vm-sizes/


    http://nicogis.blogspot.com

    Friday, August 04, 2017 10:46 AM
  • now you can deploy Dv3 and Ev3 vm with 2016 OS and install hyper-v feature and on that hyper-v you can run esxi.

    rg

    Tuesday, August 08, 2017 9:22 AM