none
IaaS VHD IOP limit of 500 is ripping disks out from under the VM

    Question

  • With Azure standard storage, we're hard limited by 500 IOPs on a single VHD disk. While this is just fine for normal use during peak hours, its wreaking havoc when back-end off-peak process/batch jobs are hitting the machines. What is happening is simple - the VMs are hitting the IOP limit of our standard blob storage on the VHD (500 IOPs).  In practicality 500 IOPs is enough, but what is actually happening is the VMs are freaking out and dropping the throttled-by-azure disks right off the OS.

    Is there any way to increase the buffer/queue on the VM so the single disks are not dropped?  Its terrible that the disks get dropped off the OS during crunch-times, and the only recovery option we have is to reboot each system.  Why in the world are the disks dropped instead of just throttled to 500? 

    I'm trying to avoid re-creating VMs with multiple/pooled disks. 

    Thank you in advance.


    • Edited by ZeroClient Monday, September 19, 2016 1:45 PM typos
    Monday, September 19, 2016 1:43 PM

All replies

  • Hi,

    Thank you for posting here!

    Kindly refer the below article for Configuring Azure Virtual Machines for Optimal Storage Performance:

    https://blogs.msdn.microsoft.com/mast/2014/10/14/configuring-azure-virtual-machines-for-optimal-storage-performance/

    You can also refer the below article:

    https://support.microsoft.com/en-au/kb/3150851#bookmark-memorybottleneck

    If it doesn’t help, please let us know, we are happy to assist you more.

    Regards,

    Vikranth S.

    Tuesday, September 20, 2016 1:07 PM
    Moderator
  • While planning and monitoring is something not to be taken lightly, it is not a solution.  Case - what if an un-planned event happens such as a check-disk at the same time as a batch-job, as well as a backup happen to run con-currently?  What about if some process arbitrarily runs-away for a few minutes while consuming IOPS?  its the im-measurable I'm concerned about - and its taking us down.

    The real problem here is the disks are dropping off the OS when they hit a limited IOPs by Azure.  The act of dropping disks from the OS should not be happening as its clipping the system at the knees moreso than throttling it down.

    I need a solution where the VMs will not actually drop the disks off the OS when Azure throttles IOPs.  Again - I am fine with limiting the VM to only 500 IOPs, but it is unacceptable to have disks dropped from the OS when that limit is hit - its causing massive data-loss and destruction to the VMs.

    Tuesday, September 20, 2016 1:24 PM
  • >the disks are dropping off the OS when they hit a limited IOPs by Azure

    That doesn't sound right.  Could you elaborate?  What guest OS are you running? 

    David


    David http://blogs.msdn.com/b/dbrowne/

    Tuesday, September 20, 2016 3:01 PM
  • OS is Windows 2012R2. D13_V2 IaaS box in NorthCentral US Azure. 

    The disk which keeps reaching significant IOPs is holding an Oracle 12c database managed by 'us'.  The OS drops the disk entirely which requires a full reboot before the disk gets re-attached.  None of the other disks get dropped from the OS.

    From a Microsoft perspective, we filed a SEV A request here: [REG:116090914649473] LAPRE | SEV A | Azure Virtual Machine – Windows

    Tuesday, September 20, 2016 3:17 PM