none
"Failed starting VM" status of cloud service

    Question

  • Hello,

    Since Friday I cannot make usual deployments which I could make for years.

    After pretty long time period, the Service's status on "Cloud Services (classic)" list is "Running" with red cross.

    When opening a service, VM status is "Failed starting VM".

    If click that VM then details show "Power: Stopped".

    I have three cloud services, and all are like this. Yesterday out of my 10 tries only one I was able to get Running service, but only one of three. This morning after next deployment I can see Failed again.

    The worst thing is that I don't have any more details about error occurred. Can someone help me with this?

    Last Thursday all was good.

    Sergei.

    Tuesday, February 26, 2019 6:42 AM

Answers

  • The problem was finally resolved. Somehow portal allowed to specify incompatible Gateway settings for VNet peering. As the result, Cloud Service deployments were good, but Cloud Service VM were never started and didn't provide any useful error description.

    I place MS response here, just in case if someone face with similar issue.

    The Customer has a Gateway in ASM (Classic) VNet as well as configuration over a VNet peering to use remote gateway transit. As a result, VNet configuration set operation fails due to multiple gateways.Exception at NetworkServiceManagerCore: Virtual network configuration has more than one gateway. (ErrorCode=TooManyGatewaysInVNetException Parameter1=d869b9a4-5bf0-48ea-9cb6-a905446de29e Parameter=null)

    In this situation customer would not be able to make any change to his VNet. So, we emcourage the affected customers to take appropriate action. We already fixed this issue in NRP which is getting rolled out, which will avoid this issue for future gateways.

    Mitigation steps from customer side:

    Due to this misconfiguration, your VNet connectivity may have been impacted. Please take one of the two steps below to mitigate the issue and restore the VNet connectivity:
    1. You can disable the gateway transit options on the peering by setting “Allow Gateway Transit” and “Use Remote Gateway” to false on the corresponding peering’s; or
    2. Delete the VPN gateway on the VNet: <VNET NAME>
    Please refer to this document for the detailed instructions: https://docs.microsoft.com/azure/vpn-gateway/vpn-gateway-peering-gateway-transit

    • Marked as answer by Sergei Kamnev Tuesday, March 12, 2019 9:41 AM
    Tuesday, March 12, 2019 9:41 AM

All replies

  • Hi,

    Please go to the VM's(stopped vm) Activity log and select a larger timespan.

    By default you will have "Last 6 hours" timespan. Check if you have any errors or warnings.

    Also check what triggered the shutdown of that VM.

    Tuesday, February 26, 2019 12:14 PM
    Moderator
  • Hello,

    There is no warnings or errors. Just information messages like "Create deployment slot" and "Start deployment slot".

    Cloud Service was dropped and re-created already several times with no success.

    Sergei.

    Tuesday, February 26, 2019 2:08 PM
  • Hi Sergei. Sounds like this might be something we want to have you work with Support on 1:1. Do you have the ability to open a technical Support ticket? If not, you can email me at AzCommunity@microsoft.com and provide me with your SubscriptionID and link to this thread. We can enable your subscription for that support request. 
    Tuesday, February 26, 2019 6:18 PM
    Moderator
  • Hello Micah,

    The ticket is there: 

    [REG:119022624001247] ] Cloud Service IaaS issue 'Failed starting VM' status; Power is 'Stopped' and no way to bring the service back

    Sergei

    Wednesday, February 27, 2019 7:26 AM
  • Thanks Sergei. I am following that ticket now. If you run into issues just let me know and I am happy to jump in. 
    Wednesday, February 27, 2019 7:58 PM
    Moderator
  • The problem was finally resolved. Somehow portal allowed to specify incompatible Gateway settings for VNet peering. As the result, Cloud Service deployments were good, but Cloud Service VM were never started and didn't provide any useful error description.

    I place MS response here, just in case if someone face with similar issue.

    The Customer has a Gateway in ASM (Classic) VNet as well as configuration over a VNet peering to use remote gateway transit. As a result, VNet configuration set operation fails due to multiple gateways.Exception at NetworkServiceManagerCore: Virtual network configuration has more than one gateway. (ErrorCode=TooManyGatewaysInVNetException Parameter1=d869b9a4-5bf0-48ea-9cb6-a905446de29e Parameter=null)

    In this situation customer would not be able to make any change to his VNet. So, we emcourage the affected customers to take appropriate action. We already fixed this issue in NRP which is getting rolled out, which will avoid this issue for future gateways.

    Mitigation steps from customer side:

    Due to this misconfiguration, your VNet connectivity may have been impacted. Please take one of the two steps below to mitigate the issue and restore the VNet connectivity:
    1. You can disable the gateway transit options on the peering by setting “Allow Gateway Transit” and “Use Remote Gateway” to false on the corresponding peering’s; or
    2. Delete the VPN gateway on the VNet: <VNET NAME>
    Please refer to this document for the detailed instructions: https://docs.microsoft.com/azure/vpn-gateway/vpn-gateway-peering-gateway-transit

    • Marked as answer by Sergei Kamnev Tuesday, March 12, 2019 9:41 AM
    Tuesday, March 12, 2019 9:41 AM