none
Azure VMs no longer see themselves via public virtual IP RRS feed

  • Question

  • At around 4:00 PM (PST) on 4/20/2015, our Azure VM instances could no longer reach themselves via their virtual public IP.  This is a major problem for our web server as we also run a mail server on the same machine.  The web applications can no longer send mail using the mail server's domain name.  We have made no changes to our networking configuration, endpoints, or firewall.  I'm assuming this is a Microsoft issue.

    The machines are reachable from outside via the virtual public IP, as well as from the other VM instances on the same virtual network.

    I've never run into anything like this.  Any ideas?

    Thank you.

    Tuesday, April 21, 2015 10:13 AM

Answers

  • Hi folks I want to provide you an update on this,

    The engineering team has rolled out a fix that is in the process of being deployed to the affected clusters. We have prioritized the fix for all customers that contacted us; for those who haven’t, you should begin to see improvement with this issue. If you are still experiencing issues, please post to this thread and we’ll connect with you to see how we can help. Please note that the rollout may take a few days to reach all of the impacted clusters, so again, let us know in this thread if you don’t see improvement.

    We’ll keep our eyes on this thread and will check back in as needed,

    -Jason

    • Marked as answer by Baz TTE Thursday, April 23, 2015 4:21 AM
    Thursday, April 23, 2015 2:30 AM

All replies

  • Hi,

    We apologize for the inconvenience. There was a Service Interruption caused to Azure Virtual Machine Services on 20th April 2015.

    However the Issue is now mitigated and the Services have now been restored. 

    Suggest you to try restarting the VM and check if its working.

    Looking forward for your reply.

    Regards,

    Sowmya

    Tuesday, April 21, 2015 10:46 AM
    Moderator
  • Sowmya,

    Thank you for the reply.  I've restarted all of my Virtual Machine instances.  The issue persists. They still cannot to connect to themselves via the public virtual IP address.

    Thank you.

    Tuesday, April 21, 2015 11:17 AM
  • I am having this same issue as well. Started at about the same time.
    Tuesday, April 21, 2015 12:44 PM
  • Hi,

    I understand that the issue still exists with respect to Azure Virtual Machines even after restarting it. Suggest you to try the below steps 

    Hope this helps !

    Regards,

    Sowmya

    Tuesday, April 21, 2015 12:54 PM
    Moderator
  • Hi,

    I have the same issue. I can confirm that resizing the Virtual Machine from the portal does not resolve it. Does anybody have any insight as to what the root cause is? I'm not tempted to waste much more time trying any more one-size-fits-all solutions until I'm certain they'll work.

    Regards,

    Alexander

    Tuesday, April 21, 2015 3:07 PM
  • Alexander,

    Thank you for confirming that resizing the virtual machine has no effect.  I won't waste my time attempting it.  The other solutions listed seem rather drastic as well for what appears to be an internal Azure issue.

    Tuesday, April 21, 2015 3:13 PM
  • We also seem to be having the same issue as of the same time and almost 24 hours later it is still not working. Restarting and re-sizing has no effect.

    Bob

    Tuesday, April 21, 2015 7:45 PM
  • Sowmya,

    I've done my due diligence and attempted the above suggestions. None of them have corrected the issue.  What's worse, the last one forced a new public virtual IP.  The servers still cannot see themselves.  This appears to be an Azure problem.  Can we escalate this issue?  The prospect of paying for support for something that Azure broke in the first place is not very palatable.

    Thank you.

    Wednesday, April 22, 2015 8:24 AM
  • Thank you Baz for confirming that the above solutions do not work.

    Please note for anyone following this thread, there's also another one going on entitled 'Cloud Apps "cannot navigate to itself"'

    Wednesday, April 22, 2015 9:03 AM
  • Hey there, thanks for posting. Here's the latest from Azure Engineering

    A recent Azure infrastructure update caused an issue that results in the symptoms discussed in this thread:

    VMs in a VNET may not be able to connect to their own VIP, especially in cases where there are very few VMs behind the VIP

    There is no impact to any other connectivity

    This issue affects a limited number of resources in all regions, but does not affect all resources in all regions.  As soon as the issue was identified, the update was quarantined to cease any further impact.

    While the customer impacting scenario is very limited, we understand that this is a pain point for customers who are affected.  We are actively working on a fix and look forward to resolving this issue soon.  We will post an update in this thread as the situation progresses.


    Wednesday, April 22, 2015 5:58 PM
  • For a short-term fix we have altered the server hosts file to route applicable domains back to the internal (10.0...) IP address.
    Wednesday, April 22, 2015 7:15 PM
  • Hi,

    i am having same issue IIS application running in VMs i am able to connect Web Application using VIP outside and locally VMs, but in same machine not able to connect using Public or Private IP, but able to connect using localhost, 

    While done the ip entry in host file it is working fine, Please provide solution. 

    Regards,

    Vivek 

    Thursday, April 23, 2015 1:20 AM
  • Hi folks I want to provide you an update on this,

    The engineering team has rolled out a fix that is in the process of being deployed to the affected clusters. We have prioritized the fix for all customers that contacted us; for those who haven’t, you should begin to see improvement with this issue. If you are still experiencing issues, please post to this thread and we’ll connect with you to see how we can help. Please note that the rollout may take a few days to reach all of the impacted clusters, so again, let us know in this thread if you don’t see improvement.

    We’ll keep our eyes on this thread and will check back in as needed,

    -Jason

    • Marked as answer by Baz TTE Thursday, April 23, 2015 4:21 AM
    Thursday, April 23, 2015 2:30 AM
  • Jason,

    Thank you for your attention in this matter.  I can confirm that this issue is now resolved for me.

    Thanks again,

    -Baz

    Thursday, April 23, 2015 4:23 AM
  • This is a problem (since yesterday) for cloud services as well. Many HTTP requests back to cloud service's own IP address are not routing back through.

    The melody of logic will always play out the truth. ~ Narumi Ayumu, Spiral

    Thursday, April 23, 2015 6:40 AM
  • If anybody is interested in a workaround, we made our web apps in the cloud service route requests locally using the server IP address by discovery via IIS site bindings.

    http://blogs.msdn.com/b/carlosag/archive/2011/01/21/get-iis-bindings-at-runtime-without-being-an-administrator.aspx


    The melody of logic will always play out the truth. ~ Narumi Ayumu, Spiral

    Thursday, April 23, 2015 6:47 AM
  • Hi Jason,

    This issue has now been resolved for me. Thank you very much for your attention to the matter.

    Alexander

    Thursday, April 23, 2015 8:42 AM
  • Hi Jason,

    We have been having a very similar issue with Instance-level Public IP (PIP) addresses on Azure. Our services are no longer accessible when accessing them via the Instance-level Public IP.

    Can you confirm whether PIPs are also affected by this issue and if so whether the fix will apply to them as well?

    Thanks

    Thursday, April 23, 2015 9:22 AM
  • We are still having issues, unable to connect to a VM from our apps.
    Thursday, April 23, 2015 4:02 PM
  • jcam213 & Scott if you're still experiencing the issue, can you please send me an email with Deployment IDs? This goes for anyone else that is experiencing this issue as well. I'll check with engineers to see if we believe impact is correlated. Please use [VIP] in the subject line so I can prioritize. I will follow up in this thread by EOD Pacific time. 


    Thursday, April 23, 2015 6:19 PM

  • I've restarted all of my Virtual Machine instances.  The issue persists. They still cannot to connect to themselves via the public virtual IP address.


    • Edited by NataliUA Friday, April 24, 2015 1:54 AM
    Thursday, April 23, 2015 11:36 PM
  • @NataliUA, I've got your request and have forwarded it on to support - would recommend that you (and others on this thread) only share your SubID and DeploymentID via email to me as specified above. You can remove this information by using the "Edit" button on your post.

    For those of you tracking, I've processed and (confirmed via email) with anyone that sent me a message with their Deployment ID information. Please keep these coming and we'll prioritize your Deployments. 

    Friday, April 24, 2015 12:23 AM
  • Good afternoon, when the network will work on my server?
    I beg to say at least the approximate timing of the completion of work on how to reset the network.
    Friday, April 24, 2015 1:54 AM
  • Hiii,

          I have forward you deployment Id. Please let us know when this issue can be resolved.

    Friday, April 24, 2015 7:27 AM
  • THEY STILL CANNOT TO CONNECT TO THEMSELVES VIA THE PUBLIC VIRTUAL IP ADDRESS.
    Friday, April 24, 2015 11:17 AM
  • A recent Azure infrastructure update caused an issue that results in some customers not being able to connect to their own Virtual IP (VIP), especially in cases where there are very few Virtual Machines (VMs) behind the VIP configuration. There is no impact to any other connectivity at this time. This issue affects a limited number of resources in all regions, but does not affect all resources in all regions. As soon as the issue was identified, the update was quarantined to cease any further impact. We understand that this is a pain point for customers who are affected and apologize for any inconvenience that has caused. Engineers are actively working on a resolution. This message will be active until this issue is resolved. 
    Friday, April 24, 2015 11:40 AM
  • Hi Jason

    Same problem here -- can access fine from public internet, cant from IIS Brows Website -- gets timeouts

    Funny when I configured the VM to my VNet/subnet it all worked fine.  Added the PIP address and VM seemed to be moved out of my assigned subnet.  Also when I remote desktop into my VM the desktop background messages show my local machine address as my subnet address 192.168.100.10  but ifconfig (and portal) show it on 192.168.0.4

    nslookup www.mydomain.ca does show my PIP address as it should (on VM and my home office PC)

    I have a problem with my production app and don't want to turn off CstomErors for all the world to see....

    Friday, April 24, 2015 5:11 PM
  • All fixed now -- thanks

    Saturday, April 25, 2015 12:28 AM
  • THEY STILL CANNOT TO CONNECT TO THEMSELVES VIA THE PUBLIC VIRTUAL IP ADDRESS. !!!!!!!!!
    Saturday, April 25, 2015 2:13 PM
  • THEY STILL CANNOT TO CONNECT TO THEMSELVES VIA THE PUBLIC VIRTUAL IP ADDRESS. !!!!!!!!! 
    Saturday, April 25, 2015 2:13 PM
  • Folks, one last update to this thread, engineering has fully finished off deployment of the patch worldwide. We prioritized patching to anyone who provided us a valid, active Deployment ID. We really appreciate the interaction on this thread and want to thank everyone who shared their experiences and provided workarounds to the community. Thanks again for working through this with us!
    • Proposed as answer by NataliUA Saturday, April 25, 2015 9:01 PM
    Saturday, April 25, 2015 7:57 PM
  • My Deployment ID THEY STILL CANNOT TO CONNECT TO THEMSELVES VIA THE PUBLIC VIRTUAL IP ADDRESS
    Saturday, April 25, 2015 9:01 PM
  • Hi Jason,

    Just emailed you with the ID - look forward to getting this fixed.  My biz is at a stand still in the meantime.

    Best,

    Heather

    Saturday, April 25, 2015 11:55 PM