none
Windows Azure Caching while debugging - Request Timeout

    Question

  • I have a problem while using Windows Azure Caching with emulator (i.e., on Local, not Cloud).

    My system configurations are:

    • Windows 8
    • Visual Studio 2012 Update 1
    • Azure SDK and Emulator 1.8
    • Windows Azure Caching 1.8.1
    • Web API
    • .NET 4.5

    The problem is: Our server utilizes (in-role) Windows Azure Caching for session managements and etc. Once the server gets requests from clients, it starts communicating with Windows Azure Caching to put/get some objects (e.g., session objects). Then a "time-out" exception (Microsoft.ApplicationServer.Caching.DataCacheException) is occurred and the server becomes unresponsive, showing the following messages in the Output log of Visual Studio.

    ...
    iisexpress.exe Error: 0 : ERROR: <DistributedCache.RoutingClient> 25cb6980-94c5-4e72-  adee-cd58defff7fe:SendMsgAndWait: Request TimedOut, msgId = 77
    
    A first chance exception of type 'Microsoft.ApplicationServer.Caching.DataCacheException' occurred in Microsoft.ApplicationServer.Caching.Client.dll
    
    'iisexpress.exe' (Managed (v4.0.30319)): Loaded 'C:\windows\assembly\GAC_MSIL\Microsoft.VisualStudio.Debugger.Runtime\11.0.0.0__b03f5f7f11d50a3a\Microsoft.VisualStudio.Debugger.Runtime.dll'
    iisexpress.exe Error: 0 : The thread '<No Name>' (0x16d0) has exited with code 0 (0x0).
    
    The thread '<No Name>' (0x1a00) has exited with code 0 (0x0).
    The thread '<No Name>' (0x2298) has exited with code 0 (0x0).
    
    ERROR: <DistributedCache.RoutingClient> e91ff46b-870c-4d83-8eef-6bdb718baab9:SendMsgAndWait: Request TimedOut, msgId = 78
    
    iisexpress.exe Warning: 0 : Microsoft.WindowsAzure.ServiceRuntime Verbose: 500 : Role instance status check starting
    
    Microsoft.WindowsAzure.ServiceRuntime Verbose: 502 : Role instance status check succeeded: Ready
    
    WARNING: <SimpleSendReceiveModule> DeadServerCallback Called, Server URI: [net.tcp://127.255.0.0:20004], Underlying exception - 
    
    A first chance exception of type 'Microsoft.ApplicationServer.Caching.DataCacheException' occurred in Microsoft.ApplicationServer.Caching.Client.dll
    
    iisexpress.exe Warning: 0 : WARNING: <SimpleSendReceiveModule> DeadServerCallback Called, Server URI: [net.tcp://127.255.0.0:20004], Underlying exception - 
    
    iisexpress.exe Warning: 0 : WARNING: <SimpleSendReceiveModule> DeadServerCallback: Matches My Server, Cleaning Pending Requests
    
    iisexpress.exe Warning: 0 : WARNING: <SimpleSendReceiveModule> 48c1dd7f-f54d-40a8-9990-25af48b91eb1:Request - 79, result - Status=SendFailed[System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host
    
       at System.Net.Sockets.Socket.Send(IList`1 buffers)
    
       at Microsoft.ApplicationServer.Caching.TcpSocketChannel.Send(IList`1 buffers, TimeSpan timeout)]
    
    iisexpress.exe Warning: 0 : WARNING: <SimpleSendReceiveModule> DeadServerCallback: Matches My Server, Cleaning Pending Requests
    
    A first chance exception of type 'Microsoft.ApplicationServer.Caching.DataCacheException' occurred in Microsoft.ApplicationServer.Caching.Client.dll
    
    iisexpress.exe Warning: 0 : WARNING: <DistributedCache.SocketClientChannel.1> Request 81 to host net.tcp://127.255.0.0:20004/ failed Status=ChannelOpening
    
    iisexpress.exe Warning: 0 : WARNING: <DistributedCache.SocketClientChannel.1> Request 80 to host net.tcp://127.255.0.0:20004/ failed Status=ChannelOpening
    
    iisexpress.exe Warning: 0 : WARNING: <SimpleSendReceiveModule> 8ec8ff74-1f04-49da-8ff5-529d0697542f:Request - 81, result - Status=ChannelOpening
    
    iisexpress.exe Warning: 0 : A first chance exception of type 'Microsoft.ApplicationServer.Caching.DataCacheException' occurred in Microsoft.ApplicationServer.Caching.Client.dll
    

    When this starts occurring, the server is just down (i.e., running, but not responsive).

    Any idea why this happens? Is it related to Windows 8?

    Thanks,

    Wednesday, February 20, 2013 7:24 PM

All replies

  • Hi,

    Have a look at this article about setting up Windows Azure Caching for Session State Management:

    http://dotnetthread.com/articles/27-Setting-up-Windows-Azure-Caching-for-Session-State-Management.aspx

    Thanks,


    QinDian Tang
    MSDN Community Support | Feedback to us
    Develop and promote your apps in Windows Store
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.

    Thursday, February 21, 2013 7:52 AM
  • Qin Dian Tang, what was the solution from that article?  That seems to be a fairly basic walkthrough which doesn't seem to give any indication as to what is causing this error.  I am getting it now on a new Win8 machine.  I haven't touched this codebase for months but I know it is/was working in deployment so the Caching Session State Management is properly configured.

    I must say it is annoying that moderators self-accept answers as they give false-hope that we'll find the answer we're looking for...

    Anyone have any idea how to diagnose this?

    I notice farther down in the debug output I start getting this:

    .................

    w3wp.exe Information: 0 : INFORMATION: <Complaint> Add hard complaint :0
    The thread '<No Name>' (0x26ec) has exited with code 0 (0x0).
    Microsoft.WindowsAzure.ServiceRuntime Verbose: 500 : Role instance status check starting
    Microsoft.WindowsAzure.ServiceRuntime Verbose: 502 : Role instance status check succeeded: Ready
    w3wp.exe Warning: 0 : WARNING: <SimpleSendReceiveModule> DeadServerCallback Called, Server URI: [net.tcp://127.255.0.0:20004], Underlying exception -
    w3wp.exe Warning: 0 : WARNING: <SimpleSendReceiveModule> DeadServerCallback: Matches My Server, Cleaning Pending Requests
    w3wp.exe Warning: 0 : WARNING: <CASClient> Failed to retrieve lookup table from net.tcp://127.255.0.0:20004: Microsoft.Fabric.Common.OperationContextAbortedException: Failed getting Lookup Table from net.tcp://127.255.0.0:20004 ---> Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0016>:SubStatus<ES0001>:The connection was terminated, possibly due to server or network problems or serialized Object size is greater than MaxBufferSize on server. Result of the request is unknown.
       --- End of inner exception stack trace ---
       at Microsoft.ApplicationServer.Caching.SocketClientProtocol.GetLookupTable(EndpointID endpoint, CacheLookupTableRequest request, TimeSpan timeout)
       at Microsoft.ApplicationServer.Caching.ThickClient.BeginRequestLookupTable(String nodeAddress, LookupTableRequest request, TimeSpan timeout, AsyncCallback callback, Object state)
       at Microsoft.Fabric.Data.ServiceResolver.RetrieveLookupTable(String addr)
    Microsoft.WindowsAzure.ServiceRuntime Verbose: 500 : Role instance status check starting
    Microsoft.WindowsAzure.ServiceRuntime Verbose: 502 : Role instance status check succeeded: Ready
    w3wp.exe Warning: 0 : WARNING: <SimpleSendReceiveModule> DeadServerCallback Called, Server URI: [net.tcp://127.255.0.0:20004], Underlying exception -
    w3wp.exe Warning: 0 : WARNING: <SimpleSendReceiveModule> DeadServerCallback: Matches My Server, Cleaning Pending Requests
    w3wp.exe Warning: 0 : WARNING: <CASClient> Failed to retrieve lookup table from net.tcp://127.255.0.0:20004/: Microsoft.Fabric.Common.OperationContextAbortedException: Failed getting Lookup Table from net.tcp://127.255.0.0:20004/ ---> Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0016>:SubStatus<ES0001>:The connection was terminated, possibly due to server or network problems or serialized Object size is greater than MaxBufferSize on server. Result of the request is unknown.
       --- End of inner exception stack trace ---
       at Microsoft.ApplicationServer.Caching.SocketClientProtocol.GetLookupTable(EndpointID endpoint, CacheLookupTableRequest request, TimeSpan timeout)
       at Microsoft.ApplicationServer.Caching.ThickClient.BeginRequestLookupTable(String nodeAddress, LookupTableRequest request, TimeSpan timeout, AsyncCallback callback, Object state)
       at Microsoft.Fabric.Data.ServiceResolver.RetrieveLookupTable(String addr)
    Microsoft.WindowsAzure.ServiceRuntime Verbose: 500 : Role instance status check starting
    Microsoft.WindowsAzure.ServiceRuntime Verbose: 502 : Role instance status check succeeded: Ready
    w3wp.exe Warning: 0 : WARNING: <SimpleSendReceiveModule> DeadServerCallback Called, Server URI: [net.tcp://127.255.0.0:20004], Underlying exception -
    w3wp.exe Warning: 0 : WARNING: <SimpleSendReceiveModule> DeadServerCallback: Matches My Server, Cleaning Pending Requests
    w3wp.exe Warning: 0 : WARNING: <CASClient> Failed to retrieve lookup table from net.tcp://127.255.0.0:20004/: Microsoft.Fabric.Common.OperationContextAbortedException: Failed getting Lookup Table from net.tcp://127.255.0.0:20004/ ---> Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0016>:SubStatus<ES0001>:The connection was terminated, possibly due to server or network problems or serialized Object size is greater than MaxBufferSize on server. Result of the request is unknown.
       --- End of inner exception stack trace ---
       at Microsoft.ApplicationServer.Caching.SocketClientProtocol.GetLookupTable(EndpointID endpoint, CacheLookupTableRequest request, TimeSpan timeout)
       at Microsoft.ApplicationServer.Caching.ThickClient.BeginRequestLookupTable(String nodeAddress, LookupTableRequest request, TimeSpan timeout, AsyncCallback callback, Object state)
       at Microsoft.Fabric.Data.ServiceResolver.RetrieveLookupTable(String addr)
    w3wp.exe Information: 0 : INFORMATION: <CASClient> Updated partition table to (1-65) generation: 634999197929558527:0 with transfer (1-65) generation: 634999197929558527:0
    The thread '<No Name>' (0x295c) has exited with code 0 (0x0).

    Tuesday, March 26, 2013 6:57 PM
  • Hi,

    Sorry for that. As my signature says you can unmark it if the reply doesn't answer your issue. It is my work process to mark the possible answer if OP doesn't come back in 3 days and the thread must be longer than 7 days.

    Thanks,


    QinDian Tang
    MSDN Community Support | Feedback to us
    Develop and promote your apps in Windows Store
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.

    Wednesday, March 27, 2013 2:06 AM
  • Did you figure out a solution to this? I'm facing the same problem.

    Tarek Ayna www.BluGraphingCalculator.com Blu Graphing Calculator is Calculator ReImagined for Windows 8

    Wednesday, May 08, 2013 5:19 PM
  • I am not sure if this could be the cause, but I think this might have some kind of relation to the problem: http://stackoverflow.com/questions/12464185/windows-8-net-tcp-acceptasync-callback-not-firing-blocked-by-console-readlin

    Azure DataCache seems to use AsyncCallback via net.tcp to communicate back to the client (when you use Caching Roles) and Windows 8 seems to block it somehow. As far as the post referenced is concerned this is a bug within the operating system. I hope the Azure team can work around it somehow? That is, IF this is actually the cause. I'm not sure.
    Friday, June 07, 2013 11:45 AM
  • Has there been any movement on this?

    The bug is a real showstopper for testing

    Thursday, June 13, 2013 2:13 PM
  • My system configuration is:
    Windows 8 Pro 12GB Ram
    Visual Studio 2012 Update 2
    Azure SDK and Emulator 2.0
    Windows Azure Caching 2.0
    Web API, .NET 4.5
    When my application tries to access the default Azure Cache in the emulator, I get:

    ERRCA0018 ES0001 The request timed out The client was trying to communicate with the server net tcp 127.255.0.0 20004

    I have a laptop with the exact same configuration, except for 8 GB ram instead.  However, Azure cache & emulator work.

    Wednesday, June 19, 2013 8:39 PM
  • I am facing the exact same issue. Please update on status of investigation of the product team.

    Tarek Ayna www.BluGraphingCalculator.com Blu Graphing Calculator is Calculator ReImagined for Windows 8

    Thursday, June 20, 2013 3:08 PM
  • I'm getting the same thing... However its only seems to be failing for my worker role consistently.  My webrole seems to be able to use the cache about 50% of the time without a problem.

    this is VERY annoying and makes me worry that I can't trust distributed cache when I deploy.

    My System is a i7 with 16GB of ram running Windows 8 and VS2012 Update 2 and Azure SDK April 2013 release.

    Josh

    Saturday, June 22, 2013 12:23 PM
  • Guess I'm in the same boat as the rest of people having a problem.

    Application had been working fine until recently.

    Now seeing frequent but not totally consistent time out errors accessing the cache in local testing.

    System: Windows 8, i7, 12GB ram

    VS2012 WAZ Tools Version 2.0

    Thursday, July 18, 2013 9:06 PM
  • I have the same problem as well. This problem has been driving me crazy.
    Friday, August 02, 2013 3:54 PM
  • Same here.
    Monday, August 05, 2013 3:18 AM
  • I will recommend to delete DevelopmentStorageDb database and on first storage use a new database will be generated. It is possible that the storage emulator is to busy because of this big database for some table storage operation.



    • Edited by boban.s Tuesday, August 20, 2013 3:20 PM
    Tuesday, August 20, 2013 3:18 PM
  • We also ran into the same problem however fixed it. I wrote a blog post about it a few days ago which you can read here: http://gauravmantri.com/2013/08/07/making-windows-azure-caching-work-in-compute-emulator/. The idea is to upgrade VS to the latest version.

    See if this helps.

    Tuesday, August 20, 2013 5:26 PM
  • I also went down the road of re-installing the OS, Windows 8 Pro all updates installed. Also, installed all the updates for Visual Studio 2012, including Update 3.

    On a fresh environment, which also means the DevelopmentStorageDb was empty I still observed this problem.

    The last time I fussed with the problem was before Windows.Azure.Caching 2.1 was released. I'm going to update that component today and will report back the results.

    Thanks for the additional posts on the issue.

    Tuesday, August 20, 2013 5:59 PM
  • Upgraded Windows.Azure.Caching to 2.1 and still observe the same result.

    Based on the all the digging I've done I agree with JovtonZA and believe the problem to be with TCP in Windows 8.

    Tuesday, August 20, 2013 8:55 PM
  • Hi AlanBlythe.

    Thanks for the confirmation. I have made a guest log entry on their blog that links back here

    (http://blogs.msdn.com/b/windowsazurestorage/archive/2013/07/12/introducing-storage-client-library-2-1-rc-for-net-and-windows-phone-8.aspx?CommentPosted=true#commentmessage)


    • Edited by JovtonZA Wednesday, October 02, 2013 9:09 PM
    Wednesday, August 21, 2013 1:09 AM
  • I'm using Azure SDK 2.0 and previously 1.8, and all machines in the office are Windows 8. Have you checked the time format in Region settings, and make sure it is HH:mm:ss.


    • Edited by boban.s Wednesday, August 21, 2013 11:52 AM
    Wednesday, August 21, 2013 11:52 AM
  • boban,

    Does the time format somehow affect Windows Azure Caching? My long time is currently set to H:mm:ss tt.

    Wednesday, August 21, 2013 3:05 PM
  • When i first started with azure cache, i didn't use the Transient Fault handing library and all was fine, until i implemented that for the caching too. But that didn't worked and i had to disable that functinality until i found a solution. While searching on the web i found that caching monitorng tool uses Logman for loging and that application fails if the format is not HH:mm and HH:mm:ss for short and long time format. I don't know which one exactly, probably the long one. After i changed that and restart the machine, the cache started to work with Transient fault handling too.

    I'm experiencing these cache issues from time to time (few times a day), and somethime i resolve the issue by stoping and starting the project, sometimg by shuting down the storage simulator, but once in a day or two after none of previous tries don't work i will restart the machine and the cache will continue working.

    It's good to run the project without debugger attached if that is not need for that moment, because i have noticed in this situation that when cache fails, mostly after some time it will recover from failed state and continue working. In order to see that you will need to use DebugView.

    Wednesday, August 21, 2013 10:47 PM
  • I went back to Windows 7. The problem simply does not appear in Windows 7 (or earlier relative).

    No config file changes, no project or solution file changes, no source code file changes. No changes to the source repository at all! All the apps and tools such as Visual Studio, Service Packs, Plugins, Azure Tools etc. the same as before. The ONLY difference is the operating system. Win8 and Win7 bitness the same (64).

    Wednesday, October 02, 2013 9:07 PM
  • JovtonZA,

    I'm doing my development on a Windows 7 VM as it's the only thing I've found that fixes the problem as well.

    Thanks

    Thursday, October 03, 2013 2:30 PM
  • I ran into this problem again :(. The problem was so severe this time that I had to repave my machine. It worked for 2 days and then bam, got the cache error today. At least on my machine, I noticed that I get the error and caching generate a lot of warning data when I have Hyper-V feature enabled (I installed WP8 SDK yesterday which enabled Hyper-V). The moment I remove Hyper-V, I can run caching without any issues.

    My machine specifications:

    Windows 8.1 Pro, VS 2013 (Update 3), 16 GB RAM.

    Hope this helps.

    Thanks

    Gaurav

    Thursday, October 03, 2013 2:51 PM
  • Exactly the same problem here. With Hyper-V running on Windows 8.1, caching stops working

    rich


    Richard Fuoco

    Monday, November 04, 2013 11:49 PM
  • I can confirm that Hyper-V is the culprit (or at least related to Hyper-V via networking components of the windows operating system core). After removing Hyper-V, all is well. :)

    Thanks to this guy's (soleiljy) efforts, we now have some direction: http://bit.ly/1b7QQcA

    Windows 7 doesn't have this problem because you cannot run Hyper-V on Windows 7 as far as I know.

    However, it is still an issue, as I feel that I should be able to run Hyper-V and Windows Azure development tools on the same machine if I choose to.

    I hope the Win 8 (and Server 2012) networking team and the Azure team can work together on solving this problem.

    Wednesday, November 20, 2013 2:55 PM
  • Since I created a boot option to boot without Hyper-V, the emulator seems to be working much better. Thanks for the pointer.

    V, but rather set up multi boot with one option to boot with the hypervisor disabled.

    Check out this link to see how to do this.

    http://blog.nicholasrogoff.com/2013/12/27/create-a-no-hyper-v-boot-option/


    CTO of Hard Medium Soft Ltd. and Independent Developer

    Friday, December 27, 2013 2:43 PM