none
Caching: socket closed errors & retries

    Question

  • I'm just starting to use AppFabric Caching, still at a low volume. I'm already seeing "socket closed" errors, which is a little annoying for something that's supposed to support high volumes. I notice that the socket timeout is 10 seconds. Should I set that higher for the cache servers somehow? 

    Presumably when I get those I need to retry the operation. For SQL connections I would reissue the Open() call, but there's nothing analogous for caching. So what should I do? Just try to Get() or Set() call again? Tear down the DataCache or DataCacheFactory and rebuild it? 

    Thanks!,

    BKR

     

     

    Here's a copy of the error I'm seeing:

    Error getting object from cache: Key=userid-f16c43bb-fa50-4fe2-bddd-27e20d369a34: Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode:SubStatus:There is a temporary failure. Please retry later. (One or more specified cache servers are unavailable, which could be caused by busy network or servers. For on-premises cache clusters, also verify the following conditions. Ensure that security permission has been granted for this client account, and check that the AppFabric Caching Service is allowed through the firewall on all cache hosts. Also the MaxBufferSize on the server must be greater than or equal to the serialized object size sent from the client.) ---> System.ServiceModel.CommunicationException: The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:10:00'. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host --- End of inner exception stack trace --- at System.Runtime.AsyncResult.EndTAsyncResult at System.ServiceModel.Channels.FramingDuplexSessionChannel.EndReceive(IAsyncResult result) at Microsoft.ApplicationServer.Caching.WcfClientChannel.CompleteProcessing(IAsyncResult result) --- End of inner exception stack trace --- at Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody) at Microsoft.ApplicationServer.Caching.DataCache.InternalGet(String key, DataCacheItemVersion& version, String region, IMonitoringListener listener) at Microsoft.ApplicationServer.Caching.DataCache.<>cDisplayClass49.b48() at (... stacktrace continues in my custom code...)  TraceSource 'w3wp.exe' event

     

    Saturday, May 7, 2011 2:53 PM

Answers

  • Hi Brian,

     

    Could you please try with value of dataCacheConfiguration.TransportProperties.ReceiveTimeout set to new TimeSpan(0,1,0); i.e. 1 minute. This has to be done for all the instances of DataCacheFactoryConfiguration that you create. Let us know if the problem persists even after the change.

     

    Thanks,

    Akshat

    • Marked as answer by Brian Reischl Wednesday, May 11, 2011 2:46 PM
    Sunday, May 8, 2011 5:30 AM

All replies

  • Hi Brian,

     

    Could you please try with value of dataCacheConfiguration.TransportProperties.ReceiveTimeout set to new TimeSpan(0,1,0); i.e. 1 minute. This has to be done for all the instances of DataCacheFactoryConfiguration that you create. Let us know if the problem persists even after the change.

     

    Thanks,

    Akshat

    • Marked as answer by Brian Reischl Wednesday, May 11, 2011 2:46 PM
    Sunday, May 8, 2011 5:30 AM
  • Hi Akshat,

      Thanks for the suggestion. I'm going to try this via the web.config, using the <transportProperties> element.

      However, will this make cache calls potentially take a long time waiting for the timeout? Would it be faster to use a short timeout, catch the exception and retry?

     

    Thanks,

    BKR

    Monday, May 9, 2011 5:02 PM
  • Hi Brian,

     

    No this won't have any impact on the performance of cache calls. This timeout simply refers to how long cache waits for any activity (cache requests) before recycling the underlying connection. Therefore, comes into the picture only when the cache is not being used.. This is not related to the latency of the requests.

     

    Thanks,

    Akshat

    Monday, May 9, 2011 5:21 PM
  • Akshat,

      That did fix the problem. Thanks!

     

      Follow up question: do I need to worry about the connection dropping after it is established? ie, should I watch for other exceptions while using the cache methods, and add retry or reconnect logic?

     

    Thanks,

    BKR

    Wednesday, May 11, 2011 2:48 PM
  • Hi Brian,

     

    Apart from the receiveTimeout change, you ideally shouldn't be needed to have any retry logic. In case you encounter any other issues, feel free to post back. We'll actively try to resolve the them.

     

    Thanks,

    Akshat

    Wednesday, May 11, 2011 3:09 PM