none
Caching exception: Socket disposed

    Question

  • Just picked off this error from my logs. Any ideas?

     

    [6] - Error getting object from cache: Key=userid-3ae70a10-44d5-442e-ac1d-107c3ac4f225: Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0017>:SubStatus<ES0006>:There is a temporary failure. Please retry later. (One or more specified cache servers are unavailable, which could be caused by busy network or servers. For on-premises cache clusters, also verify the following conditions. Ensure that security permission has been granted for this client account, and check that the AppFabric Caching Service is allowed through the firewall on all cache hosts. Also the MaxBufferSize on the server must be greater than or equal to the serialized object size sent from the client.) ---> System.ServiceModel.CommunicationException: The socket was aborted because an asynchronous receive from the socket did not complete within the allotted timeout of 00:01:00. The time allotted to this operation may have been a portion of a longer timeout. ---> System.ObjectDisposedException: The socket connection has been disposed.
    Object name: 'System.ServiceModel.Channels.SocketConnection'.
       --- End of inner exception stack trace ---
       at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at System.ServiceModel.Channels.FramingDuplexSessionChannel.EndReceive(IAsyncResult result)
       at Microsoft.ApplicationServer.Caching.WcfClientChannel.CompleteProcessing(IAsyncResult result)
       --- End of inner exception stack trace ---
       at Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody)
       at Microsoft.ApplicationServer.Caching.DataCache.InternalGet(String key, DataCacheItemVersion& version, String region, IMonitoringListener listener)
       at Microsoft.ApplicationServer.Caching.DataCache.<>c__DisplayClass49.<Get>b__48()
       at SepiaLabs.GlassboardPlatform.Caching.CacheEngine.AppFabricCacheEngine.Get(String key) in C:\projects\glassboard\SepiaLabs.GlassboardPlatform.Caching\CacheEngine\AppFabricCacheEngine.cs:line 98
       at SepiaLabs.GlassboardPlatform.Caching.CacheEngine.HttpContextCacheEngine.Get(String key) in C:\projects\glassboard\SepiaLabs.GlassboardPlatform.Caching\CacheEngine\HttpContextCacheEngine.cs:line 28
       at SepiaLabs.GlassboardPlatform.Caching.CacheClient.Get(String key, Boolean ensurePrivateInstance) in C:\projects\glassboard\SepiaLabs.GlassboardPlatform.Caching\CacheClient.cs:line 49
    ; TraceSource 'w3wp.exe' event

    Tuesday, May 17, 2011 9:26 PM

Answers

  • Akshat Sharma, of the Caching team, has a post on errors like this. He suggests they could be caused by a stale connection being used to access the Caching service. The solution is to set DataCacheFactoryConfiguration.TransportProperties.ReceiveTimeout to a value less than 1 minute.
    Wednesday, May 18, 2011 1:21 AM

All replies

  • Akshat Sharma, of the Caching team, has a post on errors like this. He suggests they could be caused by a stale connection being used to access the Caching service. The solution is to set DataCacheFactoryConfiguration.TransportProperties.ReceiveTimeout to a value less than 1 minute.
    Wednesday, May 18, 2011 1:21 AM
  • Hi Neil,

      Akshat already told me to do that on another thread, I've had that set in my web.config since then. 

     

    <dataCacheClients>
    
      <dataCacheClient name="default">
       <hosts>
        <host name="bigsecret.cache.windows.net" cachePort="22233" />
       </hosts>
    
       <securityProperties mode="Message">
        <messageSecurity authorizationInfo="<redacted>">
        </messageSecurity>
       </securityProperties>
       <transportProperties channelInitializationTimeout="10000" receiveTimeout="60000" />
      </dataCacheClient>
    
      <dataCacheClient name="SslEndpoint">
       <hosts>
        <host name="bigsecret.cache.windows.net" cachePort="22243" />
       </hosts>
    
       <securityProperties mode="Message" sslEnabled="true">
        <messageSecurity authorizationInfo="<redacted>">
        </messageSecurity>
       </securityProperties>
       <transportProperties channelInitializationTimeout="10000" receiveTimeout="60000" />
      </dataCacheClient>
     </dataCacheClients>

    Wednesday, May 18, 2011 2:51 PM
  • Brian, do you perform the Get/Put operation precisely after every 1 minute? Can you check the behavior if you set the receiveTimeout to 55 seconds?

     

    Thanks,

    Akshat

    Thursday, May 19, 2011 4:46 AM
  • Akshat,

      There's nothing that would reliably happen every 60 second, at least not that I'm aware of. It should only happen on demand. Right now the app is in beta, so the traffic is fairly low and bursty, with a lot of idle stretches.

      I will try changing that receiveTimeout and let you know what happens.

    Thanks,

    BKR

    Thursday, May 19, 2011 2:40 PM
  • Hi Brian,

    We have not heard from you in a couple of days.

    What is the result by changing DataCacheFactoryConfiguration.TransportProperties.ReceiveTimeout to a value less than 1 minute? Does it resolve the issue?

    If the issue is still there, may I ask you to create a code sample and share it with us? This will be veryful to reproduce and resolve the issue.

    Thanks,


    Wengchao Zeng
    Please mark the replies as answers if they help or unmark if not.
    If you have any feedback about my replies, please contact msdnmg@microsoft.com.
    Microsoft One Code Framework
    Monday, May 23, 2011 3:44 AM
  • We ended up not deploying again until early Saturday morning (UTC time), so the change hasn't been up for that long. I haven't seen any exceptions in that time, though it has also been very low traffic during that time.

    I think we can tentatively say that solved the problem. I want to watch it a little longer before I say for sure.

    Monday, May 23, 2011 2:32 PM
  • I recently saw this error occur again on 5/21/2011 12:45:18 AM UTC. My ReceiveTimeout is set to 1 minute.

    Here is the code I'm using to interact with the Cache: 
    http://stackoverflow.com/questions/5914860/using-the-cache-aside-pattern-with-a-singleton-datacachefactory-and-datacache

     

    Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0017>:SubStatus<ES0006>:There is a temporary failure. Please retry later. (One or more specified cache servers are unavailable, which could be caused by busy network or servers. For on-premises cache clusters, also verify the following conditions. Ensure that security permission has been granted for this client account, and check that the AppFabric Caching Service is allowed through the firewall on all cache hosts. Also the MaxBufferSize on the server must be greater than or equal to the serialized object size sent from the client.) ---> System.TimeoutException: The socket was aborted because an asynchronous receive from the socket did not complete within the allotted timeout of 00:01:00. The time allotted to this operation may have been a portion of a longer timeout. ---> System.Net.Sockets.SocketException: The I\/O operation has been aborted because of either a thread exit or an application request\u000d\u000a   --- End of inner exception stack trace ---\u000d\u000a   at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)\u000d\u000a   at System.ServiceModel.Channels.FramingDuplexSessionChannel.EndReceive(IAsyncResult result)\u000d\u000a   at Microsoft.ApplicationServer.Caching.WcfClientChannel.CompleteProcessing(IAsyncResult result)\u000d\u000a   --- End of inner exception stack trace ---\u000d\u000a   at Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody)\u000d\u000a   at Microsoft.ApplicationServer.Caching.DataCache.InternalGet(String key, DataCacheItemVersion& version, String region, IMonitoringListener listener)\u000d\u000a   at Microsoft.ApplicationServer.Caching.DataCache.<>c__DisplayClass49.<Get>b__48()\u000d\u000a

     


    http://www.IActionable.com - Gamification and Analytics Platform

    Monday, May 23, 2011 6:20 PM
  • Akshat,

      Just grabbed this one. Note the timeout is set to 55 seconds. Guess that didn't fix it entirely. :/

     

    [30] - Caught non-fatal exception talking to AppFabric caching.
    Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0017>:SubStatus<ES0006>:There is a temporary failure. Please retry later. (One or more specified cache servers are unavailable, which could be caused by busy network or servers. For on-premises cache clusters, also verify the following conditions. Ensure that security permission has been granted for this client account, and check that the AppFabric Caching Service is allowed through the firewall on all cache hosts. Also the MaxBufferSize on the server must be greater than or equal to the serialized object size sent from the client.) ---> System.ServiceModel.CommunicationException: The socket was aborted because an asynchronous receive from the socket did not complete within the allotted timeout of 00:00:55. The time allotted to this operation may have been a portion of a longer timeout. ---> System.ObjectDisposedException: The socket connection has been disposed.
    Object name: 'System.ServiceModel.Channels.SocketConnection'.
       --- End of inner exception stack trace ---
       at System.ServiceModel.Channels.SocketConnection.ThrowIfNotOpen()
       at System.ServiceModel.Channels.SocketConnection.BeginRead(Int32 offset, Int32 size, TimeSpan timeout, WaitCallback callback, Object state)
       at System.ServiceModel.Channels.SessionConnectionReader.BeginReceive(TimeSpan timeout, WaitCallback callback, Object state)
       at System.ServiceModel.Channels.SynchronizedMessageSource.ReceiveAsyncResult.PerformOperation(TimeSpan timeout)
       at System.ServiceModel.Channels.SynchronizedMessageSource.SynchronizedAsyncResult`1..ctor(SynchronizedMessageSource syncSource, TimeSpan timeout, AsyncCallback callback, Object state)
       at System.ServiceModel.Channels.FramingDuplexSessionChannel.BeginReceive(TimeSpan timeout, AsyncCallback callback, Object state)
       at Microsoft.ApplicationServer.Caching.WcfClientChannel.CompleteProcessing(IAsyncResult result)
       --- End of inner exception stack trace ---
       at Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody)
       at Microsoft.ApplicationServer.Caching.DataCache.InternalGet(String key, DataCacheItemVersion& version, String region, IMonitoringListener listener)
       at Microsoft.ApplicationServer.Caching.DataCache.<>c__DisplayClass49.<Get>b__48()
    <snip>

    Wednesday, May 25, 2011 3:45 PM
  • Hi Brian,

    How about setting the timeout to a smaller value (say 45 seconds or 40 seconds)?

    Thanks,


    Wengchao Zeng
    Please mark the replies as answers if they help or unmark if not.
    If you have any feedback about my replies, please contact msdnmg@microsoft.com.
    Microsoft One Code Framework
    Thursday, May 26, 2011 5:22 AM
  • Any news on this issue? We´re facing the same problem and even setting the timeout to a smaller value (45 secs) didn´t solve the problem.

    Is there a way to check the connection status on the DataCacheFactory object? We haven´t found anything on the API.

    My exception:

    Microsoft.ApplicationServer.Caching.DataCacheException was unhandled by user code

      HelpLink=http://go.microsoft.com/fwlink/?LinkId=164049

      Message=ErrorCode<ERRCA0017>:SubStatus<ES0006>:There is a temporary failure. Please retry later. (One or more specified cache servers are unavailable, which could be caused by busy network or servers. For on-premises cache clusters, also verify the following conditions. Ensure that security permission has been granted for this client account, and check that the AppFabric Caching Service is allowed through the firewall on all cache hosts. Also the MaxBufferSize on the server must be greater than or equal to the serialized object size sent from the client.)

      Source=Microsoft.ApplicationServer.Caching.Client

      ErrorCode=17

      SubStatus=5

      StackTrace:

           at Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody)

           at Microsoft.ApplicationServer.Caching.DataCache.InternalGet(String key, DataCacheItemVersion& version, String region, IMonitoringListener listener)

           at Microsoft.ApplicationServer.Caching.DataCache.<>c__DisplayClass49.<Get>b__48()

           at Microsoft.ApplicationServer.Caching.MonitoringListenerFactory.EmptyListener.Microsoft.ApplicationServer.Caching.IMonitoringListener.Listen[TResult](Func`1 innerDelegate)

           at Microsoft.ApplicationServer.Caching.DataCache.Get(String key)

           at Agentto.Technology.Microsoft.Azure.ObjectModel.AppFabricHelper.Cache.DataCacheHelper.GetObject(String key) in C:\Source\Agentto\Technology\Microsoft\Azure\ObjectModel\AppFabricHelper\Cache\DataCacheHelper.cs:line 112

           at Agentto.Execution.Azure.Domain.DomainEntities.MainDomain.GetObjectFromCache() in C:\Source\Agentto\Execution\Azure\Domain\Domain\DomainEntities\MainDomain.cs:line 1362

           at Agentto.Execution.Azure.Domain.DomainEntities.MainDomain.LoadFromCache() in C:\Source\Agentto\Execution\Azure\Domain\Domain\DomainEntities\MainDomain.cs:line 1444

           at Agentto.Execution.Azure.Domain.DomainEntities.MainDomain.LoadValuesById() in C:\Source\Agentto\Execution\Azure\Domain\Domain\DomainEntities\MainDomain.cs:line 572

           at Agentto.Execution.Azure.Domain.DomainEntities.MainDomain.LoadByID(Object id) in C:\Source\Agentto\Execution\Azure\Domain\Domain\DomainEntities\MainDomain.cs:line 505

           at Agentto.Semantic.Representation.Elastic.InstanceDomainGraph.InstanceManager.InstanceLoadManager.LoadGraph(String findMethod, Object[] parms, Object mainDomainPK) in C:\Source\Agentto\Semantic\Representation\Elastic\InstanceDomainGraph\InstanceManager\InstanceLoadManager.cs:line 76

           at SyncInvokeLoadGraph(Object , Object[] , Object[] )

           at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)

           at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)

      InnerException: System.ServiceModel.CommunicationException

           Message=The socket was aborted because an asynchronous receive from the socket did not complete within the allotted timeout of 00:00:45. The time allotted to this operation may have been a portion of a longer timeout.

           Source=System.ServiceModel

           StackTrace:

                at System.ServiceModel.Channels.SocketConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout)

                at System.ServiceModel.Channels.SocketConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout, BufferManager bufferManager)

                at System.ServiceModel.Channels.BufferedConnection.WriteNow(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout, BufferManager bufferManager)

                at System.ServiceModel.Channels.BufferedConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout, BufferManager bufferManager)

                at System.ServiceModel.Channels.FramingDuplexSessionChannel.OnSend(Message message, TimeSpan timeout)

                at System.ServiceModel.Channels.OutputChannel.Send(Message message, TimeSpan timeout)

                at Microsoft.ApplicationServer.Caching.CacheResolverChannel.Send(Message message, TimeSpan timeout)

                at Microsoft.ApplicationServer.Caching.WcfClientChannel.SendMessage(EndpointID endpoint, Message message, TimeSpan timeout, WaitCallback callback, Object state, Boolean async)

           InnerException: System.ObjectDisposedException

                Message=Cannot access a disposed object.

    Object name: 'System.Net.Sockets.Socket'.

                Source=System

                ObjectName=System.Net.Sockets.Socket

                StackTrace:

                     at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, SocketError& errorCode)

                     at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)

                     at System.ServiceModel.Channels.SocketConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout)

                InnerException:

     

    Monday, June 27, 2011 6:19 PM
  • Brian,

    Where is the web.config file that you're modifying?  I am experiencing the same problem and looked in the main web.config for the web application but it doesn't contain a <dataCacheClients> node.

    Thanks,
    Patty

    Monday, December 17, 2012 7:31 PM
  • Patty,

      You have to add it into the web.config when you're adding the cache client into your code. It's in the docs for setting up caching. 

      Alternatively, your configuration might be done in code somewhere, in which case it wouldn't be in the config files at all. I actually moved to doing that after this post. 

    BKR

    Wednesday, December 19, 2012 4:49 PM