none
ErrorCode<ERRCA0017>:SubStatus<ES0006>

    Question

  • I am intermittently facing the ErrorCode<ERRCA0017>:SubStatus<ES0006> with Windows Azure and the App Fabric cache in my ASP.NET application.  This brings my whole web application to a standstill until I reset it.

    I am not accessing the Cache directly in my code, I am only storing very small strings in Session state and only have a very small number of users.  I can’t imagine that I could be over any of the usage quotas (at http://msdn.microsoft.com/en-us/library/gg602420.aspx#C_BKMK_FAQ8)

    I would like to find out which quota if any I am exceeding and why.  How can I find out if and why I am being throttled, or if there is any other issue that might be causing this.

    Is there any way to find the Cache Size, Transactions Per Hour, Bandwidth MB Per Hour and Concurrent Connections?

    Any other suggestions for diagnosing resolving this would be welcomed.

    Stack trace:

     Application_Error: ErrorCode<ERRCA0017>:SubStatus<ES0006>:There is a temporary failure. Please retry later. (One or more specified cache servers are unavailable, which could be caused by busy network or servers. For on-premises cache clusters, also verify the following conditions. Ensure that security permission has been granted for this client account, and check that the AppFabric Caching Service is allowed through the firewall on all cache hosts. Also the MaxBufferSize on the server must be greater than or equal to the serialized object size sent from the client.)

    Stack Trace:    at Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody)
       at Microsoft.ApplicationServer.Caching.DataCache.ExecuteAPI(RequestBody reqMsg, IMonitoringListener listener)
       at Microsoft.ApplicationServer.Caching.DataCache.InternalPut(String key, Object value, DataCacheItemVersion oldVersion, TimeSpan timeout, DataCacheTag[] tags, String region, IMonitoringListener listener)
       at Microsoft.ApplicationServer.Caching.DataCache.<>c__DisplayClass25.<Put>b__24()
       at Microsoft.ApplicationServer.Caching.MonitoringListenerFactory.EmptyListener.Microsoft.ApplicationServer.Caching.IMonitoringListener.Listen[TResult](Func`1 innerDelegate)
       at Microsoft.ApplicationServer.Caching.DataCache.Put(String key, Object value, TimeSpan timeout)
       at Microsoft.Web.DistributedCache.DataCacheWrapper.Put(String key, Object value, TimeSpan timeout)
       at Microsoft.Web.DistributedCache.DataCacheForwarderBase.<>c__DisplayClass10.<Put>b__f()
       at Microsoft.Web.DistributedCache.DataCacheForwarderBase.<>c__DisplayClass2e`1.<PerformCacheOperation>b__2d()
       at Microsoft.Web.DistributedCache.DataCacheRetryWrapper.PerformCacheOperation(Action action)
       at Microsoft.Web.DistributedCache.DataCacheForwarderBase.PerformCacheOperation[TResult](Func`1 func)
       at Microsoft.Web.DistributedCache.DataCacheForwarderBase.Put(String key, Object value, TimeSpan timeout)
       at Microsoft.Web.DistributedCache.BlobBasedSessionStoreProvider.SetAndReleaseItemExclusive(HttpContextBase context, String id, SessionStateStoreData item, Object lockId, Boolean newItem)
       at Microsoft.Web.DistributedCache.DistributedCacheSessionStateStoreProvider.SetAndReleaseItemExclusive(HttpContext context, String id, SessionStateStoreData item, Object lockId, Boolean newItem)
       at System.Web.SessionState.SessionStateModule.OnReleaseState(Object source, EventArgs eventArgs)
       at System.Web.SessionState.SessionStateModule.OnEndRequest(Object source, EventArgs eventArgs)
       at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
       at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) on page

    Tuesday, October 25, 2011 10:00 PM

All replies

  • Having the same issue about 6 hours earlier - 1 user, session data only, barely any usage.

     

    I figured maybe the Cache service had some down time?

    Wednesday, October 26, 2011 3:18 AM
  • Hi,

    Since the issue cannot be reproduced at my side, I would recommend you to see if the instructions on http://blogs.msdn.com/b/akshar/archive/2011/05/01/azure-appfabric-caching-errorcode-lt-errca0017-gt-substatus-lt-es0006-gt-what-to-do.aspx solves the problem.

     

    Best Regards,

    Ming Xu.


    Please mark the replies as answers if they help or unmark if not.
    If you have any feedback about my replies, please contact msdnmg@microsoft.com.
    Microsoft One Code Framework
    Wednesday, October 26, 2011 5:17 AM
    Moderator
  • Yes, the following http://www.microsoft.com/windowsazure/support/status/servicedashboard.aspx reported a "Performance degradation" at that time (but not "Service interruption")

    It now seems to be working.  I would like to know how to handle this gracefully the next time this happens as my whole site went down which is a major worry.

     

     

     

    Wednesday, October 26, 2011 10:42 AM
  • Hi,

    Since the issue cannot be reproduced at my side, I would recommend you to see if the instructions on http://blogs.msdn.com/b/akshar/archive/2011/05/01/azure-appfabric-caching-errorcode-lt-errca0017-gt-substatus-lt-es0006-gt-what-to-do.aspx solves the problem.

     

    Best Regards,

    Ming Xu.


    Please mark the replies as answers if they help or unmark if not.
    If you have any feedback about my replies, please contact msdnmg@microsoft.com.
    Microsoft One Code Framework


    I read that article too but as I am using only ASP.NET Session with the AppFabric Cache Session Store Provider I am not interacting with the DataCacheFactory directly so I could not use this information to help me.  If you can advise how I can apply this information to my situation I would be most grateful.

    Wednesday, October 26, 2011 10:47 AM
  • Hi,

    I am trying to involve someone familiar with this topic to further look at this issue. There might be some time delay.

    Appreciate your patience.

     

    Best Regards,

    Ming Xu.


    Please mark the replies as answers if they help or unmark if not.
    If you have any feedback about my replies, please contact msdnmg@microsoft.com.
    Microsoft One Code Framework
    Thursday, October 27, 2011 3:53 PM
    Moderator
  • It is a temp failure.  Did you define the retry interval and retry count in the session state configuration? http://msdn.microsoft.com/en-us/library/windowsazure/gg185682.aspx 

    ================

    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

     

    Thanks & Regards,

    Emma

    Microsoft Online Community Support

     

    Wednesday, November 09, 2011 8:56 AM
  • I've had the same problem going on for months. In general, my experience over the last 6 months has been that caching will die or experience very poor performance (over 2+ second latency) at least once per week, and sometimes several times per day. Usually it's only for 10-15 minutes, but I've seen several multi-hour outages. That causes requests to slow down enough that they back up in IIS, and eventually my whole site just dies. AppFabric Caching has been responsible for about 95% of my downtime. 

    If I were you, I would not put session state in caching. You're just asking for downtime. Put it in Table or Blob Storage. Those services are pretty close to rock solid.  

    For the record, I have retries set up. Retries don't help when the cache is down for extended periods of time. I've also been talking with Microsoft Support, including calls with the caching product team. They're "investigating." In the meantime, their advice was to implement retry logic, and then more logic to detect when the cache is dead and just stop using it for a while. But they have no sample code for that - just implement it yourself. Uh huh. 

    Thursday, November 10, 2011 8:07 PM
  • LIkewise. Caching fails intermittently and very regularly. Right now, the system status show "intermittent disconnects" - but should supposedly succeed after retrying. Seems...not true. As far as I'm concerned, it's down.
    Thursday, November 10, 2011 8:14 PM
  • The appFabric Cache Session Store Provider (Microsoft.Web.DistributedCache.DistributedCacheSessionStateStoreProvider) you configured in web.config file wrapped the interaction with DataCacheFactory already, so you won't have a chance to apply the blog suggessed change to the connection timeout setting.

    Appfabric cache service can get into temprary degradation because it depends on some other Azure serivces. If any of those services is having trouble, cache service might be impacted. Like any other Azure serivces, when programing with caching service, retry logic is a must.  the configueration of  appFabric Cache Session Store Provider  already contains element to define retry interval & retry times.  During a service degradation, retry can help make the result better.

    You can also consider using other azure service (such as table storage based or SQL Azure ) based session state providers. You can find some implementations from the Azure SDK training kit or on the internet. However those are not officially supported.

    -Emma

    Wednesday, November 16, 2011 6:50 AM
  • I have seen this same issue with ASP.NET session state and the AppFabric cache over the past year, and is exactly as Brian Reischl reported earlier in this thread.  I've reported this to Azure Support as well, and the final conclusion is that it's intermittent and that there is no way to fix this according to them.  Is there any update on this?

    Cheers,

    Matt

    Here is the error again for posterity.  This was caught in the Application_Error section of ASP.NET.

    Source: Microsoft.ApplicationServer.Caching.Client
    Message: ErrorCode<ERRCA0017>:SubStatus<ES0006>:There is a temporary failure. Please retry later. (One or more specified cache servers are unavailable, which could be caused by busy network or servers. For on-premises cache clusters, also verify the following conditions. Ensure that security permission has been granted for this client account, and check that the AppFabric Caching Service is allowed through the firewall on all cache hosts. Also the MaxBufferSize on the server must be greater than or equal to the serialized object size sent from the client.)
    StackTrace:    at Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody)
       at Microsoft.ApplicationServer.Caching.DataCache.ExecuteAPI(RequestBody reqMsg, IMonitoringListener listener)
       at Microsoft.ApplicationServer.Caching.DataCache.InternalResetObjectTimeout(String key, TimeSpan newTimeout, String region, IMonitoringListener listener)
       at Microsoft.ApplicationServer.Caching.DataCache.<>c__DisplayClass68.<ResetObjectTimeout>b__67()
       at Microsoft.Web.DistributedCache.DataCacheWrapper.ResetObjectTimeout(String key, TimeSpan newTimeout)
       at Microsoft.Web.DistributedCache.DataCacheRetryWrapper.PerformCacheOperation(Action action)
       at Microsoft.Web.DistributedCache.DataCacheForwarderBase.ResetObjectTimeout(String key, TimeSpan newTimeout)
       at Microsoft.Web.DistributedCache.BlobBasedSessionStoreProvider.ResetItemTimeout(HttpContextBase context, String id)
       at System.Web.SessionState.SessionStateModule.BeginAcquireState(Object source, EventArgs e, AsyncCallback cb, Object extraData)
       at System.Web.HttpApplication.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
       at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
    User Agent Details: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0)
    Thursday, September 13, 2012 3:05 AM