App fabric crashed - Errorcode <ERRCA0017>: substatus <ES0001>: there is a temporary failure

Question

Hi,

We are having a app fabric setup running on windows server 2008 SP1 server. This is used as session provider across multiple applications. It was running fine for some time. But recently it is getting crashed quite often and the throw the error "Errorcode <ERRCA0017>: substatus <ES0001>"

After 30 minuets, it recovers itself and started to running fine.

Configuration provider: XMLConfiguration provider (XML configuration file is shared within the same server.)

Verified the permission issues and all other configurations. everything is good. 

Crash logs:

=================================================================================================

AppFabric Caching service crashed with exception {System.Runtime.CallbackException: Async Callback threw an exception. ---> System.NotSupportedException: The Write method cannot be called when another write operation is pending.
   at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at System.ServiceModel.Channels.FramingDuplexSessionChannel.EndReceive(IAsyncResult result)
   at Microsoft.ApplicationServer.Caching.WcfServerChannel.CompleteProcessing(IAsyncResult result)
   at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
   --- End of inner exception stack trace ---
   at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
   at System.ServiceModel.Channels.SynchronizedMessageSource.ReceiveAsyncResult.OnReceiveComplete(Object state)
   at System.ServiceModel.Channels.SessionConnectionReader.OnAsyncReadComplete(Object state)
   at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
   at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
   at System.Net.Security.NegotiateStream.ProcessFrameBody(Int32 readBytes, Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.NegotiateStream.ReadCallback(AsyncProtocolRequest asyncRequest)
   at System.Net.FixedSizeReader.CheckCompletionBeforeNextRead(Int32 bytes)
   at System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)
   at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
   at System.ServiceModel.Channels.ConnectionStream.ReadAsyncResult.OnAsyncReadComplete(Object state)
   at System.ServiceModel.Channels.SocketConnection.AsyncReadCallback(Boolean haveResult, Int32 error, Int32 bytesRead)
   at System.Runtime.Fx.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
   at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)}. Check debug log for more information

========================================================================================

Could someone pls help me know what is the issue?

Answer 1

Hello Vinesh,

Which version of Appfabric server are you running? Do you have a app to repro this issue?

Can you share your cache cluster config. Do you have the process crash dump when this crash happened?

Thanks

Answer 2

Hi Ranga,

Thanks for the response.

Appfabric version is 1.0.0 (http://www.microsoft.com/en-us/download/details.aspx?displaylang=en&id=15848)

There is no certain ways to reproduce the issue. It occurs in our production environment during night time when the traffic is quite lower than the day time.

pls find the cachecluster configuration below:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <configSections>
        <section name="dataCache" type="Microsoft.ApplicationServer.Caching.DataCacheSection, Microsoft.ApplicationServer.Caching.Core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
    </configSections>
    <dataCache size="Small">
        <caches>
            <cache consistency="StrongConsistency" name="CacheName">
                <policy>
                    <eviction type="None" />
                    <expiration defaultTTL="20" isExpirable="true" />
                </policy>
            </cache>
            <cache consistency="StrongConsistency" name="default">
                <policy>
                    <eviction type="Lru" />
                    <expiration defaultTTL="10" isExpirable="true" />
                </policy>
            </cache>
        </caches>
        <hosts>
            <host replicationPort="22236" arbitrationPort="22235" clusterPort="22234"
                hostId="1427243344" size="16381" leadHost="true" account="networkdomain\someaccount"
                cacheHostName="SOMECACHESERVICE" name="hostname"
                cachePort="22233" />
        </hosts>
        <advancedProperties>
            <securityProperties>
                <authorization>
                    <allow users="Hostname\localuser" />
                    <allow users="NT Authority\Network Service" />
                </authorization>
            </securityProperties>
        </advancedProperties>
    </dataCache>
</configuration>

I don't have process crash dump available now.

Answer 3

Hi Ranga,

Could you find any solution? Is it a bug in App fabric cache version 1.0?

Meanwhile we tried the below option.

Our app fabric cache session is shared across two application (A, B). We found a pattern whenever the issue occurs in production.

Those application pool's idle time out was set to 20 mins.

If A is active and B's worker process shut down due to inactivity for 20 minutes, the app fabric crashed most of the times.

So we set  application pools idle time out to 0 mins. Then app fabric cache was working fine for more than a week in the entire web farm.

Since yesterday we started seeing the issue in one of the app server, even though its idle time out is set to "0".

Please share if you have any idea.

Answer 4

Which version of Appfabric cache are you running on? Looks like this issue was fixed in the Appfabric 1.1 March update.

Answer 5

We are using Appfabric version is 1.0.0 (http://www.microsoft.com/en-us/download/details.aspx?displaylang=en&id=15848).

We will upgrade very soon and share the results.

Answer 6

try with lock and unlock

public static Object AddAndUpdate(string CacheKey, Object obj, int timeSpan)
        {
            if (obj == null)
                return null;
            OperationCache.Put(CacheKey, obj);
            OperationCache.ResetObjectTimeout(CacheKey, TimeSpan.FromMinutes(timeSpan));
            return obj;
        }
private static DataCacheFactory CacheFactory
        {
            
            get
            {
                if (_cacheFactory == null)
                {
                    lock (_cacheFactoryLock)
                    {
                        if (_cacheFactory == null)
                        {
                            try
                            {
                                _cacheFactory = new DataCacheFactory();
                            }
                            catch (Exception ex)
                            {
                                // Perform any logging and rethrow.
                                throw ex;
                            }
                        }
                    }
                }
                return _cacheFactory;
            }
        }

        private static DataCache OperationCache
        {
            get
            {
                if (CacheFactory != null)
                {
                    if (_operationCache == null)
                    {
                        lock (_operationCacheLock)
                        {
                            if (_operationCache == null)
                            {
                                try
                                {
                                    _operationCache = CacheFactory.GetCache("CacheName");
                                }
                                catch (Exception ex)
                                {
                                    // Perform any logging and rethrow.
                                    throw ex;
                                }
                            }
                        }
                    }
                    return _operationCache;
                }
                return null;
            }
        }

Answer 7

Hi sameer,

We have configured AppFabric cache as our application session state provider. Means it doesn't require any code change in the application. 

Per the below documentation, all we need to do is little change in application configuration file (web.config).

http://msdn.microsoft.com/en-us/library/ee790859(v=azure.10).aspx 

Everything is configured as needed and set up is working like a champ in the peak traffic. It crashes more frequently only during the night time.