Error while implementing custom Appfabric sessionState for an asp.net website with ErrorCode<ERRCA0016>

Question

Hi Guys,

We are using windows sever Appfabric to store frequently used reference data, and we had implemented it by a creating a API to get and put data. This worked fine and improved the performance. But now, we tried implementing custom session state provider with Appfabric, it was working fine functionality wise in our environments.

But when we went for a load test, we faced the below error randomly, the percentage of errors with total hits varied with the network traffic, like, low errors in off peak hours and the error count increased in peak hours.

Description:
ErrorCode<ERRCA0016>:SubStatus<ES0001>:The connection was terminated, possibly due to server or network problems or serialized Object size is greater than MaxBufferSize on server. Result of the request is unknown.

Stack Trace:
[DataCacheException: ErrorCode<ERRCA0016>:SubStatus<ES0001>:The connection was terminated, possibly due to server or network problems or serialized Object size is greater than MaxBufferSize on server. Result of the request is unknown.]
   Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody) +337
   Microsoft.ApplicationServer.Caching.DataCache.ExecuteAPI(RequestBody reqMsg) +29
   Microsoft.ApplicationServer.Caching.DataCache.InternalGetAndLock(String key, TimeSpan timeout, DataCacheLockHandle& lockHandle, String region, Boolean lockKey) +186
   Microsoft.ApplicationServer.Caching.DataCache.GetAndLock(String key, TimeSpan timeout, DataCacheLockHandle& lockHandle) +101
   Microsoft.ApplicationServer.Caching.DataCacheSessionStoreProvider.GetItemExclusive(HttpContext context, String id, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actions) +1204
   System.Web.SessionState.SessionStateModule.GetSessionStateItem() +173
   System.Web.SessionState.SessionStateModule.BeginAcquireState(Object source, EventArgs e, AsyncCallback cb, Object extraData) +793
   System.Web.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +173
   System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +194

Any help is greatly appreciated.

Thanks In Advance!
Praveen

Answer 1

Hi Praveen,

Can you try the following things -

1. Increase the MaxConnectionsToServer in client config

2. Increase the TransportProperties.ReceiveTimeout in client and cluster config

I am assuming that since you are storing SessionState objects, the data size is small and transportProperty MaxBufferSize should not have any impact.

Answer 2

Hi Ajay,

Thanks for ur reply!
You are right, only 1kb object only stored in the sessions.
But i had already increased the receive time out and maxbuffersize as well in transportproperties of both client and server.
For a particular run, i had even tried providing the maxConnectionsToServer as 10, even then i get the above error randomnly,

Please find below the transport properties set in client and server,

<transportProperties maxBufferPoolSize="2147483647" maxBufferSize="2147483647"
        channelInitializationTimeout="60000" receiveTimeout="900000" />

---

Regards,
Praveen

Answer 3

What is the frequency of this error ?  Can you also share your load test configuration detail.

Answer 4

Hi Ajay,
Load test with 300 concurrent users, making around 100k overall hits to the test page, among that there are 25 above said errors.

---

Regards,
Praveen

Answer 5

Do you have any retry logic upon connection terminated errors ? If not, then it will be good to add and see how it goes.

A couple of other thread around similar issues -

http://social.msdn.microsoft.com/Forums/eu/velocity/thread/47a5222a-d746-43e2-b313-47b0d1da404a

http://social.msdn.microsoft.com/Forums/eu/velocity/thread/975bb18e-3402-4251-bd10-dd084e5f308f

Answer 6

Since i had used Appfabric only for sessions, and i had made only config changes for this, i cant go ahead making any retry operation in that, as all put and get are taken care by itself.

On the other thread,  it was said that the WCF channel idle timeout has to be handled, but surprisingly, im getting this error in less than a minute after starting the execution of the load test.

---

Regards, Praveen

Answer 7

Are you using Appfabric 1.1 ? If yes, you can set retryInterval and retryCount parameters in session state provider config. The below link mentions it -

http://msdn.microsoft.com/en-us/library/hh361700.aspx

Answer 8

We are using appfabric 1.1 only, but our app is a asp.net 2.0 app, so we are not able to use  the .net 4.0 provider,

Microsoft.Web.DistributedCache.DistributedCacheSessionStateStoreProvider
provider which supports retry operation, instead we use the below provider for Custom sessions,

Microsoft.ApplicationServer.Caching.DataCacheSessionStoreProvider

which does not support retry operation i guess.

---

Regards, Praveen