worker process memory issues in conjunction with appfabric

Question

Hi,

I have set up appfabric to run on 4 web servers with a .net web application - appfabric being its caching mechanism. It recently went live on these servers but ran in to a problem. I found that for the sites iis worker process on each of the boxes that the memory constantly increased in a short period of time from 200K to over a 1000K until it recycled. I have local cache enabled timeout based on items cached in our cluster for 1 hour. High availability is enabled, xml cluster config setup with 3 lead hosts.

To resolve the issue, in the short term was to decrease the cache time to 5 minutes and disable the local cache. This resulted in the worker process memory not reaching the same levels as before, but still marginally increasing over time. There are about 10,000 items in cached on average, with the objects varying in size from strings to dto objects.

Based on this I have some question to assist me resolve this issue and just to gain further knowledge on app fabric as I am new to it.

Does the local cache deseralized object get stored in the worker process memory? Or what memory is the local cache stored in?

I was concerned that it the issue may be related to the size of objects in cache, could this be possible? Is it the size of the items in cache or the number of them hamper appfabric cache performance?

One of the servers in the cluster is a virtual server, same behaviour was seen on this. Is there anything I should be looking at related to the virtual server appfabric setup? At present it is set up same as the other cache hosts but is not a lead host?

I read somewhere but dont understand it, is that when an item is removed from cache the key remains in cache for a period of time. Is that correct?

I would appreciate any help in assisting me with this while I tweak both my application and setup to get optimum performance from this.

Answer 1

Any assistance or feedback on this would be appreciated.

Answer 2

Does the local cache deseralized object get stored in the worker process memory? Or what memory is the local cache stored in?

 - Local cache is in-process to avoid network/deserialization cost. It occupies worker process memory.

I was concerned that it the issue may be related to the size of objects in cache, could this be possible? Is it the size of the items in cache or the number of them hamper appfabric cache performance?

- Both size and count matter. Size and count of objects in a process impact GC that happens. If size of object is large, it would take longer to bring it from remote cache servers. Beyond that there should not be significant difference in performance as far as app-fabric is concerned.

One of the servers in the cluster is a virtual server, same behaviour was seen on this. Is there anything I should be looking at related to the virtual server appfabric setup? At present it is set up same as the other cache hosts but is not a lead host?

I read somewhere but dont understand it, is that when an item is removed from cache the key remains in cache for a period of time. Is that correct?

-  When object is remove, key is also removed. If notifications are enabled, Operations are stored on cache server, key is used to identify the operation. Otherwise once object is removed, key is also removed.

I would appreciate any help in assisting me with this while I tweak both my application and setup to get optimum performance from this.

---

MSFT