High memory utilization of appfabric service(DistributedCacheService.exe)

Question

I have 3 servers each 12GB of physical memory & each host is configured to use 10GB memory for Cache. Wea re using named cache for storing ASP.NET session state objects.

When i started cache cluster, day by day memory utilization on each host  is started increasing. DisstributedCacheService.exe process on each server is taking 9GB memory now, want to know how i can reduce the memory utilization with out impacting end users.

Cacheconfig:

CacheName                : default

TimeToLive               : 120 mins
CacheType                : Partitioned
Secondaries              : 1
MinSecondaries           : 0
IsExpirable              : True
EvictionType             : LRU
NotificationsEnabled     : False
WriteBehindEnabled       : False
WriteBehindInterval      : 300
WriteBehindRetryInterval : 60
WriteBehindRetryCount    : -1
ReadThroughEnabled       : False
ProviderType             :
ProviderSettings         : {}

Cachehostconfig:

HostName        : server1

ClusterPort     : 22234
CachePort       : 22233
ArbitrationPort : 22235
ReplicationPort : 22236
Size            : 2048 MB
ServiceName     : AppFabricCachingService
HighWatermark   : 90%
LowWatermark    : 70%
IsLeadHost      : True

Web.config settings to connect to appfabric cache server:

 <dataCacheClient>
  <!-- cache host(s) -->
  <hosts>
   <host name="server1" cachePort="22233"/>
   <host name="server2" cachePort="22233"/>
   <host name="server3" cachePort="22233"/>
  </hosts>
 </dataCacheClient>

<sessionState mode="Custom"
       customProvider="AppFabricCacheSessionStoreProvider">
   <providers>
    <add
      name="AppFabricCacheSessionStoreProvider"
      type="Microsoft.ApplicationServer.Caching.DataCacheSessionStoreProvider, Microsoft.ApplicationServer.Caching.Client, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
      cacheName="defalut"/>

Regards

Sivaprasad

Answer 1

It appears you are running version 1.1 of AppFabric. Memory management changed DRASTICALLY in that version. You will not want to allocate more than 50% of your total server memory to the cache, so 6gb in your case. The overhead needed for the service itself, memory management/garbage collection/etc. seems to have increased with that version. You will find that under heavy load the 1.1 version also consumes much more CPU, presumably because of more aggressive garbage collection. I haven't noticed this unless you are pushing a fair amount of load at it, however.

Hope that helps.

Answer 2

I have the same problem.

This is my config:

CacheName                : TEST
TimeToLive               : 10 mins
CacheType                : Partitioned

Secondaries              : 0
MinSecondaries           : 0
IsExpirable              : True
EvictionType             : LRU
NotificationsEnabled     : False
WriteBehindEnabled       : False
WriteBehindInterval      : 300
WriteBehindRetryInterval : 60
WriteBehindRetryCount    : -1
ReadThroughEnabled       : False
ProviderType             :
ProviderSettings         : {}

I only use AppFabric 1.1 for Session Store Provider.

The following is my Session and DataCache Config

<sessionState
		sessionIDManagerType="XXX.Configurations.SessionIDManager, CtripSZ.Frameworks.Configurations" mode="Custom" customProvider="AppFabricCacheSessionStoreProvider"
		timeout="30">
	<providers>
		<add
				  name="AppFabricCacheSessionStoreProvider"
				  type="Microsoft.Web.DistributedCache.DistributedCacheSessionStateStoreProvider,Microsoft.Web.DistributedCache"
				  cacheName="default"
				  sharedId="OnlineSiteSession"
				  />
	</providers>
</sessionState>

<dataCacheClient>
	<hosts>
		<host name="172.18.21.46" cachePort="22233"/>
	</hosts>
	<securityProperties mode="None" protectionLevel="None" /> 
</dataCacheClient>

In some page, only get session data from appfabric cache, every time when I refresh this page , memory will increase more than 200kb, but never reduce.

I don't know why. In my product environment,  DistributedCacheService.exe use more than 7G memory ( from 8/2/2012 ).

I have a table more than 20000 rows, I stored it as a List, and I notice that, every time, use "GetIfNewer" will increase about 60KB, "Get" will increase more than 1000 KB, but never reduce.

CacheName: TEST


Size              : 19083264
ItemCount         : 3
RegionCount       : 2
RequestCount      : 839
ReadRequestCount  : 830
WriteRequestCount : 3
MissCount         : 773
IncomingBandwidth : 18948419
OutgoingBandwidth : 265618715

My data only use 19083264

Why ? 

Please help me!

Answer 3

It appears that you are using the newer (1.1) session state provider. The only time I tried that provider our load tests showed that it performed worse than the 1.0 provider. So other than the one test, we have stuck with the 1.0 session provider even after we upgraded to AppFabric version 1.1 for the caching service.

Although I noticed memory leaks similar to what you describe when we were on AppFabric 1.0, I have not noticed such problems on 1.1 - using session state provider 1.0.

My advice would be to go back to using that provider and see if it makes a difference, unless you're taking advantage of new features in the 1.1 session provider that is.

One other question though - how much total memory do your AppFabric servers have and how much do you have configured for cache on them? If you have more than half of the amount configured for cache you may see severe memory consumption issues in version 1.1. The older version didn't really have this behavior probably because the memory management/garbage collection wasn't nearly as aggressive and didn't cause as much memory overhead itself.

Answer 4

Yesterday, we do some  test with loadrunner.
Test environment:
Win2008 R2 64bit  4G Memory  i5-2400 CPU  AppFabric 1.1
Test time : 9:30am ~ 4:30pm (Only appfabric service run)
At the beginning, the memory always grow until arrival at 93% (11:00 am). DistributedCachingService.exe useage about 2.5G memory.
After, memory still grow but begin release, remained at 93%. AppFabric always work fine.
CPU max usage 22%

After test, I got the data:

位于命令管道位置 1 的 cmdlet Get-AFCacheStatistics
请为以下参数提供值:
CacheName: default
Size              : 12567552
ItemCount         : 211
RegionCount       : 529
RequestCount      : 23557985
ReadRequestCount  : 23500240
WriteRequestCount : 42148
MissCount         : 23456435
IncomingBandwidth : 6506805008
OutgoingBandwidth : 5351938045


 Cache "default" only used for session, there were 211 session items in it before caching service close. Data size only 12567552 byte.
 Total output data 5351938045 byte. Total request 23557985 time.
 
CacheName: TEST
Size              : 19286016
ItemCount         : 5
RegionCount       : 3
RequestCount      : 26475
ReadRequestCount  : 26440
WriteRequestCount : 6
MissCount         : 26141
IncomingBandwidth : 19869616
OutgoingBandwidth : 667709001

Cache "Test" only store some small data (only big data is a List , have more than 50000 items),


From the test result, we can see appfabric will allocate memroy as much as possible.
So , if have other services on the server, AppFabric will squeeze othere service's memory

Is it right ?
Caching Client we use AppFabric 1.0's dll, because in our team , some developer still use windows xp.

I'm sorry , My English is very poor.

Answer 5

You mentioned other services - other than standard Win2k8 services you need I wouldn't be running any other apps on your AppFabric servers. They seem to like being the only thing running on the server.

Also, 4gb doesn't seem like a lot of memory for a server these days. There's memory overhead for the OS itself, and AppFabric (particularly v1.1) is very memory hungry. I still don't recall seeing your value you have for the cache amount on your servers - do you have it set to the default, which would be half or 2gb? On a 4gb box that leaves very little room for the OS, other needs of the distributed cache service, and miscellaneous things that may be running. Have you calculated approximately what space you need for your cache client? If you have a healthy margin you might want to lower cache amount to something like maybe 1.5gb on those servers.

You WILL see memory growing at the beginning (with AppFabric 1.1) until it consumes pretty much the entire amount you've allocated, plus whatever overhead distributedcacheservice needs. So what you described seems relatively reasonable to me. Your memory and CPU behavior will change between version 1.0 and 1.1, so if you're going by what you were used to in 1.0 that's not valid.