Memory consumption exceeds host memory size limit

Question

Hi,

We are investigating AppFabric v1.1 feasibility for our application.

There are two hosts in the cluster. one is lead host, and another a normal host, using sql server to store cluster configuration. Both hosts size are limited to 500MB.

<hosts>
            <host replicationPort="22236" arbitrationPort="22235" clusterPort="22234"
                hostId="1203628772" size="500" leadHost="true" account="xxx\xxx"
                cacheHostName="AppFabricCachingService" name="xxx"
                cachePort="22233" />
            <host replicationPort="22236" arbitrationPort="22235" clusterPort="22234"
                hostId="1096232962" size="500" leadHost="false" account="xxx\xxx"
                cacheHostName="AppFabricCachingService" name="xxx"
                cachePort="22233" />
</hosts>

Total 3 client applications are using the cache. After few hours, on both servers, "DistributedCacheService.exe"'s primary memory exceeds 500 MB limit, and reaches up to 632 MB on one server, and 649MB on another server. Below are the statistics gathered after 30 minutes of making all clients OFF (Note: defaultTTL="30").

Size              : 0
ItemCount         : 0
RegionCount       : 1024
RequestCount      : 22516
ReadRequestCount  : 7644
WriteRequestCount : 15234
MissCount         : 11083
IncomingBandwidth : 1013872274
OutgoingBandwidth : 707742538

Surprisingly, "DistributedCacheService.exe" is showing > 570 MB on both servers (even after invoking GC command!).

Looks like, some issue with AppFabric memory management or so?

Please suggest.

Answer 1

The memory limit for the hosts is a limit for your data, its not a limit for the process memory size. It would always be more than data size you configure as we need some memory for other functions. The cache service cant run by just allocating space for cache objects and not using any extra space itself.

As followup, please have a look at the capacity planning guide : http://msdn.microsoft.com/en-us/library/gg186017%28v=azure.10%29.aspx

Answer 2

OK, Reviewed the link provided. Proceeding further, setup organized is as below.

Server Profile:

RAM :    4096 MB
Cache Size Setting (Set-CacheHostConfig):    500 MB
OS Reserved Memory:    3596 MB
LowWatermark Setting (Set-CacheHostConfig): 90.00%    
Adjusted Memory for Caching: 450 MB

Memory Requirements:

Average Object Size (Post-Serialization):    910 KB
Per-Object Cache Cluster Overhead (fixed):    0.5    KB
Adjusted Average Object Size:    910.5 KB
Max Number of Active Objects:    900    
Base Required Memory:    800.24 MB
Internal Data Structures Overhead (5%):    40.01 MB
Total Cache Memory Required:    840.26 MB

Cache Cluster Estimate:

Target Caching Memory per Server:    450    MB
Aggregated Memory Requirement:    840.26    MB
Total Cache Servers:    2    
Total Cache Cluster Memory:    900    MB

Now, we have conducted some test on cluster (with Put operation for 900 objects)

Test1 (After starting cluster):

Data size of  cache (Get-CacheStatistics):    806.84 MB
Service memory – Server1:    508.26 MB
Service memory – Server2:    511.77 MB
Total memory consumed by all servers:    1020.03 MB
After 30 min, data size in cache:    0 MB
After 30 min, Service memory – Server1:    487.48 MB
After 30 min, Service memory – Server2:    491.54 MB

Test2 (Without cluster restart after Test 1):

Data size of  cache (Get-CacheStatistics):    806.84 MB
Service memory – Server1:    592.71 MB
Service memory – Server2:    595.29 MB
Total memory consume by all server:    1188 MB
After 30 min data size in cache:    0 MB
After 30 min Service memory – Server1:    590.95 MB
After 30 min Service memory – Server2:    580.81 MB

Test3 (Without cluster restart after Test 2):

Data size of  cache (Get-CacheStatistics):    806.84 MB
Service memory – Server1:    592.29 MB
Service memory – Server2:    638.1 MB
Total memory consume by all server:    1230.39 MB
After 30 min data size in cache:    0 MB
After 30 min Service memory – Server1:    590.8 MB
After 30 min Service memory – Server2:    585.84 MB

So, now the question is why DistributedCacheService.exe service is not releasing memory while cache statistics show it as released? After 2 hours of inactivity (no client request) in cluster, DistributedCacheService.exe service is still not releasing memory occupied by it. Looks strange...

Something similar has already been posted here: http://social.msdn.microsoft.com/Forums/en/velocity/thread/8c525d8c-91f3-48e2-8437-158d9d3e6d57

Answer 3

DistributedCacheService pre-allocates all the memory configured. It uses this memory for storing object. So it doesn't really matter if objects are stored or not. Reason for doing so is that Cache service is a .net process. And for better GC performance, it is recommended not to remove objects from GEN2. By managing memory itself cache service is able to have better GC performance and lesser GC pauses.

---

MSFT