Monday, September 13, 2010 6:48 PM
We are developping a silverlight application using WCF Data Services and WCF services.
On our server running IIS and hosting our business services, we have AppFabric installed with all 3 services running (we have 4 different environment setup, some with cache clusters).
The problem we have is that on all of our machines hosting the 3 services, the EventCollectorService.exe and WorkflowManagementService.exe eventually fill up all of the memory, and we get throttling on our CacheCluster. This almost looks like a memory leak, because it takes about 2 days before the memory is full (actually more like 94-95% constant)
Using the command Get-CacheClusterHealth, I get 100% throttle for environments using only 1 server, and I get 50% throttle and 50% healthy for environments using 2 hosts.
Edit : one thing I forgot to mention is : we aren't doing anything, we are not using the application on those servers yet and we don't have workflow services at the moment (but we will have long running workflows very soon, which is why we are trying to use AppFabric).
Edit 2 : also, from this point on, all our wcf services are non-responsive...
Am I missing a configuration of somekind ? We have the exact same problem on 3 different single server setup and 1 multi server setup.
Tuesday, September 14, 2010 1:00 PMHas no one had this problem before?
Thursday, September 16, 2010 5:02 PM/bump
Tuesday, September 21, 2010 7:50 PMAfter further investigation, I can say that about every minute, the CPU rises to almost 100% for about 2 secs, the I/O on disc rises to 250Kb/s+ for about 2 secs, the network usages rises to 1Mb/s+ for 2 secs and the memory used for by the WorkflowManagementService and EventCollectorService increases by 3-5Mb each and never releases it...
Tuesday, September 28, 2010 12:32 AM
How many applications do you have deployed?
Are you continually updating any configuration files, like web.config?
Both of these applications crawl the IIS configuration to discover applications you have deployed.
Wednesday, September 29, 2010 2:54 PMWe have 3 web applications hosting wcf services that we've built deployed on that machine. But, we also have Sharepoint Shared Services and a Sharepoint Portal installed on that web server. I did try to stop all of those sharepoint related web sites for a while, but the behavior was the same so I thought that was probably not the problem. Do you think it could still be ?
Wednesday, September 29, 2010 3:38 PM
It could be, but I'm not certain
Perhaps we can try something - Do you have more than 1 instance store that WMS needs to monitor? If not, can you follow the instructions on this page -> http://msdn.microsoft.com/en-us/library/ff383400.aspx on configuring WMS to monitor a particular instance store rather than crawling for instance stores.
This might be a viable workaround for you.
Thursday, October 07, 2010 2:33 PM
Are you runnig the BOTH the AppFabric Caching Service, and the AppFabric Hosting Service, on the same machne? This is what I gather from reading the problem description. If so, that is the issue. Running both AF Caching and Hostig on the same server is not recommended. If this is the case, move them to different servers and your problem will be fixed. I write about this in my Windows Server AppFabric Web Farm Guide.
"Within a Web farm of AppFabric hosting and management servers, we recommend that any servers running the AppFabric caching service exist separately outside of a load-balanced cluster of AppFabric hosting and management servers."
"...diagram shows a load-balanced AppFabric Web farm cluster that uses the AppFabric caching server to store ASP.NET session state. This caching cluster is a separate logical cluster from the AppFabric hosting and management Web farm cluster. "
"It is not a requirement for cache client applications taking advantage of AppFabric caching to be configured to use Windows Server AppFabric hosting and management."
You can go here to read about and download the AppFabric Web Farm Guide: http://social.technet.microsoft.com/wiki/contents/articles/appfabric-web-farm-guide-intro.aspx.
Hope this helps!
Mike McKeown - Senior Programming/Writer, Windows Server AppFabric
- Proposed As Answer by MikeMcKeown-Aditi Cloud Architect Thursday, October 07, 2010 2:33 PM
Wednesday, June 22, 2011 2:28 PM
Hi, following this thread I have come up with some idea because something similar is happening to our production system.
Do you have the web farm service running? I am suspecting a memory leak on the WMS and event colletor services occurring upon web.config changes.
For some reason, the web farm service syncs files even though they haven't changed and is doing it every minute, hence the services end up consuming a lot of memory and not actually working until restarted.
Wednesday, November 02, 2011 2:01 PM
I have run into this problem as well. In my environment, we have AppFabric Workflow Management Service installed (no caching service installed). There are maybe 20 to 30 IIS hosted workflow services, each with their own application pool (32-bit mode, because 64-bit mode uses too much memory!).
Several times, the WorkflowManagementService.exe process was allocated 31GB of memory (working set) on a server with 32GB of memory. Needless to say, the server was crippled until the service was restarted. After restarting the service, it consumed around 200MB of memory, then after a few days, it gradually increased until it was required to be restarted again.
There's definitely some sort of memory leak in this service.
Any update on this issue?
Thursday, November 24, 2011 9:18 AM
Currently, my WorflowManagementService.exe process memory is growing at about 1GB per hour. I have verified that the AppFabric caching service is not installed. However, the caching client is installed, but I doubt this is the cause of the problem.
We currently have a script that restarts the AppFabric Workflow Management Service every 8 hours, an ugly workaround that we have to live with until this gets fixed.
I have a feeling that this is related to a problem we have with AppFabric auto-start. Even with the hotfix installed, any new workflow services that are added to IIS with auto-start enabled will run on all application pools. Currently, the only workaround we found is to taskkill all w3wp.exe processes (yet another ugly workaround).