Performance issues

Question

User1631061264 posted

Hi,

I with someone else have slowly been trying to track performance issues on our live website.

We have been noticing that the CPU regularly goes up to 100% for about 30 seconds - 60 seconds, after tracking we know this is caused by the process our website is using. After much research using performance counters we have discovered that the CPU increase coincides with % Time in GC.  The % time in GC spikes up to 90 - 100% for the same time that the CPU increases to a 100% The threshold for % in GC is normally 5%!.

Our next step is to try and find out what  (in a very large website) is causing the issue. As a result we are going to install the CLR profiler. From what I have read this is very invasive and should not normally be used in a production environment. However, we cannot currently replicate this issue on our own in house servers.

I would appreciate any advice anyone could give to help us mitigate the effects of the profiler to the live public, also if anyone has any advice on how to reduce the % time in GC I would be very thankful.

Along with this issue we have also noticed a large amount of application restarts in our web application. When checking we can see that when an approximate number of page have compiled this then causes the application restart. At one point we noted 5 restarts in 25 minutes. We use the Immediacy CMS (now Alterian CMS) and have wondered whether it is the CMS caching each page as it is initially clicked on. If anyone has any other suggestions on the cause on how to mitigate please do let me know.

Many thanks in advance

Phil

Answer 1

User-1636183269 posted

If your database server is reaching 100% cpu usage then download your database locally and test it. Generally profiler is not keep on poduction server. If any other service is installed on server that provide all list. Else task manager check which service is consuming more and list down and give it to us.

Answer 2

User-952121411 posted

Along with this issue we have also noticed a large amount of application restarts in our web application. When checking we can see that when an approximate number of page have compiled this then causes the application restart. At one point we noted 5 restarts in 25 minutes.

 

This is normal based on your explanation.  A recycle will be forced for several reasons, one being if too much memory is consumed and surpasses the set threshold.  This behavior can be configured in IIS, and the link below can help with that:

Configure Application Pool Recycling (IIS 6.0):

http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/1eee28e2-b319-4b4e-8267-a8c0aa0dcf36.mspx?mfr=true

Fixing this is really the latter step as it is not the cause of the usage, but may help with configuration of intensive use sites and its corresponding server hardware capabilities.

Answer 3

User1631061264 posted

Hi Sandeep,

Thanks for your message, our database is not at 100% CPU, the problem is definetly with the garbage collection and the ASP.NET process (w3c something I think it is listed as).

However as part of this project we have looked at database performane and our making some updates to indexes and procedures.

Thanks

Phil

Answer 4

User1631061264 posted

Hi,

Is there any benefit in increasing the allowed number of compiles before a restart.

With our figures there are 30 compiles before a restart, this mean that in 25 minutes there could be a total of 150 compiles in 35 minutes. We could updated this to around 200 compiles meaning there is a restart every 30 minutes instead.

However there would there be any downsides to doing this?

Phil

Answer 5

User-967169866 posted

Do you do things like string s = ""; while (x < 10)  { s += x.ToString();}

you should take a look at what kind of objects you're creating and if you're properly disposing all important objects like file streams, network streams, database related objects, and such.

Take a look at Windbg and sos.  That's not as intrusive as the clr profiler.  It's a bit more complicated to work with since you have to do the whole .loadby stuff and know what you're doing.  however, if you google "Windbg Cheatsheet" there's a quick guide on how to use it for typical triaging.

Answer 6

User1631061264 posted

Hi Kragie,

Thank you for the tip on Windbg and sos, I will definetly take a look at that.

We do not use strings like that but i know parts of the wesbite are coded rather badly. We do use SringBuilder which I have read can cause problems and you should use the writer instead. There is also a case where we make a huge database call which returns 33 tables in a dataset which I have also though could be the cause of the some problems.

However, I was looking at the performance counters this afternoon and everything in the GC was behaving itself (though we will still getting the 100% CPU for 30 seconds occuring).

This was really strange as in the morning and last night the % time of GC was huge and always happened with the 100% CPU.

Is this a normal issue that can happen with GC on websites (as people are not using some particular peice of code) or does it mean that it was a red herring.

Thanks

Phil