none
Performance glitch with high RAM usage RRS feed

  • Question

  • We have a high memory usage process running here on a box that has 256GB of RAM.  We are currently typically using 50-60GB of RAM when our test version of the application is running, but expect this to grow when the application deploys.  The problem that we have is that when we are testing the application we are getting periodic slow downs.  The machine we are using has 48 cores and is running Windows 2008 r2 enterprise.

    Initially we are servicing roughly 5 complete request cycles per second (asking our application for a response, it computing the response and outputting it), then after about 90 seconds everything slows down and we are typically only achieving a response time of about 2 seconds for each request.  After about 20 seconds everything returns to normal and then we will experience another 90 seconds of fast responses before the cycle begins again.

    We have turned off virus checkers, and every other service etc. that is running on the machine.  There is no application apart from ours running on the machine.  We have tried requests coming both locally and across the network but the performance doesn't change.  We have even disabled paging to see if that could be the root cause, but without any luck.

    When we observe performance in "process explorer" what we see during the "slow" periods is there is close to 0 CPU activity, 0 disk activity and 0 network activity, it just appears as if everything stops.  However, we do notice one event that is consistent with the slow downs.  Whenever a slow down occurs the Kernel thread jumps to the highest activity thread in the process table (roughly 2% cpu usage).  Further, when we examine the threads that are running within the kernel it is always the same thread which is active.  This thread is identified as "KeDetachProcess".  Reading up on this undocumented call seems to indicate that it is related to activities that manage the RAM.  This would seem to be consistent with our experience as this problem has only emerged as we have ramped up the RAM usage of our application.

    What we need to know is:

    • Are we right in diagnosing this thread as the source of our slow down
    • If so, why is it inhibiting progress by our application so badly
    • and are there any ways in which we can reconfigure Windows so as to remove this bottle neck

     

    Tuesday, August 9, 2011 12:54 PM

All replies

  • Hi,

    I think this is a good example which we can try to unalyze by xperf.

    I understand that you has encounter one realy shortly cpu jump momentum.

    With the following command you should able to "pic up" the exactly thread for this.

    1. setup one machine with internet connection availability

    2. follow the instruction form WPT online help (using the chm file) and configure the symbol path to MS symbol server

     

    3. install the WPT version what is required for you os

    4. please try the following command first:

    start "xperf -on DiagEasy+Profile+PROC_THREAD+LOADER"

    5. wait ~2minutes and make sure that on cpu jump was done

    6. Stop the trace with the following command

    Stop "xperf -d C:\Logs\Thread.etl" --> I think this will take time because of you powerful machine :)

    7. copy the Thread.etl to your additional machine to check the trce or open it on the big machine (need access to the symbol files)

    8. mke sure that the symbol support is running
        ( typically I check the new builded "symcache" folder and expect to see some new files into this folder)

    9. Wait if the XperfViewer.exe has download all requested symbol files --> take time...

    10. Expant the frame list on the left

    11. Add the frame for "CPU sampling by Thread"

    12. select the "CPU sampling by thread" window and at the context menue you will find the option "summary table"
         At this point please make sure that the option "load symbols" is selected as well

    check it out :)

    BR

     

     


    • Proposed as answer by RPA601SG Sunday, September 18, 2011 8:39 AM
    Tuesday, August 23, 2011 9:46 PM