this issue seems complex, i will try to explain as much as possible.
i have written an application, based on .Net 3.5 in VS 2010, and it was as quick as i expected before <~1ms per request>. and i tried to use performance wizard in VS to find out if anything else i can improve. nothing found, so i closed performance wizard, cleaned all profiles left. so suppose the project & all source code should be the same. after a rebuild, the app ran much slower than before, <~20ms per request>. but after i attach the performance profiler to the app, even without restarted, the perf was increasing, and was back to normal <~1ms per request>. detach & shutdown VS, the perf was dropping as well, <~20ms per request>, also no restart needed.
i tried on another machine, the perf is almost as expected, and on another account of the same machine, the perf is also bad.
so i think the performance profiler should have changed some system settings, since i have not used it before. and i have diffed, the binary is the same before / after attach / detach profiler.
does there anyone have similar experience before, any configuration i can make to let my machine work again?
thank you.domingo, 24 de julho de 2011 19:51
Todas as Respostas
after a rebuild, the app ran much slower than before, <~20ms per request>. but after i attach the performance profiler to the app, even without restarted, the perf was increasing, and was back to normal <~1ms per request>. detach & shutdown VS, the perf was dropping as well, <~20ms per request>, also no restart needed.
This behavior is very strange. It's obvious that attached profiler cannot increase application performance, especially 20 times. Only 2 possible reasons seem possible to me:
segunda-feira, 25 de julho de 2011 06:58
- Something is affecting your application performance in non-uniform pattern and your performance measurements are not quite accurate (did you carry series of measurements to exclude noise, say, run a request 1000 times in a row and average the result?)
- The profiler has changed environment settings and modified the .NET runtime mode. This seems to be the only way the profiler might affect your application AFTER it has been rebuilt completely. And now the profiler might be switching that setting from one state to the other on attach/detach, but instead of setting it to the correct state, it just switches it in a cycle 1->0->1->0->1->0... I'm not sure if this is the case, I'm just guessing. But if you have done accurate measurements, then it is highly probable.
thank you for your reply, i have tried several more solutions, but still cannot find out the root cause.
now when profiling, the speed is quicker than i expected, when normal running, it's slower.
the basic structure of my app is,
a set of timers to receive tcp connections, and send the requests into a threadpool. <since each request may need up to 100+ individual work items, and i do not want one request timeout stuck all the tcp connections>. the timer thread wait sync for the threadpool to finish the request. the threadpool, which is not .net managed threadpool, will sleep 1ms if no job arriving, otherwise work continually. and the timer thread check the status of each work items, if at least one work item has not been finished, it will sleep 1ms.
so say, if one request needs 128 work items to finish, and the threadpool has 32 threads, each work item needs <100 ticks to be finished. ideally 1ms before really get the tcp data, 1ms to finish all work items if all the working threads in threadpool is not idle, 1-2 ms for timer to collect all the done info of each work item. the overall time needs should be ~5ms. but when normal running, it's up to 20ms. it's hard to debug in this kind of situation, i have just found the threads in thread pool are idle even with high QPS.
i tried to use AutoResetEvent for threads in thread pool to resume to work, and seems it changes little.
i cannot confirm all the code i wrote are correct, but i do really cannot explain the 'speedup' behavior when profiling.
some more strange status i have found when profiling,
1. Console.Out.WriteLine speedup
2. exception throwing speedup
3. threads resume quicker after sleep 1 ms
P.S. i am using Windows 2003 x32 Enterprise version with SP2, Visual Studio 2010 SP1, .Net 3.5 for my app.segunda-feira, 25 de julho de 2011 18:29
just added timeBeginPeriod & timeEndPeriod for 1ms sleep, and make both cpu and performance happy. seems profiler will help to do so, and after detach, it will call timeEndPeriod.
since i am using sleep too much in multi threads model, though it's not a good way when now we get event / monitor etc to handle. but it's do convenient when one thread waiting for multi other threads to finish, but not stop.segunda-feira, 25 de julho de 2011 19:54
The really important thing I tried to point to was that having 1 ms for serving a net request is low enough to have lots of performance affecting factors: socket, buffers lags, and in your case add there time for context switching (since you use Thread.Sleep). These factors might have random strength pattern, so you should make sure that your 20 times difference is consistent when repeating same requests multiple times. If you are, that's OK.
Off topic, but it seems to me that you have misunderstandings of threading and synchronization basics.
since i am using sleep too much in multi threads model, though it's not a good way when now we get event / monitor etc to handle. but it's do convenient when one thread waiting for multi other threads to finish, but not stop.
When you call Thread.Sleep(x), two things are important:
- The thread will be removed from CPU execution queue. Sleep always causes context switching, which takes lots of CPU ticks. This means it will stop executing.
- The parameter x is the lower bound for the actual sleep interval. It means that the thread will not wake faster. But the real sleep time depends on when the OS puts it back on track and on the speed contexts are switched at. So, it might sleep for more than 1 ms.
This is why thread pool threads should never be blocked or got asleep.
If you really need to throttle a thread for a small period of time (several processor ticks) without taking it off the CPU, you should probably use Thread.SpinWait.
Or consider redesigning you system to diminish wait numbers.
Hope this helps.terça-feira, 26 de julho de 2011 03:30
thank you for your reply, the solution i am using is just for reproing the 'speedup' behavior of profiler.
what i really want is a thread does not eat all remaining cpu time, so i do not use Thread.SpinWait, also it's able to switch back in a time as short as possible.
as i know, usually Windows will switch threads in 10-20ms, so i think any kind of mutex / monitor / event / Begin & End model will not be able to be noticed below this bound. i will write some small tests to prove my assumption.
p.s. after adding timeBeginPeriod / timeEndPeriod, the core time shows up in taskmanager increasing as expected.terça-feira, 26 de julho de 2011 05:31
So does the issue remain?
It still doesn't explain why after running profiler for the first time, the same application perf increases with profiler attached.
Thanks & Regards,
Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.quinta-feira, 4 de agosto de 2011 05:39