locked
Performance issue become worse upon split out Web Applications into web server and DB server RRS feed

  • Question

  • Hi,

    Currenly i'm faced web applications (in web server, 2GB RAM) with performance issues upon seperated database(4GB RAM) and web application into 2 machines. Before spliting, it have performance issue where IIS need to be reset when "Server is un-available" or hanging occur. In this case I proposed to split out for better performance and hope to resolve the issue. Now, after spliting, users complaint that the responce time even worst that before. Any one have this problem before? Any suggestions to help?

    * DB server with SQL Server 2005 and SQL Server 2000 installed but i only used SQL Server 2000 and both web servers with Window Server 2003 installed together with crystal report 10.

    Thank you..

     

     

    Friday, January 26, 2007 11:41 AM

All replies

  • Obviously the problem is not on memory, since you have a lot. There can be several issue involved.

    One may be that the problem was not actually in the usage of resources from SQL Server 2005 on your server, but in the actual application. And I really think this may be the problem. I don;t really know much of your application, but let's see this scenario:

    Another thing that could be happening (and I really doubt it) is that you have network problems and communication between your application and SQL Server 2005 is taking too long.

    Let's say you have really big queries that are constructed and sent to the database, the database then executes those queries and returns the information gathered. Here there maybe bottlenecks in : 1) The query construction, 2) the query communication, 3) the query execution or 4) the query results communication. Now you use to have everything in the same machine so if the case I just describe fits in yours, there is no communication problem between sending the query and receiving the results because they are on the same machine. But if you split them, then you have a communication problem, because it also takes longer to send and receive information.

    But this is not the source of the problem, the problem is the query building.

    Of course I am assuming too much, but I am trying rationalize the problem. It would be very useful if you could provide more information.

    The first thing you need to do is identify where the problem is, either if it is in the application, in the database or in the communication between them.

    To check bottlenecks in the application I recommend using a Profiler for your application, I use Ants and it really helps a lot. In the case of the database and communications you can use the tools that SQL Server 2005 has, like the Profiler.

    I hope this helps.

    Friday, January 26, 2007 8:39 PM
  • The problem that you're describing is something of a common one. Putting your DBMS on a separate machine while likely improving the throughput of your system adversely impacted latency. In short, you can probably handle more concurrent users but response time has suffered.

    The (again) common solution to the problem of response time is more agressive caching on your web server. To truly wring the best performance out of a given software/hardware/network combination you first need to measure exactly where the time goes. Caching solves problems that have the network as their source. DB partitioning solves contention problems also impacting response time. There is quite a bit of depth here, beyond what can be handled in a simple post.

    Friday, January 26, 2007 10:29 PM
  • You need to get to the bottom of the "server unavailable" problem.  Is this an ASP.NET application?  How does IIS invoke your code?

    If the perceived performance got worse once you put the network between the IIS and SQL machines, I would first look at how many SQL calls are actually getting sent.  If there are LOTS of SQL calls involved in the application, putting the LAN between the machines will certainly increase response time.  Use perfmon on the SQL machine to find the number of "batch requests per second".  If the value is consistently above (I'm not the expert) 3000, your app (or your ORM mechanism) is simply making too many SQL calls.  You can mitigate the issue a little by having your IIS talk to your SQL machine on a dedicated NIC. 

    If the batch request per second number looks fine, then I agree with aespinoza that either (a) your queries are taking too long for SQL to execute or (b) the result-sets are too large for the apps code or the client/IIS network link to process efficiently.

    Hope this helps a little.
    Friday, January 26, 2007 10:33 PM
  • To add to all of the above good advice, remember that a call across a physical network (seperate app server from db server) will always be slower than a call in memory (app + db server on the same machine).

    You can also tune the .Net machine.config file for physcial server configuration, check out:

    http://msdn2.microsoft.com/en-us/library/ms998583.aspx

    http://msdn2.microsoft.com/en-us/library/ms998530.aspx

    Improving .NET Application Performance and Scalability:

    http://download.microsoft.com/download/a/7/e/a7ea6fd9-2f56-439e-a8de-024c968f26d1/ScaleNet.pdf

    HTH

    Ollie Riches

    Sunday, January 28, 2007 7:56 PM
  • You have mentioned a IIS hang in the context of a performance issue.  In my experience IIS hanging is usually an application fault.  Aside of the new issue you have by introducing a network connection between the Web server and DB server, I would also be looking into the reasons for IIS hanging, there are lots of good tools around to help you with this, understanding their output can sometimes be tricky though.  My experience is limited to classic ASP and VB6, but I would a image the same principles apply.

    ADplus is one tool - http://support.microsoft.com/kb/286350

    IIS Crash/Hang - http://www.microsoft.com/downloads/details.aspx?FamilyID=01c4f89d-cc68-42ba-98d2-0c580437efcf&DisplayLang=en 

    Regards

    Si

     

     

    Monday, January 29, 2007 8:26 AM
  • Hi All,

    Thank you very much for the advises and suggestions. It give me the ideas to start for invetigation and to find out root of the problem.  

    Currently the web applications written by ASP.NET 2003 (vb.net) and it have seperated into UI(web app) and BL(web services) and it running in LAN. DB connection string stored in web.config and using OLEDB method. 

    The web applications faced "Services in un-available" and it hang the IIS and required to perform IIS reset previously. We did seperated web applications into more application pools and changes on recycle the worker process to solving the issue. It seem work fine for these few days and but need further monitoring as well.  

    regards,

    csleaw

    Monday, January 29, 2007 9:56 AM
  • Hi Folks,

    This seems like one of the issues I had in my previous deployment. We had a smart client talking to the web-services on a Win 2003 server. We got the service-unavailable errors multiple times due to different reasons. Hereby, I am stating two of the cases, which might be relevant to your case.

    Case-1

    Description: When there are too many open threads, the server reaches its max-thread-limit and starts giving this errors. This can happen due to improper handling of caching. When we create a new CachingManager, it gets created in a new thread. If it is not disposed quickly, it stays as an open thread.

    Symptom: Open the task manager and look for the thread-count and handle-count of the IIS process (ASPNET/ NetworkServices/ W3P). If the threads are going up consistenty and are high before service crashing then it is an issue. Usually the thread-count stays at 10-20. If it has increased beyond 50-60, it is a problem.

    Solution: Please investigate, which statement of code is creating this new thread by debugging through your applicaiton. Take preventive measure. E.g. if it is a CachingManager causing this issue, you might decide not to create new CachingManager objects. Rather, have just one CachingManager object for the entire applicaiton and reuse it in all sessions.

    Case-2

    If you are using 3rd party authentication/authorization software (e.g. Tivoli access management), they often work by hizecking the web request, performing the authentication/authorization activity and then returning the control to the CLR. This process often creates the thread of the 3rd party software. these threads might accumulate to case service-unavailable errors.

    Symptom: Same as above

    Solution: Try unning your solution without authentication/authorization. If it is not possible, try hard-coding an identity and run the application. If the problem does not reoccur, the issue might be with the 3rd party software (or the way you are interacting with the software). Consult with your vendor.

     Regards,

    Anshu

    Monday, January 29, 2007 8:48 PM