locked
How to Improve ASP.Net Web Service performance RRS feed

  • Question

  • User1139447206 posted

    We have one old ASP.Net asmx webservice in our application which receives bulk requests at sometime. Service is taking less than 5 seconds for a single request. But It is taking more than a minute when it receives 20 or more concurrent requests. Following is the way it is implemented,

    1)receives a request with input data from external clients

    2)Will get 20 possibilities from database for one request based on input data after validation

    3)Then It will iterate all 20 possibilities using foreach and gets solutions either from other external service or data base based on possibility data.  Here in old implementation we have used Parallel.Foreach to perform all 20 calls (service calls or DB calls) parallely to improve the performance.

    4)After that Service will send back the all 20 solutions to the client.

    This old approach is working fine for few (1or 2 ) requests and resonse time of asmx service is very fast(less than 5 seconds)  considering external service calls which are taking 2-3 seconds .But This approach is taking more than 60 seconds when the number of concurrent requests are more  than 20.Concurrent requests are pushing CPU utilization to 100% and thread pool starvation as per experts analysis and there by causing requests to queue for threads allocation.

    So we got a recommendation to replace parallel extensions and complete service with async/await implementation from end to end.I have implemented async/await end to end and also replaced Parallel.foreach with Task.WhenAll in TPL. But response time has increased a lot after this implementation.for a single request 20 secconds and it its taking more than 2 minutes for bulk requests.

    I also tried async foreach in place of parallel.foreach as mentioned in below article  but still performance is really bad.

    https://stackoverflow.com/questions/14673728/run-async-method-8-times-in-parallel/14674239#14674239 

    As per logs basic issue is with external service calls/DB calls inside foreach in both old parallel or new async/await implementations.But these service responses are very fast for a single request. Async implementation is taking more time in completing service calls than parallel extensions implementation.

    I think service should not take more than 20 seconds for bulk request if it is lessa than  5 seconds for single request.

    Can anyone please me what should be the way forward here to improve the performance ?

    Thanks in advance.

    Regards,

    Raghu.

    Tuesday, November 10, 2020 7:13 AM

All replies

  • User303363814 posted

    If the CPU utilisation is 100% then it doesn't matter how you rearrange tasks or threads you just don't have enough CPU for the job.  Tinkering with threading and async stuff can only rearrange the work to be done so that it happens at different times.  It does not change the amount of work to be done.

    The options you have available are

    1) Find a better algorithm

    2) Offload processing - can clients do some of the work before the request comes in - can the services you depend on do some more work before returning - can you improve the external services

    3) Get a bigger CPU

    4) Distribute the work between servers

    5) Cache work already done - useful if all or parts of multiple requests are the same.

    6) Make sure database queries are optimised so that work that could be done by the DB server is not being done by the compute server.  Are all your indexes optimised?

    Other people will have other ideas but the bottom line is that if your CPU is 100% busy then async/task/process management cannot possibly help.

    Tuesday, November 10, 2020 9:32 PM
  • User1139447206 posted


    @PaulTheSmith

    Thank you so much for the response. We are using dual Core CPU with 12GB RAM and we think that should be sufficient.

    As per my understanding and based on logs data CPU utilization is reaching to 100% due to following reasons,

    1)we are making 5 external service calls and one DB stored call call(total 7 IO calls) in each iteration of foreach loop. Then we are trying to make 20*7 =140 IO calls to external services and DB since  we have total 20 iterations. So most of the CPU is being utilized for few requests itself.

    2)So CPU is fully being utilized for  requests and then it is taking lot of time to process when the number of incoming concurrent requests are more. And it is happening since all IO threads are blocked out by few requests.

    Stored proc we are using inside each iteration is a simple one and its response time is very fast when it is not under load. Even the external services response is not more than 2 seconds when they are not under load.

    So can you please tell me what could be the effective way of making concurrent  external service/DB calls with out blocking IO threads or CPU utilization ?

    Please let me know if you need any further details.

    Regards,

    Raghu.

    Wednesday, November 11, 2020 9:30 AM
  • User303363814 posted

    Unless you actually measure you do not know where the CPU is being used.  Performing I/O is a low CPU activity but you are reporting 100% CPU usage.  The trivial number of I/O operations you have will not cause this.

    If threads are blocked because they are waiting for I/O to complete then you will see LOW CPU usage.

    Statement of the obvious --> High CPU utilization means you are using a lot of CPU.

    My previous post gave you six things to think about to reduce spread/reduce the CPU load.  Which one do you think will be the best for you to investigate?  Collecting some statistics will guide you in you selection.

    (It seems that you have decided that the I/O is the cause of your problem.  The main enemy to investigating performance problems is assumptions.  You need to measure, measure, measure eg https://stackoverflow.com/questions/3927/what-are-some-good-net-profilers)

    Wednesday, November 11, 2020 10:37 PM
  • User1139447206 posted


    @PaulTheSmith

    Thank you for the valuable information .Yes, It's true that I am assuming  I/O is the problem for higher CPU utilization. I assumed like that since there is not much business logic in my service apart from reading the data from external services/DB and responding it to client. Most of the logic is  either in DB procedures or in external services.

    I do not have much idea about how CPU or IO  threads work in the background. I will definitely try using profilers to understand the root cause .

    Thank you again :)

    Thursday, November 12, 2020 5:04 AM
  • User1139447206 posted

    I have done profiling for my service and it is showing me 80% of CPU is utilized for external  I/O Calls for single request itself. There are 5 I/O calls to external services. So it looks like slow  I/O calls are pushing CPU utilization ?

    Friday, November 13, 2020 10:38 AM
  • User303363814 posted

    Which exact lines of code or methods are consuming the cpu?  You have to drill down.

    Friday, November 13, 2020 9:30 PM
  • User475983607 posted

    RaghuMasineni

    I have done profiling for my service and it is showing me 80% of CPU is utilized for external  I/O Calls for single request itself. There are 5 I/O calls to external services. So it looks like slow  I/O calls are pushing CPU utilization ?

    The CPU does not make IO calls.  Specialized hardware handle IO tasks details.  The CPU orchestrates the tasks and the hardware just lets the CPU know when a task is complete.  

    Consider a browser can easily make 5 HTTP requests every time a link is clicked.  Your CPU usage does not go 80%.  As a matter, of fact you probably don't see a blip in CPU usage while surfing the Internet. 

    I'm guessing your code is not as efficient as it should be. 

    Friday, November 13, 2020 10:35 PM
  • User1139447206 posted


    @PaulTheSmith -

    After running profiler, I could few specific areas in service itself  which are performing badly  and modified them. 

    Also I observed two stored procs are taking lot of time under load for some concurrent requests. I am trying to move the business

    logic from DB to service  and reduce the number of I/O calls to DB by loading all data once to the service so that It can be processed

    for all requests instead of going to DB multiple times. 

    Is there a way to manage service level data in asp.net asmx web service  to use for all requests ? (my service is implemented with async /await)

    Thank you so much for the response.

    Wednesday, November 18, 2020 5:53 AM
  • User1139447206 posted


    @mgebhard -

    Thank you for the response, It's true that service is not written efficiently and it is very old implementation which we are supposed to improve.

    But I could see clearly 80% CPU utilization is due to I/O calls to external services or DB  based on Performance profiler results in VS 2019. Please correct me if I am wrong here.

    Under load it's even more utilizing CPU and could see this is again with bad response time .So I am trying to reduce number of I/O calls and improve DB stored procs business logic .

    Thank you so much

    Wednesday, November 18, 2020 6:00 AM
  • User475983607 posted

    RaghuMasineni

    But I could see clearly 80% CPU utilization is due to I/O calls to external services or DB  based on Performance profiler results in VS 2019. Please correct me if I am wrong here.

    I'm not sure what you're asking...

    You already told us the CPU jumps to 80% when making I/O calls.  I thought excessive CPU Utilization is the main problem you are trying to solve.  A web server should be able to handle 100s of I/O requests per second.  You also told us the current code base is having a hard time with 5 I/O requests. 

    RaghuMasineni

    (my service is implemented with async /await)

    ASMX services were created well before the async/await pattern and it uses APM style with begin and end request.  I don't think you can slap async/await on ASMX.  How did you do that?  

    https://docs.microsoft.com/en-us/previous-versions/dotnet/articles/aa480516(v=msdn.10)?redirectedfrom=MSDN

    I'm guessing part of the problem is your asynchronous design.   All your clients converted to asynchronous code as well?  And your internal logical is all asynchronous?  Every DB call?  Or did you wrap synchronous code within asynchronous Task.Run?  

    Anyway, it is pointless to go back and forth as we cannot see your code.  Share the code if you want a community code review.  If sharing the code is not possible, look into hiring someone that can help you. 

    Wednesday, November 18, 2020 1:11 PM
  • User753101303 posted

    Hi,

    To be on the safe side, a single client does multiple requests in close succession or each of those 20 requests are each coming from a distinct client? In the former case being able to submit a list rather than doing a request for each and every item could be a quick win especially if most of the 3rd party services are allowing that as well.

    Working on a simplified proof of concept could also help perhaps to see clearer or experiment quicker. For example something likel quueying multiple weather services as fast as I can as the main concern seems to optimize parallel requests to external services.

    Also with ASMX it could be best to provide a "best effort" (ie the simplest thing provide a good enough benefit) and then to consider upgrading to a more modern approach (possibly doing a POC to see how it is supposed to behave then).

    Wednesday, November 18, 2020 4:58 PM
  • User1139447206 posted


    @mgebhard

    Yes, I have wrapped Async/await(TAP) in ASMX service with APM style and at client side as well did the same(APM to TAP) to consume the service. I have made everything asynchronous including DB calls/Service calls with async/await except for a 3rd party library which is making few DB calls. we have known performance issues with service Business logic which is using this 3rd party library .I did not make any 3rd party library calls asynchronous as they are not supporting it. So we are trying to improve the rest of the functionality 

    It's true we are trying to solve excell cpu utilization and our understanding is that this is happening due to inappropriate use of I/O calls to DB/external services like using parallel extensions. Also due to bad logic in DB stored procedures which are taking more time to perform under load blocking I/O threads  and inturn causing excessive CPU utilization.

    I will be sharing the sample code.

    Thank you so much for you patience and bearing with my queries.

    Regards,

    Raghu.

    Friday, November 20, 2020 5:10 AM
  • User1139447206 posted

    @PatriceSc

    Thank you for the response, In our case issue is mostly when single client making multiple requests. our ASMX service can accept list of requests but external services don't have such option.

    We will definitely work on a POC as you suggested.

    Friday, November 20, 2020 5:13 AM
  • User1139447206 posted

    Hi All,

    This is how I wrapped asmx web method with async/await and APM,

    1)Begin method


    [WebMethod(Description = "Begin GetDetailsById Method")]
     public IAsyncResult BeginGetDetailsById(Request5 FeasRequest, AsyncCallback callback, object state)
        {
            var tcs = new TaskCompletionSource<Result>();
            var task = GetFeasibilityDetailsByIdv5Async(FeasRequest);
            task.ContinueWith(t =>
            {
                if (t.IsFaulted)
                    tcs.TrySetException(t.Exception.InnerExceptions);
                else if (t.IsCanceled)
                    tcs.TrySetCanceled();
                else
                    tcs.TrySetResult(t.Result);

                if (callback != null)
                    callback(tcs.Task);
            });

            return tcs.Task;
        }

    2)End Method   

    [WebMethod(Description = "End GetDetailsById Method")]
        public Result EndGetDetailsById(IAsyncResult result)
        {
            return ((Task<Result>)result).GetAwaiter().GetResult();
        }

    3)Actual Web method which I implemented with async/await,

    [WebMethod(Description = "New Async Method")]
        public async Task<CW.WebServices.Mozart.FeasibilityManager.Result> GetDetailsByIdAsync(Request5 feasRequest)
        {
          //business losic
        }

    Please let me know if this is a valid implementation or not ?

    Wednesday, November 25, 2020 10:29 AM
  • User753101303 posted

    I never used this approach in ASMX. I wonder if the issue is not what you are doing for calling external services inside GetDetailsByIdAsync. Do you await each task one after the otheror do you create all tasks one after the one and then await them all? See for example https://docs.microsoft.com/en-us/aspnet/web-forms/overview/performance-and-caching/using-asynchronous-methods-in-aspnet-45#Parallel

    Also if not done already you could add a trace to maybe better undetstand where most of this time is spent:

    Edit; you are doing 20x20 requests then? I'm wondering about https://docs.microsoft.com/en-us/dotnet/api/system.net.servicepointmanager.defaultconnectionlimit?view=net-5.0#System_Net_ServicePointManager_DefaultConnectionLimit

    What if you yry 30 instead in case the problem would be that you have 20 requests to your service causing in turns 20 connection to each external service and exceeding maybe this connection default limit?

    Wednesday, November 25, 2020 12:15 PM