none
How to implement background service in azure web app. RRS feed

  • Question

  • I am using Azure app service, and I have few web app is running as part of my App service plan. 

    Architecture of my web app:

    Web app exposes few endpoints, to where user can send requests to webapp, then web app will contact few other system and DB to fulfil the request.
    it is a kind of integration system for our Organizations. If we have less number of requests to our web app ( like 1000-2000) in an hour, our web app is able to process the requests. In some case we got a huge of events to process like(40000-80000) per hour. In this case most of the events are failing with below error.

    2016-10-06 10:09:12 cloudmix-sharepoint-production POST integration/Services/eventprocessor.svc - 0 - - - - - cloudmix-sharepoint-production 500 121 0 548 6307 230250 

      

    As per my understanding( please correct me if I am wrong), this error is coming because our web app is not able to process all the requests within 230 seconds. any request which are in web servers's Http Queue for more than 230 second, will be terminated. So that request will not be processed.

    our Web app takes approximately 2 seconds to process one request ( As it need to do few Rest calls to external systems). 

    To solve this issue we have tried to scale up to more number of instances(10). It solves the problem temporarily, but the problem of loosing Event/request comes back again when we have a high load  events.

    Also any  event from web app client  should be processed within 15-20 second of time.

    What could be the ideal solution for this. How to make our web app to handle huge number of events in a short time span ?
    Does web job helps here ? if yes what should be strategy?
    Is there any other Azure component in Azure app service which helps me offload the work to some background services ?

    Any help will be highly appreciated.
    Friday, October 7, 2016 12:15 PM

All replies

  • Hi,

    Thank you for posting on the azure forums!

    First I would suggest you to enable the diagnostic logging for your Web App incase if you haven’t enabled earlier to check the request/response log traces to identify the root cause.

    You may enable diagnostic logs following the steps given in the link.

    Also, check the logs browsing through the Kudu console. Check if there are any request/response logs. Please refer the link along with Investigating issues for more information on this.

    If scaling up instances for Web App could solve the problem, you may try enabling Auto Scale option available on the portal. You could also enable “Always On” option to your Web App just in case if your Web App is being idle for some time and it could cause issues for handling requests.

    Regarding your specific query:

    You can run programs or scripts in WebJobs in your App Service web app in three ways: on demand, continuously, or on a schedule. Refer the article Azure Web Apps and WebJobs for more details.

    You may want to refer the Azure Web Apps Best Practices, Scaling Web App to handle more queries/requests and blog post about different methods to follow.

     

    Hope this helps.

     

    Regards,

    Ashok

     

    ___________________________________________________________________

    When you see answers and helpful posts, please click Vote As Helpful, Propose As Answer, and/or Mark As Answer so that other customers can benefit from it.

    Saturday, October 8, 2016 5:48 AM
    Moderator
  • Hi Ashok,

    Thanks a lot.

    I had enabled diagnostic logs, and the log statement posted  posted in my question before was from my Azure web app log.

    Only After log analysis, I found that my request processing is taking 4 second per request. And if  get 40000 request in an hour then lot of request is throwing timeout error.

    What I want to know is

    1. Is there a way to increase the timeout, so that all requests are queued up for processing, no request will time out ?

    2. What pattern/component  Microsoft suggests to offload request processing to background service in an web app,   What components are available in Azure to help in this kind of scenario.

    3. Scaling up is not always an option. We have a limit, upto how many instances we will scale. We need to solve this kind of problem from application design prospective. in Azure cloud services there were options like webrole, and worker role. worker Role does the work in a background service. What is the corresponding component for Azure web app? is it webjob?

    4. Is webjob a correct choice to process 40000-80000 request per hour ? If i create a continuous webjob having a function bound to a service bus queue, and the trigger the function for each request. is it a Scalable approach?

    Thanks, 
    Saturday, October 8, 2016 8:21 AM
  • A continuous WebJob is analogous to a Worker Role, so that should be a good option for background processing.

    Anothing thing you could looking into using is the new Azure Function (still in Preview), which can do background processing with a pay-for-what-you-use model.

    David

    Wednesday, October 12, 2016 3:28 PM
    Moderator
  • Hi David,

    Thanks a lot. I love your articles and technical solution on github, MS forum and other sites.

      I am thinking of this architecture. Every event will be pushed to a service bus queue. one webjob will be waiting for that servicebus queue trigger. for each event one webjob function will be executed, which will process that event. ( when I say process means , it needs 1-2 Http calls to external systems).

    Will limit the number of webjob function  instances to 100. So that 100 events can be processed at once.

    1. Is starting one function for each event is costly, Is there any startup latency for a webjob function. If I put my message to Queue, does the webjob starts  immediately or it takes few second.

    2. How many webjob functions can run in parallel in a Azure vm ( 4 core, 7gb ram) machine. (I mean what will be the ideal number, our event processing is IO bound not cpu bound).

    Thanks,

    Abhiram 

    Wednesday, October 12, 2016 6:05 PM
  • 1. Starting a function is not costly. Once 'warmed up', the WebJobs SDK will dequeue multiple items very quickly.

    2. The biggest constraints are CPU and memory. If it's just I/O bound and uses async, you should be able to run a very large number in parallel on the VM.

    Those statements are true whether you use WebJobs or Azure Functions.

    Wednesday, October 12, 2016 6:16 PM
    Moderator
  • Hi David,

    I didn't understand point 2.

    my web job function will be as below.

    //Get the event detail from external system.

    //normalize the message.

     // Push it to another system.

    How will I make it as Async. Please give some code example, it will be very helpful.

    Thanks,

    Wednesday, October 12, 2016 7:05 PM
  • See the async section in this post for an example. But it's pretty much standard C# async concepts.

    David

    Wednesday, October 12, 2016 9:40 PM
    Moderator
  • Hi David,

    If I have a web job having batch size 100. and my web app scale out to 10 instances. Can I assume that 1000 instance of webjob will be running in parallel.

    Is there a limit how many web job instance  can be spawned ? or What is maximum batch size of the web job.

    One more thing, Does webjob slows my web app ? From the day we deployed webjobs, we are observing slowness in our web app( it is processing less number of requests per minute). 

    Thanks,

    Abhiram

     


    • Edited by saishn Thursday, October 13, 2016 1:35 PM
    Thursday, October 13, 2016 1:26 PM
  • The batch size is limited. e.g. with Azure Storage Queues, the limit is I think 32. However, there is an additional NewBatchThreshold concept (some info here) which let you process more than that in parallel. So effectively, you can run higher numbers than the batch size.

    David

    Thursday, October 13, 2016 10:44 PM
    Moderator
  • Hi David,

    Thanks a lot for answering my queries, I have few more questions.

    1. If I have a web job having batch size 32. and my web app scale out to 10 instances. Can I assume that 320 instance of webjob will be running in parallel.

    2. In my web app, I am seeing that only 2 threads are processing the request. From the Application log, I am able to see that all the requests are being processed by 2 threads. Is there a way we can increase the thread count of the Azure web app, So that it can process more requests.

    Thanks again for all you help.

    Thanks, 

    Sunday, October 16, 2016 5:32 PM
    1. It could be more than that, depending on NewBatchThreshold. See link in my previous response for details.
    2. It shouldn't be limited to 2 thread. You might need to open an issue on https://github.com/Azure/azure-webjobs-sdk/issues with more details. But make sure you make your Functions async to avoid blocking threads unnecessarily (unless they're actually doing CPU intensive activity). If you're I/O bound and use async, even a single thread can allow a very large number of items to be processed concurrently.
    Sunday, October 16, 2016 5:41 PM
    Moderator
  • Hi David,

    Sorry for the confusion.

    Actually I am asking how to increase thread count for Azure web app ? What I am seeing is, my Web app is processing 2 request concurrently at a time. From my Azure web app log

    2016-10-17T16:46:24  PID[4764] Verbose     [22]  Processing Request for new event

    2016-10-17T16:46:24  PID[4764] Verbose     [33]  Processing Request for new event

    If I generate 1000 events also, it is processing 2 events at a time concurrently. For each event processing it takes nearly 1 second ( It just pushes the event to Service bus Queue, from where web job will pickup).

    With one instance if I give 1000 events to process then, with the above processing rate and above thread count, it will take 500 seconds to process these events(Just to push them to service bus queue).

    A lot of Events are failing due to timeout issue ( As Azure/IIS has this setting that "Any Http event will be terminated if it is not processed within 230 second." ), I got below error in Azure log.

    2016-10-06 10:09:12 cloudmix-sharepoint-production POST integration/Services/eventprocessor.svc - 0 - - - - - cloudmix-sharepoint-production 500 121 0 548 6307 230250

    What I need is to make my web app to process with a faster rate.

    How Can I increase thread count of my web app to higher number instead of just 2 ?

    followed below article but didn't seems to helping.

    http://www.codeproject.com/Articles/133738/Quick-Ways-to-Boost-Performance-and-Scalability-of

    How Can I overcome this Request time out issue ? 

    Even if I scale the Azure instances also, I am not able to handle the event brust ( 80000 events in an hour/ very  short span of time). Most of them are timing out .

    Thanks,


    Monday, October 17, 2016 5:06 PM
  • To reiterate my previous answer, I would not expect it to be limited to 2. If you do see this, please open an issue on GitHub as suggested.

    thanks,
    David

    Monday, October 17, 2016 9:28 PM
    Moderator
  • Thanks a lot David,

    Does Webjob scales automatically with web app or I need to code something specifically to scale it up.

                config.UseServiceBus(serviceBusConfig);
                config.Queues.BatchSize = 32;
                config.Queues.NewBatchThreshold = 10;
                serviceBusConfig.MessageOptions.AutoRenewTimeout = new System.TimeSpan(1, 0, 0);
    
                JobHost host = new JobHost(config);
                host.RunAndBlock();

    I am scaling up my web app to 10 instances, here my expectation is I will have around 320 web jobs running on all my 10 instances ( 32 per 1 instance).

    But I have a feel that it is not scaling up, because I am not seeing any improvement in my processing. If I have 40000 messages to process in my service bus queue, I am not finding any difference between the processing time to complete all the messages, whether I have scales up to 10 instance or just running with 1 instance.

    Thanks,

    Thursday, October 20, 2016 4:56 PM
  • Unless you declare a function as singleton, it should definitely scale to all instances and all instances should be picking up items from the queue.

    David

    Thursday, October 20, 2016 5:58 PM
    Moderator
  • Hi David,

    My function is not singleton, below is my function definition:

            public static void MessageHandler([ServiceBusTrigger(APPConstants.MessageQueue)] string message, TextWriter log)
            {
                log.WriteLine(message);
                new WebJobHelper(log).ProcessEvent(message);
            }

     Is there a way I can see how many web jobs are running concurrently ? or average  statistics of all web jobs in last few hour,   Azure webjob dashboard shows around 10 running functions, but I not able to track there properly.

    Let me run another test, will post how it goes .

    Thanks,

     
    Thursday, October 20, 2016 6:13 PM
  • Note that the JobHostConfiguration.Queues knobs that you configured above don't affect ServiceBus queues, they're only for Azure Queues, in case that wasn't clear to you.

    There are some performance knobs I suggest you try to increase your ServiceBus message throughput. Please see the notes on configuring ServiceBusConfiguration.PrefetchCount that I made on this PR. There are also some comments on disabling/reducing Dashboard logging for high throughput functions.

    ServiceBus Prefetching is described in their documentation here.


    Mathew Charles [MSFT]




    Thursday, October 20, 2016 10:37 PM