Best Practice method for executing long running, queued tasks from ASP.NET

Question

User-1963256800 posted

Hi,

We currently have a data access website that also runs long running calculations - these vary from 5 seconds for an individual calculation to half an hour for bulk calculations. The individual calculation runs synchronously, the longer ones are spun off onto separate asynchronous threads with no callback. Based on information stored in the database the user is given a progress update on each refresh of the page.

There are many problems with this. One is that we currently have no control over the number of bulk calculations that are set running simultaneously. While this is fine at the moment increased usage might cause a problem to the online users of the system. The other is that we have no ability to queue or schedule long running (and resource hungry) bulk calculations.

Also I recently had a problem where the whole application pool failed due to unhandled errors on the asychronous threads. And we have had problems with the worker processes ending on the idle timeout while the bulk calculations were still running.

I think it would probably be a very good idea to get these bulk processes out of the ASP.NET context, but I'm not completely sure on that.

Requirements:
Run single calculations (5s) from the web application and wait with an AJAX timer. Preferrable the calc should complete even if the user navigates away from the page or IIS recycles.
Run bulk calculations (up to 30 minutes), which use the same code as the single calculations, asynchronously from the web application. Allow a set number of simultaneous bulk processes and then queue any remaining bulk processes and run them as and when other bulk excecutions finish. These bulk runs should not care if IIS recycles, although they don't need to survive server reboots.
Another option would be to allow scheduling of bulk  processing for out of hours runs - in this case cancellation should be possible.

What do people recommend for this? I was thinking MSMQ with a Windows Service (possibly a bit out of date), but I have seen references to Workflow (WF) as well? This won't need to be load balanced or anything like that. I would like the simplest solution that will address the requirements and the current problems. I certainly don't mind writing some bespoke code if it is simpler to roll our own.

We are currently using .net 2.0, but would consider .net 3.5 solutions as well

Cheers,

James

 

Answer 1

User1011739529 posted

Hi,

These tutorials may help you:

http://www.aspfree.com/c/a/VB.NET/Executing-LongRunning-Tasks-with-the-Progress-Bar-in-ASPNET/

http://geekswithblogs.net/mnf/articles/asyncronous-long-running-tasks-in-asp.net-application.aspx

http://www.interworks.com/blogs/banderton/2009/10/30/long-running-tasks-aspnet

http://www.codeproject.com/KB/aspnet/asynctransactionhandler.aspx

 

Answer 2

User-952121411 posted

Start by looking into a few of these links; WF allows the ability to persist long running tasks (minutes, days, hours, weeks) to be persisted to an external medium, so that the process can be continued later.  This will help eliviate some of the issues you mentioned.

Patterns for Long Running Activities in Windows Workflow Foundation: 

http://blogs.msdn.com/pandrew/archive/2007/11/13/patterns-for-long-running-activities-in-windows-workflow-foundation.aspx

Execute a Long-Running Workflow Using Persistence in Windows WF:

http://www.devx.com/dotnet/Article/32247

 

Answer 3

User-1963256800 posted

Hi,

Thanks for the pointers. I am now well and truly out of my depth. The technologies that have been mentioned that look interesting are WCF and WF.

My impression of WCF is that it is SOA related and therefore web services, which is not what I am looking at. My impression of WF is that it is for multi stage tasks that have a long overall timespan (a series of quick individual actions with large amounts of idle time between them). Am I barking up the wrong tree?

I was thinking of using MSMQ to communicate with a .NET Service. Is this a bad idea and what are the advantages of WCF or WF over this?

I should point out that the long running calculation does lots of thing like write to database tables and intensive calculations, but it all happens in one go. It doesn't need workflow (in my understanding of workflow). They just need to be queued so we don't get loads running at the same time. And I am trying to get away from anything to do with ASP.NET or IIS (idle timeouts, crashing app pools etc), so web services would be silly too. The calculation code is all in VB.NET so it would be relatively easy to implement as a Windows service instead of the pseudo web service it is at the moment.

What do you think?

Cheers,

James

Answer 4

User-389939489 posted

we have had problems with the worker processes ending on the idle timeout

The idle timeout can be set to infinite. Fine-tuning your application pool settings might help mitigate your current problems.

For the rest, from the top of my head, and assuming you know in advance if a calculation is going to take short or long:

I would think of  a windows service spawning two sub-threads or "worker processes", to handle the two kinds of calculations. The two processes would pick items from two respective queues (MSMQ sounds ideal: it is not out of date, as far as I know), and the processes would write to some datastore any intermediate and final data, including information about progress status.

For long-running, asynchronous calculations, a web client might simply gather information from the datastore and show it in a web page. For the short-running calculations, IMO the "synchronicity" should be just virtual (otherwise you'd need to implement it at some lower level): e.g. performing xml-http-requests from a script is intrinsically asynchronous, but freezing the controls until you get a call back or a timeout is easy enough. Here you just need an AJAX-enabled web service in between. (Then you might thing to reuse the AJAX-enabled page for the long running processes too...)

Hope this gives you some ideas,

-LV

Answer 5

User-952121411 posted

I was thinking of using MSMQ to communicate with a .NET Service. Is this a bad idea and what are the advantages of WCF or WF over this?

 

No not at all; those are just some of the older .NET technologies, but there is no reason to avoid them.  I typically try to keep moving forward with the newer technology when possible, but there are so many factors in a project (time, money, etc.) that taking on (2) entirely new tools can be difficult to do.

If you thought about mixing the two together (i.e. WCF with MSMQ), you may be able to do it with the WCF MsmqIntegrationBinding:

MsmqIntegrationBinding Class:

<!---->

http://msdn.microsoft.com/en-us/library/system.servicemodel.msmqintegration.msmqintegrationbinding.aspx

 

How to: Exchange Messages with WCF Endpoints and Message Queuing Applications:

<!---->

http://msdn.microsoft.com/en-us/library/ms789008.aspx

 

 

 

 

Answer 6

User-1963256800 posted

Many thanks everyone. A lot to think about!

James

Answer 7

User-525215917 posted

While you are looking for solutions and trying to get the best one of them you may find articles that show how to run background processes on web server. DON'T DO IT! Web server is not controlled by web applications and web server knows nothing about your long running threads. When IIS feels it is needed, it can unload application that is not active. If you have long-running processes then keep them running under windows or WCF services.

Answer 8

User-389939489 posted

DigiMortal:> When IIS feels it is needed, it can unload application that is not active. 

That is just incorrect. Indeed, this is informatics, not magic: IIS has no feelings, please just RT*M.

-LV