Best way for background task processing? RRS feed

  • Question

  • I'm designing a windows service that basically acts as a task scheduler to run certain processes in my application at configurable intervals.  There may be data importing, email notifications, batch reporting, etc. that all needs to happen at certain times regardless of what else is happening at the time.  In most cases, the application is not going to do anything in response to these tasks, they should just fire off, do their work and log what happens.

    Any opinions on the best method of achieving this?  At a very high level the obvious is that I will have a timer going off every n seconds and determining whether any jobs should be getting kicked off this cycle and then spawning a thread? for each job?

    Should I be looking at anything specific to do this?  Thread pools?  Just spawning a new thread each time a job is due to run?  Some other method?

    Monday, May 31, 2010 9:58 PM

All replies

  • You've considered SQL Server jobs + SSIS + SSRS?   It's an obvious candidate for your described requirements.

    With batch processing and services you're talking sleep rather than timer.  Timers sit there chomping on resources.

    How many people can schedule how many processes that take how long to run?  So they operate on the same resources and is the order they are requested important?

    Would be my first set of questions.

    How often are all these things going to collide and how critical is it that your process must run at that exact time?  If "the system" isn't doing anything in response to these actions then the timing can't be very critical.  So maybe you can avoid any multi threading - which can be tricky.


    Tuesday, June 1, 2010 9:18 AM
  • Our application can be configured to run against SQL Server or Oracle backend so the solution must be able to work under both scenarios.

    Initially we will have processes for:

    • System event processing (happens very frequently (every 5 seconds?) and feeds other services/tasks - sort of an enterprise service bus)
    • importing of data from various sources at a configurable interval that is not real time but likely will be every 10 minutes roughly
    • sending notifications of system events every 2 minutes (again can be configured by customer)
    • some dynamically scheduled jobs that can affect data in the system or again spawn email notifications based on some arbitrary criteria.

    What I meant by the system not doing anything in response to these requests is that the user will not be expecting a returning message from the system.  This all should happen in the background and be a fire-and-forget scenario.  If errors happen during one of the tasks, the task should handle and report to the log.

    I wouldn't think a single timer that just checks to see if any jobs are scheduled to run would be a resource hog?  Is that some better way to have these tasks spawn off at the correct time without a timer controlling things?

    The end result is that I will have arbitrarily scheduled tasks that should kick off at approximately the correct time.  The exact time is not critical, but emails should be sent out approximately every 2 minutes....not every 2 minutes for the first half hour and then have to wait for 15 minutes for some other longer running process.  The current tasks that we have to handle are not going to be very long running, most probably less than a minute in duration.  Future jobs could be longer running, reports that must calculate a bunch of data, mass import/export of data, etc.

    Thanks for the response, any further recommendation?  This is a commercial app that will be installed on site for a client.

    Tuesday, June 1, 2010 12:23 PM
  • In a similar vein, you can use the Windows Scheduler to invoke your service. That way all the calendar work is implemented for you and your service can look/change whatever data you need. The key is to avoid letting the scheduler look at the data as it makes scaling out complicated.
    Wednesday, June 2, 2010 6:21 AM
  • Hi dbstrat,

    Your problem can be divided into two parts here-

    1. Whether I should use Windows Service or Windows scheuler to achieve the task.

    2. What mechanism I should follow to achieve Fire-and-forget scenario as required by you since there are 4 subtasks in your service (listed above in your thread) each is mutually exclusive to other.

    1. Whether I should use Windows Service or Windows scheuler to achieve the task- Since your service does not work on a regularly basis you can always go for Windows scheduler as suggested by pkr2000 ensuring not utilizing system resources in idle state. In windows scheduler you have flexibility to configure your service at the time period you want. But again this won't be a background process unlike of a windows service. If you feel that your processing period is very frequent(say after every 5-10 seconds), then using windows service is also a not bad option.

    2. What mechanism I should follow to achieve Fire-and-forget scenario as required by you since there are 4 subtasks in your service (listed above in your thread) each is mutually exclusive to other - Actually being a C# developer instead of going into multithreading to spawn 4 threads to perform 4 subtasks you can prefer delegates. Delegates covers lots of abstarctness compared to Threads but can be used beutifully to give asynchronous calls(fire-and-forget) as per your requirements. I suggest you to go for delegates to achive fire-and-forget scenario.

    Summarizing you can schedule your master service/process (which has inline 4 subtasks) with windows scheduler/service (based on the frequency) and inside your master service code you can use delegates to invoke each sub task independently and always you can maintain separate logging for each subtask useful in case of any crash.

    Hope this helps.

    Wednesday, June 2, 2010 7:17 AM
  • I would expect some front end that communicated with a service or group of services via message queue or database. 

    You didn't answer whether the order or interaction matters, so I'll assume mostly not.

    I would suggest a service per function.  Loading data is one service.   Sending emails is another service.  Etc. Thus code is split up into manageable chunks and you can add other ones one painlessly.  Perhaps avoid multithreading.  If you need multithreading then at least all the functionality isn't in one hugely complex service.

    I'd stick some (free) compact database on your server these services run on.

    Time and description or pointers to job data would be added to these.  If one task spawns another then it adds an entry.  Each task marks the entry as started and eventually finished.  You then have some front end which adds entries to this.

    Each service starts up, reads the database for it's entries and decides if it has any to process.  If it finds one it goes and does the task. Then it sleeps for some used configurable time before repeating. 

    Wednesday, June 2, 2010 11:34 AM
  • Thanks for the tips folks.

    The window's scheduler is not really an option for me as users will be using the application to create custom tasks via the UI, not the Windows Task scheduler.  Again this is a commercial application that will be used by non-technical users.  Everything must be kept self-contained within the application and run off of data in our database backend which is installed onsite at the client.

    Ideally I could poll for any task that is scheduled to run at this moment and begin running that task (which may take 1 second or 1 hour, theoretically).  These tasks would go about in the background importing data, sending emails, calculating reports.  They are not dependent on one another but they should run as close to the scheduled time as possible.  I don't want to have emails waiting to go out for 30 minutes because some other job is taking a long time...  At any given time a new job may be added and needs to be handled by the scheduler.

    Being a multi-threading newbie I feel like I should just be able to spawn a new thread whenever the polling timer finds a task that needs to run.  They won't be reporting back to anything but the database or a log file and the UI is not dependent on them.  There will be no interacting between the threads, they should be completely independent.

    Yes these could be independent services, but then the customer would have to update a growing number of service applications upon new releases instead of just updating one master service application that controls these jobs so I'd like to avoid going down that road if possible.

    Wednesday, June 2, 2010 3:34 PM
  • Why would the customer care how many services your installer updates?

    Unless they poke round in the machine they're not even going to know what services are there.

    Have you ever looked at how many services are running on your machine?


    Wednesday, June 2, 2010 7:23 PM
  • Why would the customer care how many services your installer updates?

    Unless they poke round in the machine they're not even going to know what services are there.

    Have you ever looked at how many services are running on your machine?


    Is there a great advantage to breaking this out into separate assemblies vs. new Thread(new ThreadStart(TestTasks.TaskA)).Start() when a task is scheduled to run? 

    Without going into the hairy details of our dysfunctional company, lets assume that there will not be time to put together a nice slick custom installer that can bundle together multiple services...

    Wednesday, June 2, 2010 8:40 PM
  • Breaking functionality into smaller chunks will make it way easier to develop, maintain and extend.  The operating system gets to worry about your threading and stuff that will contend with itself is hopefully running one at a time in it's own service and not contending with the other services.

    It is much harder to write multi threaded apps and debugging is a nightmare.  It's all too easy to create a problem that only occurs occasionally due to some contention with multiple threads.

    Of course if you have an office full of multi-threaded service coding-gods then that might not be an issue.  For mere mortals such as myself, writing an install script is the easy bit.

    Thursday, June 3, 2010 8:49 AM
  • Check on codeplex there you can find batch server based on your requirment where you can schedule a job and can run any code. It would be easy for you to look that code and there after decide how to process.


    In my opinion it's bad idea to re-create entire wheel again instead of resuing that. So please check various option and make decision. If you want you can do that but in that case fullproof application require lot of tesing along with development. If you want to go with that way I can share you my Idea.


    Thursday, June 3, 2010 10:38 AM
  • Thanks Mayank.  Do you have a link?  I have searched for such a project and am not finding it...
    Thursday, June 3, 2010 12:03 PM
  • Hi,

    I had gone thorugh majoirty of your replies here, it seems you want some handy mechanism that runs behind the screens without much of user intervention. for this you already had a approach of window service along with timer to trigger-on and off particular tasks. This approach is good one. But for the sake of other options i would like to list out few:

    • As some one previously pointed out as alternative to win service we can build plane simple .exe applications and schedule it under scheduler tasks of OS
    • Other option could be design a complete framework which compresses of MSMQ kind of mechanism which takes care of importing data, sending emails, calculating reports. Now in this case you can encapsulate this MSMQ stuff in single window service or more.

    for more information on MSMQ please check the link of mine http://patilmanishrao.wordpress.com/2010/03/09/wcf-and-msmq-technical-links-on-msdn/. I'm quite confident if you use this (MSMQ approach) then you will not need to worry about threading and pool mgmt stuff.

    Let us know your feedback on this

    Manish Patil http://patilmanishrao.wordpress.com Posting is provided AS IS with no warranties, and confers no rights.
    Friday, June 4, 2010 9:02 AM
  • Checkout this one


    There are some paid softwares are also available.


    Monday, September 2, 2013 3:05 PM