locked
multithreading RRS feed

  • Question

  • Hi,

    I have to read a group of records from a database and do some action based on the data. for example, if I get 100 records, first record will call webservice-1 and update the status on table-A, second record will call another webservice-2 and update table-A, third record will call a method of the same assembly and update table-A and it goes on. What is the best approach to do this?

    I am planning to split these actions into multithreads so that i can get everything done at the same time. Is there any drawbacks on this? more over, exceptions of one single thread should not cause others or parent thread to stop. How do I go ahead??

    Thanks
    San
    Friday, April 24, 2009 9:00 PM

All replies

  • Specifics for this may depend on what language or frameworks your using.... but in general, I would recommend creating or using some form of Thread Pool to split up your tasks for each record.  This will allow you to do your processing efficiently.

    As for the exception handling - just make sure that your "task" has proper exception handling, so an error on one record doesn't tear down the whole process.  I'd also recommend having a clear sense of reporting of the problem - either a good logging framework, or some other means to track when there was an error, and what record it occurred on.


    Reed Copsey, Jr. - http://reedcopsey.com
    Friday, April 24, 2009 9:51 PM
  • Your best off using a threadpool for this sort of thing.  You probably don't want 100 threads executing at once (really depends on the hardware), because if you get too much concurrency going you start losing overall perforamance.

    http://msdn.microsoft.com/en-us/library/system.threading.threadpool.aspx

    for the exceptions, this is not a problem as long as you try/catch the work inside the thread.  If one throws an error then just log the error (or notify the parent) and move on.
    Monday, April 27, 2009 1:43 AM
  • Regardless of the specifics of threaded programming, you have to be really careful about the archtecture of your programming in terms of thread safety. You can never depend on the performance of thread creation and execution. If the tables or the data in the tables are dependent on each other in any way, you want to make sure that you don't thread batch jobs concurrently since starting a second job half way through the first one might cause nesting errors in the table data. Depending on how your table architecture works, you might want to use a combination of threading and custom event handling where you create your own events to fire at the end of a job or even during as well.

    As stated in another reply, you don't want to go nuts on creating lots of concurrent threads. There is a convergance point where the rate of thread creation starts to cause overall performance degredation due to overhead. Also, when compling as debug and executing your application using the Visual Studio Host shell, the tracer introduces a rediculous amount of extra overhead every time you create a new thread so it can track it. So don't base your final implementaion around debug testing. Create a stable build then performance test it by building Release and running it outside of vshost.
    Wednesday, April 29, 2009 3:24 AM