locked
Design for multile http requests via wcf and/or threads RRS feed

  • Question

  • User348644168 posted

    I've designed a website that polls other websites and displays information about those sites with a link back to them.

    The design currently reads a table in a sql db that has a list of the sites to query and specific information needed to process the site's data. The program then reads through the list and requests each website sequentially until it is completed.

    Since this process is sometimes slow I would like to try improving speed by running each request in a seperate thread or maybe use a wcf service in order to process the requests simultaneously.

    I've been reading about threads and wcf but need some advice (and maybe some hand-holding) to get this design going.

    Can anyone give me an idea of what would be the "best-practice" design for this type of problem?

    Will this improve performance?

    Note that I am using ASP.NET 2.0 but my hosting supports up to 4.0 if upgrading will provide a benefit.

    My hosting also supports WCF and Silverlight although I would like to avoid having to convert everything to a new platform like Silverlight or Ajax.

    Thanks.

    Monday, January 24, 2011 11:04 AM

Answers

  • User2081202125 posted

    Here is a good example from MSDN: http://msdn.microsoft.com/en-us/library/aa645740(VS.71).aspx

    Look at example 2. It is a good example of how to work with threads. You would need to have a data processing method in a separate class and call the method from a loop in the main thread where you will be kicking off the threads. 

    For  handling the results, you need to have a collection object in the separate class and add the results to that collection. And finally you would have to wait until all the threads have completed and then access the results collection.

    • Marked as answer by Anonymous Thursday, October 7, 2021 12:00 AM
    Monday, January 24, 2011 4:55 PM

All replies

  • User2081202125 posted

    I am not someone who does design work but, here goes..

    I think there really isnt a need to use WCF, what you need can be achieved in a web app using c#. Threads would be the most straight forward way. Once you have all list of sites you need to process, you can spawn multiple threads which make requests to external sites for data and then process them.

    This will definitely improve performance over sequential execution, although may increase CPU time while it is running. While you can certainly use Silverlight and Ajax for the frontend to improve the usability of the web app that you develop, you can achieve your goal without them as well.

    Monday, January 24, 2011 3:14 PM
  • User348644168 posted

    I think you may be right about the WCF.

    I am just learning about Threads and have seen multiple postings about mutex and other strange things. Any suggestions on where I could get a good solid example of running multiple threads and handling the results?

    The page is currently in VB.net but I could probably translate C# examples, if you know of any.

    Thanks! 

    Monday, January 24, 2011 3:32 PM
  • User2081202125 posted

    Here is a good example from MSDN: http://msdn.microsoft.com/en-us/library/aa645740(VS.71).aspx

    Look at example 2. It is a good example of how to work with threads. You would need to have a data processing method in a separate class and call the method from a loop in the main thread where you will be kicking off the threads. 

    For  handling the results, you need to have a collection object in the separate class and add the results to that collection. And finally you would have to wait until all the threads have completed and then access the results collection.

    • Marked as answer by Anonymous Thursday, October 7, 2021 12:00 AM
    Monday, January 24, 2011 4:55 PM
  • User348644168 posted

    Thank you. This is very helpful. I'm sure I can work out the details based on this information. 

    Monday, January 24, 2011 5:22 PM