none
Deleted RRS feed

All replies

  • Hello,

    In code you provide almost impossible to figure out what you doing.

    For the task: implement TThreadManager and use it to manage thread's.



    Sincerely, Highly skilled coding monkey.

    Monday, January 28, 2019 3:04 PM
  • Deleted
    Monday, January 28, 2019 3:36 PM
  • You don't want to limit the # of threads your app can use. You want to limit the # of threads your scraper uses. For that you can build it into your scraper. You'll need a manager-like class that manages the requests for scraping. You would filter it there.

    You could do that any # of ways. How I've done it in the past is to create a factory to return WebClient instances (or HttpClient or whatever) from Task. I then used the limited concurrency task scheduler that is an example in the TaskScheduler documentation in MSDN. Set the scheduler to the desired # of connections. Then use the scheduler to schedule the tasks to run. Because the scheduler is enforcing the limits the caller will get back a Task that is blocked while the previous clients are running (up to the configured limit). This prevents you from swamping the system while still allowing you to effectively schedule all the tasks at once and then just wait for them to complete.


    Michael Taylor http://www.michaeltaylorp3.net

    Monday, January 28, 2019 5:08 PM
    Moderator