none
How to visit all web pages using SemaphoreSlim RRS feed

  • Question

  • Hello:
    I have the following C# code to visit some web sites: (about 20 to 30)

    private static SemaphoreSlim _semaphore = new SemaphoreSlim(3);
    private static List<string> total_pages_urls = new List<string>();
    public static async Task Browse_Page1(string page_url1)
    {
    _semaphore.Wait();
    try
    {
        _page = await _proxy_browser.NewPageAsync();
        await _page.SetViewportAsync((new ViewPortOptions { Width = 1920, Height = 3938 }));
        await _page.GoToAsync(page_url1);
        await _page.WaitForTimeoutAsync(10000);
    }
    finally
    {
        _semaphore.Release();
    }
    }
    
    for (int i = 1; i <= 20; i++)
    {
        string page_url1 = string.Format("https://www.mysite.com/today/{0}", i);
        total_pages_urls.Add(page_url1);
    }
    List<Task> all_page_tasks = new List<Task>();
    foreach (string page_url1 in total_pages_urls)
    {
        Task page_task1 = Browse_Page1(page_url1);
        all_page_tasks.Add(page_task1);
    }
    Task.WaitAll(all_page_tasks.ToArray());
    
    The issue is, no matter what is my settings, each iteration, there are some web sites not vistied.
    I got error message like the following:
    Inner Exception 1:
    NavigationException: net::ERR_ABORTED at https://www.mysite.com/today/10 
    https://www.mysite.com/today/10
    For the total 20 web pages, there should be 3 to 5 pages not visited.  The error is: timeout.
    Is there any way that I collected all the missing pages (3 to 5) and build a new list of Tasks and run the task again, I hope the second iteration, all the missing pages get visited.
    By the way, I am using Visual Studio 2019 Version 16.3.5 targetting .Net Core 3.0.
    Please advice!

    Saturday, October 19, 2019 10:00 AM

Answers

  • Hello:

    What is the difference between ‘await _semaphore.WaitAsync()’ and ‘_semaphore.Wait()’

    By the way, from my testing, if I added a little more delay before visit the web page, the success rate is much higher, some times, it is 100%, so once is enough.

    But I can also do this: add catch time out exception, and add the NOK web links to a string list called 'Take2', and after WaitAll, check to see if there is any string in the 'Take2' list, and visit all of the web links in 'Take2' one by one.  Then it works!

    • Marked as answer by zydjohn Monday, October 21, 2019 4:14 PM
    Monday, October 21, 2019 6:39 AM

All replies

  • I think that you should use ‘await _semaphore.WaitAsync()’ instead of ‘_semaphore.Wait()’. If you can intercept the errors using a catch in your try block, then maybe repeat the operations using a loop. Make sure that _page is a local (not static) variable.

    Saturday, October 19, 2019 6:23 PM
  • Hello:

    What is the difference between ‘await _semaphore.WaitAsync()’ and ‘_semaphore.Wait()’

    By the way, from my testing, if I added a little more delay before visit the web page, the success rate is much higher, some times, it is 100%, so once is enough.

    But I can also do this: add catch time out exception, and add the NOK web links to a string list called 'Take2', and after WaitAll, check to see if there is any string in the 'Take2' list, and visit all of the web links in 'Take2' one by one.  Then it works!

    • Marked as answer by zydjohn Monday, October 21, 2019 4:14 PM
    Monday, October 21, 2019 6:39 AM
  • Hi zydjohn, 

    Thank you for posting here.

    Semaphore.Wait() method will block the caller if semaphore is not available at the moment.

    You can refer to the following reference.

    Semaphore Wait vs WaitAsync in an async method

    Besides, Is your problem solved? If so, please click "Mark as answer" to the appropriate answer, so that it will help other members to find the solution quickly if they face a similar issue.

    Best Regards,

    Xingyu Zhao


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Monday, October 21, 2019 7:54 AM
    Moderator