none
UserAgent settings? RRS feed

  • Question

  • My multitude of attempts to obtain data from a specific website continue to fail. A simple WebClient DownloadString used to work until just recently, but it appears the site is now rejecting anything that does not behave as a browser.

    Here's what I am trying:

        string news = "" ;
        using (var client = new WebClient())
        {
            client.Headers[HttpRequestHeader.Accept]         = "text/html, application/xhtml+xml, image/jxr, */*" ;
            client.Headers[HttpRequestHeader.AcceptLanguage] = "en-AU,en-US;q=0.8,en-GB;q=0.5,en;q=0.3" ;
            client.Headers[HttpRequestHeader.UserAgent]      = "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0" ;
            client.Headers[HttpRequestHeader.AcceptEncoding] = "gzip, deflate" ;
            news = client.DownloadString("https://www.forexfactory.com/calendar.php?week=this") ;
        }
    

    And the result is System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Received an unexpected EOF or 0 bytes from the transport stream.

    As I said, the simple DownloadString alone previously returned the expected results. But it appears that in an effort to reduce web scraping, the site is closing off non-browser requests. My code executes only a few times a day so I want to appear to be a browser for that purpose.

    I am constrained to do this in .NET 3.5. However, I should also say that I do not specifically need to use WebClient. If there is a more robust/efficient solution using, say, HttpWebRequest/HttpWebResponse, I'm happy to use it.

    Working code to achieve this would be greatly appreciated! Thanks!

    Friday, March 17, 2017 8:24 PM

All replies

  • My multitude of attempts to obtain data from a specific website continue to fail. A simple WebClient DownloadString used to work until just recently, but it appears the site is now rejecting anything that does not behave as a browser.

    Here's what I am trying:

        string news = "" ;
        using (var client = new WebClient())
        {
            client.Headers[HttpRequestHeader.Accept]         = "text/html, application/xhtml+xml, image/jxr, */*" ;
            client.Headers[HttpRequestHeader.AcceptLanguage] = "en-AU,en-US;q=0.8,en-GB;q=0.5,en;q=0.3" ;
            client.Headers[HttpRequestHeader.UserAgent]      = "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0" ;
            client.Headers[HttpRequestHeader.AcceptEncoding] = "gzip, deflate" ;
            news = client.DownloadString("https://www.forexfactory.com/calendar.php?week=this") ;
        }

    And the result is System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Received an unexpected EOF or 0 bytes from the transport stream.

    As I said, the simple DownloadString alone previously returned the expected results. But it appears that in an effort to reduce web scraping, the site is closing off non-browser requests. My code executes only a few times a day so I want to appear to be a browser for that purpose.

    I am constrained to do this in .NET 3.5. However, I should also say that I do not specifically need to use WebClient. If there is a more robust/efficient solution using, say, HttpWebRequest/HttpWebResponse, I'm happy to use it.

    Working code to achieve this would be greatly appreciated! Thanks!

    I don't see any issue with your code. Either you must be bugging the server too much that it closing your connection. OR there must be a network issue in anywhere between your application and the server. 

    One thing. how are you going to decompress if you get gzip compressed response from the server? do you need it for any specific reason?

     client.Headers[HttpRequestHeader.AcceptEncoding] = "gzip, deflate" ;
    Saturday, March 18, 2017 7:00 AM
  • Hi Jeronymite,

    This problem occurs when the client computer cannot send an HTTP request. The client computer cannot send the HTTP request because the connection has been closed or is unavailable. This problem may occur when the client computer is sending lots of data. To resolve this problem, see resolutions A, D, E, F, and O.

    https://support.microsoft.com/en-us/kb/915599

    Best regards,

    Cole Wu


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Monday, March 20, 2017 9:02 AM
    Moderator