none
How to make a valid HttpWebRequest?

    Question

  • Hi there,

    I am struggling with a proper HttpWebRequest to finishline.com. I spent several hours debuging the code, playing with Fiddler and googling, but still no luck, so hopefully you will help me.

    I am trying to get the source of www. finishline. com. The catch is, the HttpWebRequest works in some regions (like here in Slovakia), but doesn't work in USA what I need to achieve.

    For USA the request.GetResponse() just time outs. I have tried countless headers combinations, but without success. Can you please help? Thank you

    var request = (HttpWebRequest)WebRequest.Create("http://www.finishline.com");
    
                request.CookieContainer = new CookieContainer();
                request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36";
                request.AutomaticDecompression = (DecompressionMethods.GZip | DecompressionMethods.Deflate);
                request.Headers.Add("Accept-Encoding", "gzip, deflate");
                request.Accept = " text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
                request.Headers.Add("Upgrade-Insecure-Requests", "1");                
                request.Headers.Add("Accept-Language", "sk,cs;q=0.8,en-US;q=0.5,en;q=0.3");
                request.KeepAlive = true;
                request.Headers.Add("Cache-Control", "max-age=0");
    
                var responseText = "";
                using (var response = request.GetResponse())
                {
                    var httpWebResponse = response.GetResponseStream();
    
                    using (var sr = new StreamReader(httpWebResponse))
                    {
                        responseText = sr.ReadToEnd();
                    }
                }

    Wednesday, March 29, 2017 5:38 AM

All replies

  • You probably hit on an IP blocker (firewall that updates rule to block continuous/suspicious web request, or those which simply block IP regions)

    To workaround that, you need a proxy server in between. Because of this, I'll just stop here with partial answer. Since if you're site owner of that website, you can simply ask their network admin to add your IP to exemption rules. When you need way to workaround protection of a company, it falls into boundary of malicious code.


    Wednesday, March 29, 2017 6:17 AM
    Answerer
  • don't worry, we are not talking about malicious software here at all.

    Anyway, I have tried proxies, US VPN and also friends directly from US confirmed it is not working. Moreover, if this would be a firewall issue, I believe it will end up with some error instead of timeout.

    Wednesday, March 29, 2017 6:26 AM
  • The default firewall rule for blocking requests is to "drop" connection instead of "close". Since nothing ever will come from the remote side, you'll get "timeout" as result.

    From what you see, it could be blocking IP by region. Try using proxies from another country to access it then.

    Wednesday, March 29, 2017 7:03 AM
    Answerer
  • yes it works for different regions. But how come it works via browsers in those blocked regions? Shouldn't they be blocked as well?
    Wednesday, March 29, 2017 7:06 AM
  • This works for me from France (I have also a TimeOut without "Accept-Language")

    =>

                var webRequest = (HttpWebRequest)WebRequest.Create("http://www.finishline.com");
                if (webRequest != null)
                {
                    webRequest.Method = "GET";
                    webRequest.Timeout = 5000;
                    webRequest.Headers.Add("Accept-Encoding", "gzip, deflate");
                    webRequest.Headers.Add("Accept-Language", "fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3");
                    webRequest.AutomaticDecompression = (DecompressionMethods.GZip | DecompressionMethods.Deflate);
                    using (Stream s = webRequest.GetResponse().GetResponseStream())
                    {
                        using (StreamReader sr = new StreamReader(s))
                        {
                            var httpWebResponse = sr.ReadToEnd();
                            Console.WriteLine(String.Format("Response: {0}", httpWebResponse));
                        }
                    }
                }

    Wednesday, March 29, 2017 7:39 AM
  • This works for me from France (I have also a TimeOut without "Accept-Language")

    =>

                var webRequest = (HttpWebRequest)WebRequest.Create("http://www.finishline.com");
                if (webRequest != null)
                {
                    webRequest.Method = "GET";
                    webRequest.Timeout = 5000;
                    webRequest.Headers.Add("Accept-Encoding", "gzip, deflate");
                    webRequest.Headers.Add("Accept-Language", "fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3");
                    webRequest.AutomaticDecompression = (DecompressionMethods.GZip | DecompressionMethods.Deflate);
                    using (Stream s = webRequest.GetResponse().GetResponseStream())
                    {
                        using (StreamReader sr = new StreamReader(s))
                        {
                            var httpWebResponse = sr.ReadToEnd();
                            Console.WriteLine(String.Format("Response: {0}", httpWebResponse));
                        }
                    }
                }

    hm, thanks. Will try altering that header, maybe that will fix it. Will try it and post the result. Thanks again
    Wednesday, March 29, 2017 7:41 AM
  • I don't know, maybe they're using some Apache modules like "fail2ban" to do that?

    You'll never know unless you have chance to take a look in their web server.

    But as far as I can tell, they do also have web monitoring tool that can generate traffic analysis (region list, client download speed, page counter, etc.) to achieve that effect.


    You'll need to have machines that located to blocking IP and other machines that can access the website to do compare. Without access to blocking IPs it's difficult to know what is blocking.
    Wednesday, March 29, 2017 7:53 AM
    Answerer