locked
Can't getting webpage source RRS feed

 > 

  • Question

  • Question
    Sign in to vote
    0
    Sign in to vote
    User284589102 posted

    The Below function is my Getting pagesource function which is working fine with the other links to get page source but when i am trying with the "http://search.lycos.com/web?q=web" it gives an error "The remote server returned an error: (500) Internal Server Error."

    string getHtml(string url)
{
HttpWebRequest myWebRequest = (HttpWebRequest)HttpWebRequest.Create(url);
myWebRequest.Method = "GET";
// make request for web page
HttpWebResponse myWebResponse = (HttpWebResponse)myWebRequest.GetResponse();
StreamReader myWebSource = new StreamReader(myWebResponse.GetResponseStream());
string myPageSource = string.Empty;
myPageSource= myWebSource.ReadToEnd();
myWebResponse.Close();
return myPageSource;
}

    Please help me to get pagesource from the link above as early as possible..

    Looking Forward for your quick response..

    Thanks & Regards,

    Parth Pandya

    Tuesday, April 24, 2012 6:37 AM

Answers

  • Question
    Sign in to vote
    0
    Sign in to vote
    User710903421 posted

    This is a protection from lycos.

    you need  complete browser headers.

    Following code work:

            string getHtml(string url)
        {
            HttpWebRequest myWebRequest = (HttpWebRequest)HttpWebRequest.Create(url);

            myWebRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.2; BRI/2; .NET4.0C; .NET4.0E; Zune 4.7; InfoPath.3)";
            myWebRequest.Accept = "application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, application/x-shockwave-flash, */*";
            myWebRequest.AllowAutoRedirect = true;
            myWebRequest.Headers.Add("Accept-Language", "de-DE");
            myWebRequest.Headers.Add("Accept-Encoding", "gzip, deflate");
     
            myWebRequest.Method = "GET";
            // make request for web page

            HttpWebResponse myWebResponse = (HttpWebResponse)myWebRequest.GetResponse();
            StreamReader myWebSource = new StreamReader(myWebResponse.GetResponseStream());
            string myPageSource = string.Empty;
            myPageSource = myWebSource.ReadToEnd();
            myWebResponse.Close();
            //myWebResponse.StatusCode
            return myPageSource;
        }



    • Marked as answer by Anonymous Thursday, October 7, 2021 12:00 AM
    Tuesday, April 24, 2012 9:24 AM