none
c# get dynamic webpage data RRS feed

  • Question

  • Dear all!

    I'm new to programing. I am trying to get data from a webpage. Example: I tring to get if an IP is listed at a blacklist, the webpage takes time to search or fully load the page, and even when it does, the source code does not have the information what i 'd like to obtain, like the IP is on the blacklist. Any help would be welcome

     HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a42.51.217.54&run=toolpage");
    //set big timeout 30 sec
     request.Timeout = 30 * 1000;
    // put result to a textbox to see
     string postData = textBox1.Text;
    
     request.Method = "Post"; // we will post the data using post method
    
                    // data to be posted using HttpWebrequest post method
                    string postData = textBox1.Text;
                    // Convert this string into stream of bytes
                    byte[] arrPostDAta = System.Text.Encoding.GetEncoding(1252).GetBytes(postData);
                    // set request content length = post data length
                    request.ContentLength = arrPostDAta.Length;
                    // get request stream
                    System.IO.Stream strmPostData = request.GetRequestStream();
    
                    // write post data to stream of request
                    strmPostData.Write(arrPostDAta, 0, arrPostDAta.Length);
    
                    strmPostData.Close();
    
                    // upload post data and Get Response from server 
                      
                    HttpWebResponse response = (HttpWebResponse)request.GetResponse();
     
                    StreamReader reader = new StreamReader(response.GetResponseStream());
                    textBox1.Text = reader.ReadToEnd();
    
                    reader.Close();
    
                    response.Close();

    Tuesday, November 20, 2018 3:31 PM

All replies

  • What is your question exactly? The code you posted just loads a webpage. That wouldn't be related to parsing a webpage looking for stuff. For that you'll need a site crawler like HtmlAgilityPack. We won't be able to help you with the enumeration part as that is very much specific to the HTML page you're crawling. You'll have to use a combination of the browser tools to get the layout of the page in combination with HAP to read it programmatically to find what you want.

    Michael Taylor http://www.michaeltaylorp3.net

    Tuesday, November 20, 2018 4:25 PM
    Moderator
  • Hello, 

    I think you a bit misunderstand what and where going on in the web-app. 

    You need to look on this - what formed on the server, what are send to the client and what (and how) are getting from client. After you get this - will be a scene to look what you mean under "IP is on the blacklist"


    Sincerely, Highly skilled coding monkey.

    Tuesday, November 20, 2018 4:45 PM
  • Thank you for all replies, i do not know where to search, so keywords like site crawler and html agilitypack are helpful, thank you. 

    "You need to look on this - what formed on the server, what are send to the client and what (and how) are getting from client. " - and how can i do that?

    Tuesday, November 20, 2018 5:55 PM

  • I am trying to get data from a webpage. Example: I tring to get if an IP is listed at a blacklist...

    You want to see which IPs are blacklisted on a website? Why should the admin wanting to get this informations leaked? Maybe he wants to get some IPs inside a teergrube or wanting them to think they are on the real website. And what is if there is a cdn in the middle? Do you want to see the blacklist of them, too? I think there is no way to see the whole blacklist of a server without access... Maybe you can find out what services are running on the server, after that you try to find some exploits for the services and try them... Maybe you find a way in...
    Tuesday, November 20, 2018 6:31 PM