screen scrapping web page by login automatically by passing username,pwd t login page RRS feed

  • Question

  • User-256234869 posted

    Hi All.,

    am new to asp.net.

    am dealing with a task of screen scrapping by entering web site say http://www.sitename.com

    it has one login page say members.php which containe username and password fileds and submitt actions.

    i need to pass these two values from my code to memebers.php and enter to in sitename's index page which is redirected page after successfull login. i have username and pwd values. i don't want to manually key in the values in memebers.php. this should be done automatically from my code.

    i came to know using webrequest method. but how to handle the submitt action and passing values to members.php.

    i appriciate if some body help me .. this is bit urgent..

    thanks for ur time.m



    Sunday, May 8, 2005 11:14 PM

All replies

  • User1385293158 posted

    did you try to pass the user and pswd as http://user:pswd@www.sitename.com/

    Monday, May 9, 2005 10:19 AM
  • User-1469150719 posted
    that only works if the directory is protected by either Apache or Windows Integrated Security in IIS.  To do this you need to simulate a form post to the action of the login page.

    There are numerous resources on how to do this, but basically you construct your own HttpWebRequest, give it the names & values of the variables to submit, (login/password).  The uri should be the action of the login form, and make sure & set the content type.  Then you fire the request and it will return a response.   You can stream this response directly into a file if you like, or parse it for other content.

    Saturday, May 21, 2005 10:58 PM
  • User-256234869 posted
    Thanks for info. but am trying to pass username passwords to httpwebrequest. i couldn't make it.
    Could u help me in this.
    as i brief previously, i need to login to sitename.com which ontains login.php and navigate directly with out key in the username/password manually.
    for this am using HTTPWebRequest.

    I appriciate for help asap..

    Thanks for u r time.,

    Monday, June 27, 2005 1:18 AM
  • User-1469150719 posted
    what you need to do is view the source of that login.php and see what form variables you need to set.  You do not need to programmatically type in the username and password, you just submit a form post to the Action page.

    Read my above post again.

    Monday, June 27, 2005 9:07 AM
  • User-182265138 posted

    hi all,

    bellow is a class i've created that simulates  a form post to a page,

    all you have to do is creating an instance, sert the action page, and add form fields and their values.

    namespace Arshak.FormPost


    public class RemotePost


    private System.Collections.Specialized.NameValueCollection Inputs

    = new System.Collections.Specialized.NameValueCollection();

    public string Url = "";

    public string Method = "post";

    public string FormName = "form1";

    public void Add(string name,string value)




    public void Post()




    System.Web.HttpContext.Current.Response.Write(string.Format("</head><body onload=\"document.{0}.submit()\">",FormName));

    System.Web.HttpContext.Current.Response.Write(string.Format("<form name=\"{0}\" method=\"{1}\" action=\"{2}\" >",FormName,Method,Url));

    for(int i=0;i< Inputs.Keys.Count;i++)


    System.Web.HttpContext.Current.Response.Write(string.Format("<input name=\"{0}\" type=\"hidden\" value=\"{1}\">",Inputs.Keys[ i ],Inputs[Inputs.Keys[ i ]]));








    Tuesday, July 5, 2005 7:18 AM