locked
How to properly use HttpWebRequest to login to a web site using forms? RRS feed

  • Question

  • Hello everyone...

    After my crash-course (trial by fire) experience with Visual Basic 2008, I am nearly finished with my first application after 3 days of programming. My validations are put in place, my app is working (by some manual methods of passing data) and all is well.

    My final step is to automate my manual process of logging in to a web site using forms (passing username and password stored in My.Settings within the application) and then redirecting silently to a page and passing in the user's input within my VB form to the HTML form on the web site.

    My main quesetion with all that said is, does anyone have sample code on how to properly utilize HttpWebRequest to login to a web page with cookies, and pass in the username and password and ultimately, redirect to a submission form where I will pass in other text field values passed into the VB app into the web form and ultimately submit?

    I have seen some examples of HttpWebRequest (with cookies) on the MSDN forums but nothing that I can get to work. I need the simple version of the code without many bells and whistles but robust enough to alert me of an exceptions, etc.

    Does anyone haave some advice on how I could go about proceeding with this and wrapping up my application?

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

    P.S. Why when other users download and test my software, it requires them to download .NET Framework? As you know, this is a larger sized download in comparison to my applicaiton - is there a way to not require the install of .NET Framework prior to launching the application?
    Sunday, November 25, 2007 5:06 PM

Answers

  •  

    It's a complicated class.....

     

    And I wrote it to be universal. Not just login. It does have some terse doumentation

     

    Dim a As New Webpost
        Private Sub Form1_Load(ByVal sender As Object, ByVal e As System.EventArgs) Handles Me.Load
            InitGEOPostRequest()
            Dim b As String = a.PostRequest("68.142.207.145")

     

        End Sub
        Private Sub InitGEOPostRequest()
            a.Url() = "http://www.geobytes.com/IpLocator.htm?GetLocation"
            a.PostStringPrelog() = "cid=0&c=0&Template=iplocator.htm&ipaddress="
            a.Accept() = "image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash," _
             & "application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*"
            a.ContentType() = "application/x-www-form-urlencoded"
            a.UserAgent() = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)"
        End Sub
        'requires Imports of System.net, system.text and system.text

       

     

     Public Class Webpost
            Private m_UserAgent As String
            Private m_PostDataPrelog As String
            Private m_LookupIP As String
            Private m_URL As String
            Private m_Accept As String
            Private m_ContentType As String

            Public Property Url() As String
                Get
                    Url = m_URL
                End Get
                Set(ByVal Value As String)
                    m_URL = Value
                End Set
            End Property
            Public Property UserAgent() As String
                Get
                    UserAgent = m_UserAgent
                End Get
                Set(ByVal Value As String)
                    m_UserAgent = Value
                End Set
            End Property
            Public Property ContentType() As String
                Get
                    ContentType = m_ContentType
                End Get
                Set(ByVal Value As String)
                    m_ContentType = Value
                End Set
            End Property

            Public Property PostStringPrelog() As String
                Get
                    PostStringPrelog = m_PostDataPrelog
                End Get
                Set(ByVal Value As String)
                    m_PostDataPrelog = Value
                End Set
            End Property

            Public Property Accept() As String
                Get
                    Accept = m_Accept
                End Get
                Set(ByVal Value As String)
                    m_Accept = Value
                End Set
            End Property


            Public Function PostRequest(ByRef IPstr As String) As String
                Const cRoutineName As String = "PostRequest"
                Const cPropMsg As String = " property is not set."
                PostRequest = ""
                ' Set the 'Method' property of the 'Webrequest' to 'POST'.

                If m_URL = "" Then
                    MsgBox("URL" & cPropMsg, MsgBoxStyle.Exclamation, cRoutineName)
                    Exit Function
                End If

                If m_ContentType = "" Then
                    MsgBox("ContentType" & cPropMsg, MsgBoxStyle.Exclamation, cRoutineName)
                    Exit Function
                End If

                If m_PostDataPrelog = "" Then
                    MsgBox("PostDataPrelog" & cPropMsg, MsgBoxStyle.Exclamation, cRoutineName)
                    Exit Function
                End If

                If m_Accept = "" Then
                    MsgBox("Accept" & cPropMsg, MsgBoxStyle.Exclamation, cRoutineName)
                    Exit Function
                End If

                If m_UserAgent = "" Then
                    MsgBox("UserAgent" & cPropMsg, MsgBoxStyle.Exclamation, cRoutineName)
                    Exit Function
                End If

                Dim encoding As New ASCIIEncoding()
                Dim byte1 As Byte() 'm_PostDataPrelog
                Dim postData As String = m_PostDataPrelog & IPstr ' Create a new string object to POST data to the Url.
                Dim PostStream As Stream = Nothing
                Dim response As HttpWebResponse = Nothing
                Dim ResponseStream As Stream = Nothing

                Try
                    Dim webreq As HttpWebRequest = WebRequest.Create(m_URL)
                    webreq.Method = "POST"
                    webreq.UserAgent = m_UserAgent
                    byte1 = encoding.GetBytes(postData)
                    ' Set the content type of the data being posted.
                    webreq.Accept = m_Accept
                    webreq.ContentType = m_ContentType
                    ' Set the content length of the string being posted.
                    webreq.ContentLength = postData.Length
                    PostStream = webreq.GetRequestStream()
                    PostStream.Write(byte1, 0, byte1.Length)
                    response = CType(webreq.GetResponse(), HttpWebResponse)
                    ResponseStream = response.GetResponseStream()
                    Dim reader As New StreamReader(ResponseStream)
                    PostRequest = reader.ReadToEnd()
                Catch e As Exception
                    MsgBox("WEB I/O Exception: " & e.Message, MsgBoxStyle.Exclamation, cRoutineName)
                End Try
                ResponseStream.Close()
                PostStream.Close()

            End Function
        End Class

    Sunday, November 25, 2007 5:37 PM
  • Qoute:
    P.S. Why when other users download and test my software, it requires them to download .NET Framework? As you know, this is a larger sized download in comparison to my applicaiton - is there a way to not require the install of .NET Framework prior to launching the application?

    VB.NET requires the .NET framework.  If you downgrade to Visual Basic 2005 much less people will have to download the .NET framework since it only requires 2 instead of 3, and most people already have the .Net 2 frame work.
    Sunday, November 25, 2007 9:47 PM
  •  michaelamburgey wrote:


            ' Part 1: Use WebBrowser control to load web page
            WebBrowser1.Navigate("http://freemywii.com/forums/member.php?action=login")

            ' Delay 1 second to render login page
            System.Threading.Thread.Sleep(1000)

            ' Part 2: Automatically input username and password
            Dim theElementCollection As HtmlElementCollection = WebBrowser1.Document.GetElementsByTagName("input")
            ' The error will ocuur here


    This however returns the following error: System.NullReferenceException
    My options are to use the "New" keyword to create an object instance or check to determine if the object is null before calling the method.
    ~ Any ideas on what I am amissing...?

     

    Hi Mike,

     

    The cause to the error is the WebBrowser has not completed loading the webpage yet, so cannot find elements (e.g.<input>).

     

    Solution: To ensure the WebBrowser has completed loading the webpage, please use WebBrowser1_DocumentCompleted event.

    Code Block

    Public Class Form1

     

        Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click

            ' Part 1: Use WebBrowser control to load web page

            WebBrowser1.Navigate(http://freemywii.com/forums/member.php?action=login)

        End Sub

     

        Private Sub WebBrowser1_DocumentCompleted(ByVal sender As System.Object, ByVal e As System.Windows.Forms.WebBrowserDocumentCompletedEventArgs) Handles WebBrowser1.DocumentCompleted

            ' Part 2: Automatically input username and password

            Dim theElementCollection As HtmlElementCollection = WebBrowser1.Document.GetElementsByTagName("input")

     

            For Each curElement As HtmlElement In theElementCollection

     

                Dim controlName As String = curElement.GetAttribute("name").ToString

     

                If controlName = "username" Then

     

                    ' curElement.GetAttribute("Value")

     

                    curElement.SetAttribute("Value", My.Settings.username)

     

                ElseIf controlName = "password" Then

     

                    curElement.SetAttribute("Value", My.Settings.password)

     

                End If

     

            Next

     

            ' Part 3: Automatically click the Login button

            Dim theWElementCollection As HtmlElementCollection = WebBrowser1.Document.GetElementsByTagName("input")

            For Each curElement As HtmlElement In theWElementCollection

                If curElement.GetAttribute("value").Equals("Login") Then

                    curElement.InvokeMember("click")

                    ' javascript has a click method for we need to invoke on the current submit button element.

                End If

            Next

        End Sub

    End Class

    I have tried and it works fine.

    It can automatically input username/password and click login button.

    BTW, you also consider loading webpage in Form1_Load event.

    Code Block

        Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load

            ' Part 1: Use WebBrowser control to load web page

            WebBrowser1.Navigate("http://freemywii.com/forums/member.php?action=login")

        End Sub

     

    Best regards,

    Martin

    Monday, December 3, 2007 3:11 AM

All replies

  •  

    "is there a way to not require the install of .NET Framework prior to launching the application?"

     

    No.

     

    I've built some classes regarding what you are doing BUT, no cookies. I'm sorry.

    Sunday, November 25, 2007 5:18 PM
  • ReneeC,

    Thanks for the input on the .NET Framework install.

    Could you post some code of how to do the login which you have built. I may be able to integrate the cookies part into the code myself if I have a good starting point.

    Let me know.
    Thanks,
    Mike
    Sunday, November 25, 2007 5:26 PM
  •  

    It's a complicated class.....

     

    And I wrote it to be universal. Not just login. It does have some terse doumentation

     

    Dim a As New Webpost
        Private Sub Form1_Load(ByVal sender As Object, ByVal e As System.EventArgs) Handles Me.Load
            InitGEOPostRequest()
            Dim b As String = a.PostRequest("68.142.207.145")

     

        End Sub
        Private Sub InitGEOPostRequest()
            a.Url() = "http://www.geobytes.com/IpLocator.htm?GetLocation"
            a.PostStringPrelog() = "cid=0&c=0&Template=iplocator.htm&ipaddress="
            a.Accept() = "image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash," _
             & "application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*"
            a.ContentType() = "application/x-www-form-urlencoded"
            a.UserAgent() = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)"
        End Sub
        'requires Imports of System.net, system.text and system.text

       

     

     Public Class Webpost
            Private m_UserAgent As String
            Private m_PostDataPrelog As String
            Private m_LookupIP As String
            Private m_URL As String
            Private m_Accept As String
            Private m_ContentType As String

            Public Property Url() As String
                Get
                    Url = m_URL
                End Get
                Set(ByVal Value As String)
                    m_URL = Value
                End Set
            End Property
            Public Property UserAgent() As String
                Get
                    UserAgent = m_UserAgent
                End Get
                Set(ByVal Value As String)
                    m_UserAgent = Value
                End Set
            End Property
            Public Property ContentType() As String
                Get
                    ContentType = m_ContentType
                End Get
                Set(ByVal Value As String)
                    m_ContentType = Value
                End Set
            End Property

            Public Property PostStringPrelog() As String
                Get
                    PostStringPrelog = m_PostDataPrelog
                End Get
                Set(ByVal Value As String)
                    m_PostDataPrelog = Value
                End Set
            End Property

            Public Property Accept() As String
                Get
                    Accept = m_Accept
                End Get
                Set(ByVal Value As String)
                    m_Accept = Value
                End Set
            End Property


            Public Function PostRequest(ByRef IPstr As String) As String
                Const cRoutineName As String = "PostRequest"
                Const cPropMsg As String = " property is not set."
                PostRequest = ""
                ' Set the 'Method' property of the 'Webrequest' to 'POST'.

                If m_URL = "" Then
                    MsgBox("URL" & cPropMsg, MsgBoxStyle.Exclamation, cRoutineName)
                    Exit Function
                End If

                If m_ContentType = "" Then
                    MsgBox("ContentType" & cPropMsg, MsgBoxStyle.Exclamation, cRoutineName)
                    Exit Function
                End If

                If m_PostDataPrelog = "" Then
                    MsgBox("PostDataPrelog" & cPropMsg, MsgBoxStyle.Exclamation, cRoutineName)
                    Exit Function
                End If

                If m_Accept = "" Then
                    MsgBox("Accept" & cPropMsg, MsgBoxStyle.Exclamation, cRoutineName)
                    Exit Function
                End If

                If m_UserAgent = "" Then
                    MsgBox("UserAgent" & cPropMsg, MsgBoxStyle.Exclamation, cRoutineName)
                    Exit Function
                End If

                Dim encoding As New ASCIIEncoding()
                Dim byte1 As Byte() 'm_PostDataPrelog
                Dim postData As String = m_PostDataPrelog & IPstr ' Create a new string object to POST data to the Url.
                Dim PostStream As Stream = Nothing
                Dim response As HttpWebResponse = Nothing
                Dim ResponseStream As Stream = Nothing

                Try
                    Dim webreq As HttpWebRequest = WebRequest.Create(m_URL)
                    webreq.Method = "POST"
                    webreq.UserAgent = m_UserAgent
                    byte1 = encoding.GetBytes(postData)
                    ' Set the content type of the data being posted.
                    webreq.Accept = m_Accept
                    webreq.ContentType = m_ContentType
                    ' Set the content length of the string being posted.
                    webreq.ContentLength = postData.Length
                    PostStream = webreq.GetRequestStream()
                    PostStream.Write(byte1, 0, byte1.Length)
                    response = CType(webreq.GetResponse(), HttpWebResponse)
                    ResponseStream = response.GetResponseStream()
                    Dim reader As New StreamReader(ResponseStream)
                    PostRequest = reader.ReadToEnd()
                Catch e As Exception
                    MsgBox("WEB I/O Exception: " & e.Message, MsgBoxStyle.Exclamation, cRoutineName)
                End Try
                ResponseStream.Close()
                PostStream.Close()

            End Function
        End Class

    Sunday, November 25, 2007 5:37 PM
  • Qoute:
    P.S. Why when other users download and test my software, it requires them to download .NET Framework? As you know, this is a larger sized download in comparison to my applicaiton - is there a way to not require the install of .NET Framework prior to launching the application?

    VB.NET requires the .NET framework.  If you downgrade to Visual Basic 2005 much less people will have to download the .NET framework since it only requires 2 instead of 3, and most people already have the .Net 2 frame work.
    Sunday, November 25, 2007 9:47 PM
  • Man that is a bummer. For such a small application, it requires the entire framework package which is rather large in comparison... It's a shame we have to go that route just to run this small app.

    Regardless, feedback on my original post regarding the HttpWebRequest would be helpful. I couldn't follow the code supplied by ReneeC for "my" purposes...

    If someone has a method to simply, on click of a button ,login and then follow my remaining commands, let me know. I am utilzing My.Settings.Username and My.Settings.Password in my User Options and Preferences form which stores that information.

    Any further assistance is greatly appreciated.

    Thanks,
    Mike
    Sunday, November 25, 2007 10:50 PM
  •  michaelamburgey wrote:

    My final step is to automate my manual process of logging in to a web site using forms (passing username and password stored in My.Settings within the application) and then redirecting silently to a page and passing in the user's input within my VB form to the HTML form on the web site.

     

    If someone has a method to simply, on click of a button ,login and then follow my remaining commands, let me know. I am utilzing My.Settings.Username and My.Settings.Password in my User Options and Preferences form which stores that information.

     

    Hi Mike,

     

    We can access elements of a webpage loaded in WebBrowser control.

    Code Block

    Public Class Form1

     

        Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load

     

    ' Part 1: Use WebBrowser control to load web page

            WebBrowser1.Navigate("http://website/login.aspx")

            System.Threading.Thread.Sleep(2000) ' Delay 2 seconds to render login page

     

    ' Part 2: Automatically input username and password

    ' Similarly, a TextBox control format in HTML is as below:

    ' <input name="UserNameTextBox" type="text" value="myUser" id="UserNameTextBox" />

    ' Thus we can retrieve and reset TextBox.Value like this:

     

           Dim theElementCollection As HtmlElementCollection =Me.WebBrowser1.Document.GetElementsByTagName("input")

            For Each curElement As HtmlElement In theElementCollection

                Dim controlName As String = curElement.GetAttribute("name").ToString

                If controlName = "UserNameTextBox" Then

                    ' curElement.GetAttribute("Value")

                    curElement.SetAttribute("Value", My.Settings.Username)

                Else If controlName = "PasswordTextBox" Then

                    curElement.SetAttribute("Value", My.Settings.Password)

                End If

            Next

    ' Part 3: Automatically clck that Login button

    ' Reference: http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=2029127&SiteID=1

    ' To click programically that Login button or Submit button in a form within the webbrowser control, you need to invoke that button like this:

    Dim theElementCollection As HtmlElementCollection = Me.theWebBrowser.Document.GetElementsByTagName("input")
    For Each curElement As HtmlElement In theElementCollection
        
    If curElement.GetAttribute("value").Equals("Login"Then
            
    curElement.InvokeMember("click")
            
    ' javascript has a click method for we need to invoke on the current submit button element.
        
    End If
    Next
     

        End Sub

     

       End Class

     

    Please check these threads about how to access each element (such as link or img) on a webpage loaded in WebBrowser control.

    http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=2400267&SiteID=1

    http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=2355599&SiteID=1

     

    I hope that can help you.

     

    Regards,

    Martin

    Friday, November 30, 2007 1:38 PM
  • Tremendous thanks to you Martin. I will review this now and get back to you.

    It appears as if this may be exactly what I need to accomplish the task in the easiest fashion.

    Thanks again,
    Mike A.
    Saturday, December 1, 2007 12:56 AM
  • Martin,

    I have my code as follows:

    [code]
    Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click

            ' Part 1: Use WebBrowser control to load web page
            WebBrowser1.Navigate("http://freemywii.com/forums/member.php?action=login")

            ' Delay 1 second to render login page
            System.Threading.Thread.Sleep(1000)

            ' Part 2: Automatically input username and password
            Dim theElementCollection As HtmlElementCollection = WebBrowser1.Document.GetElementsByTagName("input")

            For Each curElement As HtmlElement In theElementCollection

                Dim controlName As String = curElement.GetAttribute("name").ToString

                If controlName = "username" Then

                    ' curElement.GetAttribute("Value")

                    curElement.SetAttribute("Value", My.Settings.Username)

                ElseIf controlName = "password" Then

                    curElement.SetAttribute("Value", My.Settings.Password)

                End If

            Next

            ' Part 3: Automatically click the Login button
            Dim theWElementCollection As HtmlElementCollection = WebBrowser1.Document.GetElementsByTagName("input")
            For Each curElement As HtmlElement In theWElementCollection
                If curElement.GetAttribute("value").Equals("Login") Then
                    curElement.InvokeMember("click")
                    ' javascript has a click method for we need to invoke on the current submit button element.
                End If
            Next

    End Sub
    [/code]

    This however returns the following error: System.NullReferenceException

    My options are to use the "New" keyword to create an object instance or check to determine if the object is null before calling the method.

    ~ Any ideas on what I am amissing...?
    Saturday, December 1, 2007 1:19 AM
  •  michaelamburgey wrote:


            ' Part 1: Use WebBrowser control to load web page
            WebBrowser1.Navigate("http://freemywii.com/forums/member.php?action=login")

            ' Delay 1 second to render login page
            System.Threading.Thread.Sleep(1000)

            ' Part 2: Automatically input username and password
            Dim theElementCollection As HtmlElementCollection = WebBrowser1.Document.GetElementsByTagName("input")
            ' The error will ocuur here


    This however returns the following error: System.NullReferenceException
    My options are to use the "New" keyword to create an object instance or check to determine if the object is null before calling the method.
    ~ Any ideas on what I am amissing...?

     

    Hi Mike,

     

    The cause to the error is the WebBrowser has not completed loading the webpage yet, so cannot find elements (e.g.<input>).

     

    Solution: To ensure the WebBrowser has completed loading the webpage, please use WebBrowser1_DocumentCompleted event.

    Code Block

    Public Class Form1

     

        Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click

            ' Part 1: Use WebBrowser control to load web page

            WebBrowser1.Navigate(http://freemywii.com/forums/member.php?action=login)

        End Sub

     

        Private Sub WebBrowser1_DocumentCompleted(ByVal sender As System.Object, ByVal e As System.Windows.Forms.WebBrowserDocumentCompletedEventArgs) Handles WebBrowser1.DocumentCompleted

            ' Part 2: Automatically input username and password

            Dim theElementCollection As HtmlElementCollection = WebBrowser1.Document.GetElementsByTagName("input")

     

            For Each curElement As HtmlElement In theElementCollection

     

                Dim controlName As String = curElement.GetAttribute("name").ToString

     

                If controlName = "username" Then

     

                    ' curElement.GetAttribute("Value")

     

                    curElement.SetAttribute("Value", My.Settings.username)

     

                ElseIf controlName = "password" Then

     

                    curElement.SetAttribute("Value", My.Settings.password)

     

                End If

     

            Next

     

            ' Part 3: Automatically click the Login button

            Dim theWElementCollection As HtmlElementCollection = WebBrowser1.Document.GetElementsByTagName("input")

            For Each curElement As HtmlElement In theWElementCollection

                If curElement.GetAttribute("value").Equals("Login") Then

                    curElement.InvokeMember("click")

                    ' javascript has a click method for we need to invoke on the current submit button element.

                End If

            Next

        End Sub

    End Class

    I have tried and it works fine.

    It can automatically input username/password and click login button.

    BTW, you also consider loading webpage in Form1_Load event.

    Code Block

        Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load

            ' Part 1: Use WebBrowser control to load web page

            WebBrowser1.Navigate("http://freemywii.com/forums/member.php?action=login")

        End Sub

     

    Best regards,

    Martin

    Monday, December 3, 2007 3:11 AM
  • Thanks Martin

    This is just what I have been looking for.

    The only problem I am having is once I have used code to login, I am unable to log out of the site as when I log out, the WebBrowser1_DocumentCompleted fires and logs me back in again, how do I stop this from happening?

     

    Also I have saved username/passwords to an xml file and would love to pull login data frm an xml file instead of MySettings, how do I code for this?

     

    Thanks

    Tuesday, April 29, 2008 10:57 PM
  •  v7web wrote:

    The only problem I am having is once I have used code to login, I am unable to log out of the site as when I log out, the WebBrowser1_DocumentCompleted fires and logs me back in again, how do I stop this from happening?

     

    Hi v7web,

     

    You can add a switch variable indicating whether you has logged out.

    When you log out, set the switch variable as True.

    Code Snippet

    Public Class Form1

        Dim isLogOut As Boolean  ' Set a switch indicating whether you has logged out.

     

        Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load

            isLogOut = False ' Initialize the switch as False

        End Sub

     

        Private Sub WebBrowser1_DocumentCompleted(ByVal sender As System.Object, ByVal e As System.Windows.Forms.WebBrowserDocumentCompletedEventArgs) Handles WebBrowser1.DocumentCompleted

            If isLogOut = True Then

              '  don't need to login again    

            Else 

              '   login again  

            End If

     

        End Sub

    End Class

     

    The following thread has the similar issue, please look through code samples inside.

    http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=2485067&SiteID=1

    Search result (Table data) contains several pages

    click Continue link to show next page

     

     

     v7web wrote:

    Also I have saved username/passwords to an xml file and would love to pull login data frm an xml file instead of MySettings, how do I code for this?

     

    Please check these threads about How to read/write XML document:

    http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=2688591&SiteID=1

    http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=3233039&SiteID=1

     

    XML and the .NET Framework Forum.

     

    Regards,

    Martin

    Wednesday, April 30, 2008 2:58 AM
  • Thanks Martin you are awesome.


    Wednesday, April 30, 2008 5:41 AM
  • Martin,
    Thanks for all your help.

    Would it be possible to invoke an auto log out process using a button, for the following.

    The log out is a link within the following tag:

    <td class="vbmenu_control"><a href="login.php?do=logout&amp;logouthash=91534186da1f6cbae8ec7255cbcbdf1ef9b9e0e2" onclick="return log_out('Are you sure you want to log out?')">Log Out</a></td>

    However the  "91534186da1f6cbae8ec7255cbcbdf1ef9b9e0e2" part changes each time, so how could I grab that number and add it to an url

    "login.php?do=logout&amp;logouthash="
    Then on button click the url plus number is put together as string and visited , therefore logging out.

    If that makes sense!

    Wednesday, April 30, 2008 9:29 AM
  •  v7web wrote:
    Martin,
    Would it be possible to invoke an auto log out process using a button, for the following.
    The log out is a link within the following tag:
    <td class="vbmenu_control"><a href="login.php?do=logout&amp;logouthash=91534186da1f6cbae8ec7255cbcbdf1ef9b9e0e2" onclick="return log_out('Are you sure you want to log out?')">Log Out</a></td>

    However the  "91534186da1f6cbae8ec7255cbcbdf1ef9b9e0e2" part changes each time, so how could I grab that number and add it to an url

    "login.php?do=logout&amp;logouthash="
    Then on button click the url plus number is put together as string and visited , therefore logging out.

     

    You can use GetElementsByTagName or GetElementsByID function to locate webpage elements and automate them (e.g. get/set element value, click button).

     

    No matter what the logouthash is changed, you can locate the Link element and retrieve the href url.

    <a href="login.php?do=logout&amp;logouthash=91534186da1f6cbae8ec7255cbcbdf1ef9b9e0e2" onclick="return log_out('Are you sure you want to log out?')">Log Out</a>

    Code Snippet

     

            Dim theElementCollection As HtmlElementCollection = WebBrowser1.Document.GetElementsByTagName("a)

            For Each curElement As HtmlElement In theElementCollection

                Dim controlText As String = curElement.GetAttribute("InnerText).ToString

                If controlText = "Log Out" Then

                   Dim url As String = curElement.GetAttribute("href")

                   curElement.InvokeMember("click") ' Click the hyperlink

                End If

            Next

     

    Sample 1: http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=2486275&SiteID=1

    Automatically login in one website (input uername/password and click Login button)

     

    Sample 2:

    Automatically input search words and click search button on google homepage http://www.google.com.

     

    The Search TextBox:

    <input maxlength=2048 name=q size=55 title="Google Search" value="">

    The Search Button:

    <input name=btnG type=submit value="Google Search">

    Code Snippet

    Public Class Form1

     

        Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load

            ' Load homepage in Form_Load event

            WebBrowser1.Navigate("http://www.google.com")

            System.Threading.Thread.Sleep(2000) ' Delay 2 seconds to render login page

     

            ' Locate the Search TextBox and automatically input username

            ' <input maxlength=2048 name=q size=55 title="Google Search" value="">

            Dim theElementCollection As HtmlElementCollection = WebBrowser1.Document.GetElementsByTagName("Input")

            For Each curElement As HtmlElement In theElementCollection

                Dim controlName As String = curElement.GetAttribute("Name").ToString

                If controlName = "q" Then

                    curElement.SetAttribute("Value", "Some search words")

                End If

            Next

     

            ' Locate the Search Button and automatically click it

            '<input name=btnG type=submit value="Google Search">

            Dim theWElementCollection As HtmlElementCollection = WebBrowser1.Document.GetElementsByTagName("Input")

            For Each curElement As HtmlElement In theWElementCollection

                If curElement.GetAttribute("value").Equals("Google Search") Then

                    curElement.InvokeMember("click")

                End If

            Next

        End Sub

     

    End Class

    Trackback: http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=3064982&SiteID=1

     

    Regards,

    Martin

    Friday, May 2, 2008 5:52 AM
  • Thanks again Martin,
    I have learnt so much from these posts.

    Paul
    Friday, May 2, 2008 10:02 AM
  • Martin,
    Thanks for the great solution you provided!

    I am in the need to come up with the same functionality provided by your sample application, so when I read your code I though I was saved.  To my surprise I noticed that either the WebControl or the WebBrowser (object) are not available in VS2005 when developing an .aspx page, it is also not available either when developing an .asmx (web service), thus I can not implement the solution you provided in either of those two formats.  Could you kindly suggest (with a code example....Smile ) on how to develop the functionality provided in your sample within the context of an
    .aspx or a .asmx (imo the later .asmx would be better for my scenario).  I will be truly grateful if you can lend me your expertise on this..

    Regards,
    Raf
    Friday, May 2, 2008 11:22 PM
  •  Raf.Fig wrote:
    Martin,
    To my surprise I noticed that either the WebControl or the WebBrowser (object) are not available in VS2005 when developing an .aspx page, it is also not available either when developing an .asmx (web service), thus I can not implement the solution you provided in either of those two formats.  Could you kindly suggest (with a code example...) on how to develop the functionality provided in your sample within the context of an .aspx or a .asmx (imo the later .asmx would be better for my scenario).  I will be truly grateful if you can lend me your expertise on this..

     

    Hi Raf,

     

    Welcome to MSDN forums!

     

    You’ll need to post it at the following forums for better responses.  

     

    ASP.NET experts live in ASP.NET forums

    http://forums.asp.net

     

    ASMX Web Service forum

    http://forums.microsoft.com/MSDN/ShowForum.aspx?ForumID=46&SiteID=1

     

    References:

        How to place webbrowser control in asp.net page

        http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1555046&SiteID=1

     

        Hosting a WebBrowser control in WPF application   

        http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=2828431&SiteID=1

     

     

    Best regards,

    Martin

    Monday, May 5, 2008 4:18 AM
  • Hi Martin,
    I am trying to do the following and struggling!

    Auto navigate 2 pages and then change a "drop down list" value and click "save changes.

    I have the following which gets me to the "dropdownlist" page:

    'Click on User CP.
            Dim theElementCollection1 As HtmlElementCollection = WebBrowser1.Document.GetElementsByTagName("a")
            For Each curElement As HtmlElement In theElementCollection1
                Dim controlText As String = curElement.GetAttribute("InnerText").ToString
                If controlText = "User CP" Then


                    Dim url As String = curElement.GetAttribute("href")
                    curElement.InvokeMember("click") ' Click the hyperlink
                End If
            Next
            'Click on Edit Options.
            Dim theElementCollection2 As HtmlElementCollection = WebBrowser1.Document.GetElementsByTagName("a")
            For Each curElement As HtmlElement In theElementCollection2
                Dim controlText As String = curElement.GetAttribute("InnerText").ToString
                If controlText = "Edit Options" Then

                    Dim url As String = curElement.GetAttribute("href")
                    curElement.InvokeMember("click") ' Click the hyperlink
                End If
            Next

    But then I am struggling with the rest, here are the tags for the choices( i need to be able to choose the
    "Do not subscribe option"):


    <SELECT id=sel_autosubscribe name=autosubscribe> <OPTION value=-1 selected>Do not subscribe</OPTION> <OPTION value=0>No email notification</OPTION> <OPTION value=1>Instant email notification</OPTION> <OPTION value=2>Daily email notification</OPTION> <OPTION value=3>Weekly email notification</OPTION></SELECT>

    And here is the "save changes" tags:

    <INPUT class=button accessKey=s type=submit value="Save Changes"> <INPUT class=button accessKey=r type=reset value="Reset Fields">


    I have tried changing html elements and control text but just cant seem to complete it.
    Help!!!

    All the best,
    Paul
    Sunday, May 11, 2008 10:45 AM
  •  v7web wrote:

    But then I am struggling with the rest, here are the tags for the choices( i need to be able to choose the "Do not subscribe option"):

    <SELECT id=sel_autosubscribe name=autosubscribe>

    <OPTION value=-1 selected>Do not subscribe</OPTION>

    <OPTION value=0>No email notification</OPTION>

    <OPTION value=1>Instant email notification</OPTION>

    <OPTION value=2>Daily email notification</OPTION>

    <OPTION value=3>Weekly email notification</OPTION>

    </SELECT>

    And here is the "save changes" tags:
    <INPUT class=button accessKey=s type=submit value="Save Changes">

    <INPUT class=button accessKey=r type=reset value="Reset Fields">

     

    Hi Paul,

     

    Please check this code sample:

    Code Snippet

    ' Automatically select the "Do not subscribe" option from ComboBox

            Dim theElementCollection As HtmlElementCollection = WebBrowser1.Document.GetElementsByTagName("select")

            For Each curElement As HtmlElement In theElementCollection

                Dim controlName As String = curElement.GetAttribute("name").ToString

                If controlName = "autosubscribe" Then

                    curElement.SetAttribute("Value", -1)

                End If

            Next

     

     ' Automatically click the "Save Changes" button

            Dim theWElementCollection As HtmlElementCollection = WebBrowser1.Document.GetElementsByTagName("input")

            For Each curElement As HtmlElement In theWElementCollection

                Dim controlValue As String = curElement.GetAttribute("value").ToString      

                If controlValue = "Save Changes" Then

                    curElement.InvokeMember("click")          

                End If

            Next

    Trackback: http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=2485067&SiteID=1

     

    Best regards,

    Martin

    Monday, May 12, 2008 5:00 AM
  • Many thanks Martin,

    The code so far gets me to the "save changes" process but, for some strange reason, it will not auto complete this last process. I have to click the control button again for the "save changes" to complete. I cant see what I am doing wrong.
    This is what I have:


    Dim theElementCollection1 As HtmlElementCollection = WebBrowser1.Document.GetElementsByTagName("a")
            For Each curElement As HtmlElement In theElementCollection1
                Dim controlText As String = curElement.GetAttribute("InnerText").ToString
                If controlText = "User CP" Then


                    Dim url As String = curElement.GetAttribute("href")
                    curElement.InvokeMember("click") ' Click the hyperlink
                End If
            Next

            Dim theElementCollection2 As HtmlElementCollection = WebBrowser1.Document.GetElementsByTagName("a")
            For Each curElement As HtmlElement In theElementCollection2
                Dim controlText As String = curElement.GetAttribute("InnerText").ToString
                If controlText = "Edit Options" Then

                    Dim url As String = curElement.GetAttribute("href")
                    curElement.InvokeMember("click") ' Click the hyperlink
                End If
            Next



            Dim theElementCollection3 As HtmlElementCollection = WebBrowser1.Document.GetElementsByTagName("select")
            For Each curElement As HtmlElement In theElementCollection3
                Dim controlName As String = curElement.GetAttribute("name").ToString
                If controlName = "autosubscribe" Then
                    curElement.SetAttribute("Value", 0)
                End If
            Next

            Dim theWElementCollection4 As HtmlElementCollection = WebBrowser1.Document.GetElementsByTagName("input")
            For Each curElement As HtmlElement In theWElementCollection4
                Dim controlValue As String = curElement.GetAttribute("value").ToString
                If controlValue = "Save Changes" Then

                    curElement.InvokeMember("click")
                End If
            Next


    All the best,
    Paul
    Monday, May 12, 2008 6:20 AM
  • Your code looks OK.

    You need to debug (e.g. set breakpoint) this code snippet and determine which code in question. Or delay 2 seconds as below.

     

    Code Snippet

          ' Delay 2 seconds

          Thread.Sleep(2000)

           Dim theWElementCollection4 As HtmlElementCollection = WebBrowser1.Document.GetElementsByTagName("input")

            For Each curElement As HtmlElement In theWElementCollection4

                Dim controlValue As String = curElement.GetAttribute("value").ToString

                If controlValue = "Save Changes" Then

     

                    curElement.InvokeMember("click")

                End If

            Next

     

     

    Monday, May 12, 2008 8:49 AM
  • Thanks Martin,

    I was unable to find the issue and the Thread.Sleep didn't solve it, however I put a message box in before the code and that seems to have fixed it.

    All the best,
    Paul
    Monday, May 12, 2008 11:08 AM
  • Hi!

    How I can populate this:

    <input class="input" type="text" name="user" size="30" MAXLENGTH="30">

    <input class="input" type="password" name="pass" size="30" MAXLENGTH="30">


    There is no value...

    Thanks!!!


    Friday, August 22, 2008 9:34 PM
  • Martin,
    This works very well, Thank you.
    Now I have automatically logged on and the home page is displayed how do I get the program to automatically select one of several short cuts on the home page and subsequently navigate to that page. (And beyond)
    For example, on the home page a short cut might be "Mail". When this is clicked  another log in page opens that needs a different username and password. Then when the login is clicked my company mail appears.
    Regards
    Wednesday, August 12, 2009 5:52 AM
  • Please check this FAQ for detailed instruction.
    FAQ: How do I automate a web page (e.g. retrieve page text, login website, search, click button or hyperlink) via WebBrowser?
    http://social.msdn.microsoft.com/Forums/en-US/vbgeneral/thread/23dfc3f9-3274-4929-8dc5-9ba543f4911d
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    Welcome to the All-In-One Code Framework! If you have any feedback on our support, please contact msdnmg@microsoft.com
    Thursday, August 13, 2009 4:47 AM
  • Hi Martin
    I have followed this thread and also been reading documentation, I also am trying to utilize Autologin on my personal App.
    My Usernames and Passwords are stored in a Local Database and are being displayed in Textboxes.  For the Time being I use a couple of buttons to copy and paste login information at the sites I visit.  I have been trying to find a solution for some time now but have not been able to implement anything on my own.
    I want to fire the login event with a button "Quick Login"
    If I'm understanding what I read them my Code should look something like this
    Private Sub btnQuickLogin_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles btnQuickLogin.Click
            WebBrowser1.Navigate(lblLinkClick.Text)
    End Sub
    
    Private Sub WebBrowser1_DocumentCompleted(ByVal sender As System.Object, ByVal e As System.Windows.Forms.WebBrowserDocumentCompletedEventArgs) Handles WebBrowser1.DocumentCompleted
    
            Dim theElementCollection As HtmlElementCollection = WebBrowser1.Document.GetElementsByTagName("input")
    
            For Each curElement As HtmlElement In theElementCollection
    
                Dim controlName As String = curElement.GetAttribute("name").ToString
    
                If controlName = "username" Then
    
                    curElement.SetAttribute("Value", tbUser.Text)
    
                ElseIf controlName = "password" Then
    
                    curElement.SetAttribute("Value", tbPass.Text)
    
                End If
    
            Next
    
            Dim theWElementCollection As HtmlElementCollection = WebBrowser1.Document.GetElementsByTagName("input")
    
            For Each curElement As HtmlElement In theWElementCollection
    
                If curElement.GetAttribute("value").Equals("Login") Then
    
                    curElement.InvokeMember("click")
    
                End If
    
            Next
    
    End Sub
    I would greatly appreciate any help. 

    Sunday, December 20, 2009 12:18 PM
  • Hi ALL,

     

    I am trying to login and go thru some protected forms using webrequest method = post. But, for some reason, i am re-directed back to the login page.

     

    Below is the code:

    // button click event
        protected void Button1_Click(object sender, EventArgs e)
        {
            CookieContainer cookieJar = new CookieContainer();
            string results = string.Empty;
            string form_post_data = string.Empty;
            string username = "test";
            string password = "test";
            string temp = EncodeTo64(password);
            string temp2 = HttpUtility.UrlEncode(temp);
            string LOGIN_URL = "https://10.10.10.10/login.htm?username=" + username + "&password=" + temp2 ;
            string MANAGE_REPORTS = "https://10.10.10.10/protetectedpage.htm";
                   
            Post(LOGIN_URL, ref cookieJar,  out results, "none");
            Post(LOGIN_URL, ref cookieJar, out results, form_post_data);
            Post(MANAGE_REPORTS, ref cookieJar, out results, form_post_data);
            test.InnerHtml = results;
          
           
        }

    // actual function

      public bool Post(string url, ref CookieContainer cookieJar,  out string result, string form_post_data )
        {
           
            string[] str = url.Split(new char[] { '?' });
            result = string.Empty;
            // Creat the cookie           
            Cookie ck1 = new Cookie("areCookiesEnabled", "val");
            ck1.Domain = "10.10.10.10"; // dummy domain
           
            url = str[0];
            if (str.Length == 2)
            form_post_data = str[1];       
            byte [] data;

            HttpRequestCachePolicy noCachePolicy = new HttpRequestCachePolicy(HttpRequestCacheLevel.BypassCache);
            HttpWebRequest myRequest;
            //get response from request
            HttpWebResponse objResponse;
            if (form_post_data == "none")
            {
                myRequest = WebRequest.Create(url) as HttpWebRequest;
                myRequest.Method = "POST";
                myRequest.Accept = "image/gif, image/jpeg, image/pjpeg, image/pjpeg, application/xaml+xml, application/vnd.ms-xpsdocument, application/x-ms-xbap, application/x-ms-application, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, application/x-shockwave-flash, */*";
                myRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; MS-RTC LM 8; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729) Paros/3.2.13";
                //myRequest.ContentType = "application x-www-form-urlencoded";
                myRequest.KeepAlive = true;
                myRequest.Referer = "https://10.10.10.10/login.htm?mode=Request"; // dummy ip address
                myRequest.CachePolicy = noCachePolicy;
                myRequest.CookieContainer = cookieJar;
                myRequest.AllowAutoRedirect = true;

                System.Net.ServicePointManager.ServerCertificateValidationCallback +=
                  delegate(object sender, System.Security.Cryptography.X509Certificates.X509Certificate certificate,
                           System.Security.Cryptography.X509Certificates.X509Chain chain,
                           System.Net.Security.SslPolicyErrors sslPolicyErrors)
                  {
                      return true; // **** Always accept
                  };       
                objResponse = (HttpWebResponse)myRequest.GetResponse();
                StreamReader sr = new StreamReader(objResponse.GetResponseStream());
                result = sr.ReadToEnd();
                sr.Close();
                objResponse.Close();
                return true;
            }
            else
            {
                myRequest = WebRequest.Create(url) as HttpWebRequest;
                System.Text.ASCIIEncoding encoding = new ASCIIEncoding();
                data = encoding.GetBytes(form_post_data);

                System.Net.ServicePointManager.ServerCertificateValidationCallback +=
               delegate(object sender, System.Security.Cryptography.X509Certificates.X509Certificate certificate,
                        System.Security.Cryptography.X509Certificates.X509Chain chain,
                        System.Net.Security.SslPolicyErrors sslPolicyErrors)
               {
                   return true; // **** Always accept
               };

                myRequest.Method = "POST";
                //myRequest.ContentType = "application/x-www-form-urlencoded";       
                myRequest.Accept = "image/gif, image/jpeg, image/pjpeg, image/pjpeg, application/xaml+xml, application/vnd.ms-xpsdocument, application/x-ms-xbap, application/x-ms-application, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, application/x-shockwave-flash, */*";
                myRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; MS-RTC LM 8; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729) Paros/3.2.13";
                myRequest.ContentType = "application x-www-form-urlencoded";
                myRequest.KeepAlive = true;
                myRequest.Referer = "https://10.10.10.10/login.htm?mode=Request";
                myRequest.ContentLength = data.Length;
                myRequest.CachePolicy = noCachePolicy;
                myRequest.AllowAutoRedirect = true;
                myRequest.CookieContainer = cookieJar;
                cookieJar.Add(ck1);
                Stream newStream = myRequest.GetRequestStream();
                newStream.Write(data, 0, data.Length);
                newStream.Close();

                //get response from request
                objResponse = (HttpWebResponse)myRequest.GetResponse();

                StreamReader sr;

                sr = new StreamReader(objResponse.GetResponseStream());
                result = sr.ReadToEnd();
                sr.Close();
                objResponse.Close();
            }
           
            return true;

        }

     

    Thanks in advance

    Tuesday, April 6, 2010 5:53 PM
  • very helpful

    Varun Kumar
    Tuesday, April 19, 2011 11:51 AM
  • Hi Martin,

    How can you accomplish this running on the browser instead of using a WebBrowser control inside of a form?

    I want to open and login to a website in the browser. Also, if you can provide a thread to follow about how to download files from web automatically using VB.net would be appreciatted.

    I know it's been a long 4 years since you published this solution. Hope you are still alive :)

    Thanks.

    Friday, December 2, 2011 7:40 PM