best way to foudn all if webbrowser is totally ready

Question

I am trying to automate saving data from browsing some websites.

 I am having great difficulty to judge whether the webbrowser has completed all frames and becomes ready.  I tried matching download complete against beforenavigate2,...etc as well as trying to use webBrowser.ReadyState

 

Answer 1

Assuming you are using the VB Webbroswer control....Use the "DocumentCompleted" Event of the webbrowser control..if this is an asp app then use the page loaded event

Answer 2

unfortunately some websites cause DocumentCompleted to be rasied more than once. esp for those with frames

Answer 3

    Private Sub wb_DocumentComplete(ByVal pDisp As Object, ByRef URL As Object) Handles wb.DocumentComplete

 

        if pDisp is wb.Parent then document_complete = True       

 

    End Sub

 

Took me a while to find it out, it is nowhere in the documentation.

 

If the page has frames and is loading for the first time the above is good.

 

If you are reloading just one frame then

 

While wb.busy

  Thread.Sleep(100)

End While

 

will do

 

 

 

When loading for the first time a page with frames in the main module I use:

 

document_complete = False

 

While Not document_complete

  While wb.busy

    Thread.Sleep(100)

  End While

End While

 

 

Let me know your experience.

 

See my prior posts on this topic

Answer 4

thank you for sharing your experience

Before seeing your reply, I dug further into various documentation and by experimentation came up with the following:

    Sub address_Changed(ByVal sender As Object, ByVal e As System.EventArgs) Handles cboAddress.SelectedIndexChanged
        Dim o As Object
        Dim url As String
        url = Me.cboAddress.SelectedItem.ToString
        docCptdCnt = 0
        WebBrowser1.Navigate(url, o, o, o)
    End Sub

 Public Overloads Sub webBrowser1_DocumentCompleted(ByVal sender As Object, ByVal e As WebBrowserDocumentCompletedEventArgs) Handles WebBrowser1.DocumentCompleted
        If WebBrowser1.IsBusy Then Return ' <=== tihs is an important test
        If (WebBrowser1.ReadyState = WebBrowserReadyState.Complete) Then ' <== so is this important critiera
            Dim strBuf As String = ""
            docCptdCnt += 1
            strBuf = strBuf & "docCptdCnt=" & docCptdCnt.ToString & vbCrLf & " DocumentCompleted" & vbCrLf & "Busy?:" & WebBrowser1.IsBusy _
                & ", readystate=" & Me.WebBrowser1.ReadyState.ToString
            '"docCptdCnt=" & docCptdCnt.ToString & vbCrLf & " DocumentCompleted"
            TextBox1.Text = strBuf
            Try
                Dim strhtml As String = getdocHtml()

                'MsgBox(strhtml.Substring(1, 4048), MsgBoxStyle.Information, "html")
                'rtb1.setcolor(
                'rtb1.Rtf = strhtml 'strBuf
                'rtb1.Show()
            Catch 'ex As Exception
            End Try

        End If
    End Sub

I tired 4 websites some with frames some without. they all trigger the above event handler only once as attested to by the docCptdCnt  shows up int the the textbox as only one each time

 

Answer 5

Hi,

 

I have a keen interest in this issue.

 

My first comment is that your routine seems too complicated for something that should be very straight forward. But if it works it may be worth digging into it.

I cannot quite understand all that you do. Can you send the full code or explain a bit further.

 

In your opinion, what is the difference between wb.busy and wb.readystate=webbrowser.readystate.complete?

Also, you use dim as object and you get the whole html doc as a string, which usually takes too much time and resources.

 

 

 

 

Answer 6

Hello:

Has anyone figured out the solution to the above?

I have tried to use the documentcompleted event but it fires numerous times for the web page.  The Microsoft documentation says it should only fire once when the document is done.  But it fires for all IFRAMES on the page, which many sites have.

Thanks for any help you can give...this problem needs solved to make the webbrowser control usable within .NET 2.0

Joe

NOTE: anyone replying please note the documentcompleted event IS NOT the same as the documentcomplete event used before .NET 2.0.  The documentcompleted event is different.

Answer 7

have you tried any of the solutions above?

Answer 8

when busy, the browser still has outstanding tasks possibly other frames or some additional rendering I think, while the readystate  complete does not guarantee not isbusy.

 

 

all I know is that the code works and I had trouble getting the pdisp thing to work.  I don't care exactly what  is the difference readystate between isbusy.

 

The code sample provided with get Html and msgbox just to confirm that I got the page and yes, don't use that  on messagebox for large html page.  it may hang.  however instead of  displaying to msg but save as file takes very little time for me anyway ( AMD X64 3700+ with 2 GB ram and SATA Raid 5 even for  1 meg or two of html)

 

So if  you find the suggestion of pdisp simple and can get it work, be my guest.  Meanwhile I stick to what I found that is working and just too little  to change the working code.  I encapsulated  the web browser into a user control and that will be consumed by various application

Answer 9

oh, btw,   I have already provide the complete working code sample related to documentcompleted

you only need to look at the webBrowser1_DocumentCompleted sub

keep in mind that webBrowser1 is the name of the webbrowser control

  you just put in your code to do whatever you want instead of the block of code below

           Dim strBuf As String = ""
            docCptdCnt += 1
            strBuf = strBuf & "docCptdCnt=" & docCptdCnt.ToString & vbCrLf & " DocumentCompleted" & vbCrLf & "Busy?:" & WebBrowser1.IsBusy _
                & ", readystate=" & Me.WebBrowser1.ReadyState.ToString
            '"docCptdCnt=" & docCptdCnt.ToString & vbCrLf & " DocumentCompleted"
            TextBox1.Text = strBuf
            Try
                Dim strhtml As String = getdocHtml()

                'MsgBox(strhtml.Substring(1, 4048), MsgBoxStyle.Information, "html")
                'rtb1.setcolor(
                'rtb1.Rtf = strhtml 'strBuf
                'rtb1.Show()
            Catch 'ex As Exception
            End Try

For more detail go look up vb studio help on readystate, isbusy.  WebBrowserReadyState

Good luck.  you are welcome to post back if my oce breaks on what or the pdisp code

 

Answer 10

I just find the documentation very poor in detail.

One is just in the dark, playing at random and hoping for the best, which ends up in a waste of time most of the times.

 

For example, I have found out that the wbbrowser.busy works fine for one frame pages. However, when you use Public Withevents wbbrowser then the wbbrowser.busy does not pick up the full loading of a one frame page, for some pages. I do not understand why this is the case, and, certainly, it is not documented in the help pages. It resembles more of an erratic behavior, which makes any real attempt to write good code about this too much of a guess work.

 

On top of that, I have noticed how the help pages are not directly applicable to VBE in many occasions. This is not a serious tool.

 

It is not admissible that several people are trying to determine when any web page is fully loaded and we are still concatenating strings at best.

 

The MSDN example in C++ seems to be the type of solid programming that is needed, with full control of the pDisp variable.

Answer 11

I found this solution...and it is easy and it works...

Thought I would share it with you all.

Joe

PS This applies to the NET 2.0 webbrowser event "DocumentCompleted".  For NET 1 and VB6 the code posted by the previous poster (using pdisp) in the "DocumentComplete" event does the job.

---

Private Sub wb_DocumentCompleted(ByVal sender As Object, ByVal e As System.Windows.Forms.WebBrowserDocumentCompletedEventArgs) Handles wb.DocumentCompleted

        If e.Url.ToString = wb.Url.ToString Then
             'web page is done loading - do stuff here

            Dostuff()

        End If

End Sub

---

Answer 12

Hi, thank you for sharing. It looks promising.

 

Can you please post the full code of an example. I have tried without success.

Thanks,

 

Antonio

Answer 13

joep's solution works by an large.  however, I ran into sites that fails that.

my example solution is a bit more complex but it works for all the sites that I am using. I don't claim that is it will always work 100% but it is good enough and is not overly complex.  go back to review previous thread and adapt to your own code wb was th name of the .net 2 webbrowser on the form. I will not be monitoring this thread. good luck

Answer 14

I am adding this since people are looking for this solution. 

 

Basically neither solutions work.  The website URL is: http://www.glguitars.com/

 

The first time the events are tripped: ??regular DocumentCompleted?? means a console write with no checking

*** DocumentCompleted***  is  Joepg solution and %%DocumentCompleted%% is gg1's solution

 

Both do not work on certain Frame sites (see below)  They are more acurate than what DocumentCompleted provides but they both fire twice at 13:40 and finally at 14:41:699

 

System.Windows.Forms.WebBrowser

ProgressChanged time =13:39:830

ProgressChanged time =13:39:830

DocumentTitleChanged time =13:39:830

ProgressChanged time =13:39:830

'????????? Regular DocumentCompleted ????????? time =13:40:10

********** DocumentCompleted ********* time =13:40:41

%%%%%%%%%%%%%% DocumentCompleted %%%%%%%%%% time =13:40:391

ProgressChanged time =13:41:573

StatusTextChanged time =14:5:16

System.Windows.Forms.WebBrowser

StatusTextChanged time =14:5:16

 

AT END-----

StatusTextChanged time =14:41:629

System.Windows.Forms.WebBrowser

ProgressChanged time =14:41:629

'????????? Regular DocumentCompleted ????????? time =14:41:629

StatusTextChanged time =14:41:699

System.Windows.Forms.WebBrowser

ProgressChanged time =14:41:699

ProgressChanged time =14:41:699

'????????? Regular DocumentCompleted ????????? time =14:41:699

********** DocumentCompleted ********* time =14:41:699

%%%%%%%%%%%%%% DocumentCompleted %%%%%%%%%% time =14:41:699

StatusTextChanged time =14:56:621

System.Windows.Forms.WebBrowser

StatusTextChanged time =14:56:621

System.Windows.Forms.WebBrowser

StatusTextChanged time =14:56:651

System.Windows.Forms.WebBrowser

StatusTextChanged time =14:56:711

System.Windows.Forms.WebBrowser

StatusTextChanged time =14:56:721

System.Windows.Forms.WebBrowser

StatusTextChanged time =14:56:741

System.Windows.Forms.WebBrowser

StatusTextChanged time =15:15:648

System.Windows.Forms.WebBrowser

StatusTextChanged time =15:15:648

System.Windows.Forms.WebBrowser

StatusTextChanged time =15:15:668

System.Windows.Forms.WebBrowser

StatusTextChanged time =15:15:668

System.Windows.Forms.WebBrowser

StatusTextChanged time =15:15:678

System.Windows.Forms.WebBrowser

StatusTextChanged time =15:15:678

System.Windows.Forms.WebBrowser

StatusTextChanged time =15:15:698

System.Windows.Forms.WebBrowser

StatusTextChanged time =15:15:968

System.Windows.Forms.WebBrowser

StatusTextChanged time =15:15:988

System.Windows.Forms.WebBrowser

StatusTextChanged time =15:15:999

System.Windows.Forms.WebBrowser

StatusTextChanged time =15:16:19

System.Windows.Forms.WebBrowser

StatusTextChanged time =15:16:49

System.Windows.Forms.WebBrowser

StatusTextChanged time =15:17:190

System.Windows.Forms.WebBrowser

StatusTextChanged time =15:17:190

System.Windows.Forms.WebBrowser

HandleDestroyed time =15:17:190

ControlRemoved time =15:17:220

Answer 15

let us keep this thread alive until we solve this issue

Answer 16

so what is your solution?

 

I am afraid , I can't help you with that. The solution posted earlier continue to work for all the website I have to deal with so far.  as previously posted.

 

 I don't claim the solution works for all nor the solution will alwys work regardless of the .net sdk version or upgrade.  The solution just worked for the sites I have to deal with for .net 2 at the time and even now after the security update.  btw the website I deal with have multi-frames  none of the webiste's frame were served/hosted on different site than the man frame.  that is for any particular webiste, all the frames on teh pages were served directly formt he website itself and does not  use cross site service

 

 

As for the website that caused trouble for you, I have not the time to investigate as it is out of scope for my project and I am fighting another .net related trouble.  sorry.

 

 

However I definitely like to know your solution

Answer 17

Don't know if this helps anyone with .NET. Don't know if this helps on any of the sites others are working with either.

 

I'm coding an app in vb6. Reason being that this is a copier (Bizhub rules!) installation utility which needs to work on any flavor xp/vista I might encounter on the road which may or may not have .NET installed.

 

I've found the following code works in vb6 for pages with multiple frames served up from a MultiFunction Device. (setting up SMB scanning "automagically"!)

 

Just call the following function after the webpage navigates.

 

Private Sub WaitWebPageLoaded(WebBrowserControl As WebBrowser, TargetWebAddress As String)
    Do
        DoEvents
    Loop Until (Not WebBrowserControl.Busy) And (WebBrowserControl.ReadyState = READYSTATE_COMPLETE) And (WebBrowserControl.LocationURL = TargetWebAddress)
End Sub

Probably should be spiced up with a timeout, else you're in an endless loop.

 

Target web address should be what the browser responds with after the page is loaded, eg. http://www.mywebsite.com becomes http://www.mywebsite.com/index.html

 

Hope this helps someone!

Answer 18

 

Hello DigitalDragon,

 

Thank you for your solution, although I've had only a few minutes, so far, it works great!

 

It is simple, and, actually, it makes good sense, by incorporating the  WebBrowserControl.LocationURL = TargetWebAddress concept that had already being mentioned.

 

I'll let you know if it fails.

 

Regards,

 

 

Antonio

Answer 19

You Could say it this way and avoid the loop

Private Sub Wb1_DocumentCompleted(ByVal sender As Object, ByVal e As System.Windows.Forms.WebBrowserDocumentCompletedEventArgs) Handles wb1.DocumentCompleted

Console.WriteLine("E " & "Wb1_DocumentCompleted")

'This if statement get us closet to a real document complete

If Not wb1.IsBusy And (wb1.ReadyState = WebBrowserReadyState.Complete) And (e.Url.ToString = wb1.Url.ToString) Then

Answer 20

GregCost said:

You Could say it this way and avoid the loop

Private Sub Wb1_DocumentCompleted(ByVal sender As Object, ByVal e As System.Windows.Forms.WebBrowserDocumentCompletedEventArgs) Handles wb1.DocumentCompleted

Console.WriteLine("E " & "Wb1_DocumentCompleted")

'This if statement get us closet to a real document complete

If Not wb1.IsBusy And (wb1.ReadyState = WebBrowserReadyState.Complete) And (e.Url.ToString = wb1.Url.ToString) Then

Intereting topic. 

Do you think the control is buffering any of the info?  I have written similar code in C# to implement a URLProxy, which displays a page to the user that basically says "loading" until the link is finished loading.  If I navigate to pages that I have already visited, I do not get the "loading" page.  Sounds like to me that behind the scenes, something is getting cached locally.

Rudedog   =8^D

---

Mark the best replies as answers. "Fooling computers since 1971."

Answer 21

GregCost said:

You Could say it this way and avoid the loop

Private Sub Wb1_DocumentCompleted(ByVal sender As Object, ByVal e As System.Windows.Forms.WebBrowserDocumentCompletedEventArgs) Handles wb1.DocumentCompleted

Console.WriteLine("E " & "Wb1_DocumentCompleted")

'This if statement get us closet to a real document complete

If Not wb1.IsBusy And (wb1.ReadyState = WebBrowserReadyState.Complete) And (e.Url.ToString = wb1.Url.ToString) Then

I had combinations of these conditions but not all three.  What I have noticed is problems with sites using client side scripting to render pages.  The html triggers the documentcomplete but the javascript has not finished.   In some cases, I have to loop the navigate event until I get a response.  Only problem is the loop might hang so I will likely have to add a timeout.

One possible solution which I would great regret is to use the activex control but it appears it already resolved this issue.  If the webbrowser control is a wrapper, why not wrap all the functionality ?   Sounds backward to me.

I'll give this option a try since I have tried just about everything else.   Very annoying issue for what appears to be a simple problem.