none
WebBrowser.Navigate v. Net.HttpWebRequest.GetResponse RRS feed

  • Question

  • Not sure what I'm doing wrong.  I have been trying to get this to work for some time.

            Const webReqTMO As Integer = 1000
            'this works
            ' Const Hpage As String = "https://revisor.mo.gov/main/OneSection.aspx?section=1.025&bid=44"
            'this fails but shouldn't
            Const Hpage As String = "https://revisor.mo.gov/main/Home.aspx"
            Dim WEBreqst As Net.HttpWebRequest = Nothing
            Dim WEBResp As Net.HttpWebResponse = Nothing
            Dim siteUri As New Uri(Hpage)
    
            WEBreqst = CType(Net.HttpWebRequest.Create(siteUri), Net.HttpWebRequest) 'build request
            WEBreqst.Timeout = webReqTMO
            WEBreqst.KeepAlive = False
    
            Try
                WEBResp = CType(WEBreqst.GetResponse(), Net.HttpWebResponse) 'get the page
                Using reader As IO.StreamReader = New IO.StreamReader(WEBResp.GetResponseStream)
                    reader.ReadToEnd()
                End Using
            Catch ex As Exception
                Stop
            Finally
                Try
                    If WEBResp IsNot Nothing Then
                        WEBResp.Close()
                        WEBResp.Dispose()
                        WEBResp = Nothing
                    End If
                Catch nex As Exception
                End Try
            End Try
    

    I get 404 errors which doesn't make sense, the page exists.  To verify this I added a WebBrowser control and do this,

    WebBrowser1.Navigate("https://revisor.mo.gov/main/Home.aspx")

    Works like a charm.  I'm so confused!


    Search Documentation

    SerialPort Info

    Multics - An OS ahead of its time.

     "Those who use Application.DoEvents have no idea what it does

        and those who know what it does never use it."    former MSDN User JohnWein

    Wednesday, February 26, 2020 9:33 PM

Answers

  • Any idea why it is happening?  Your code does work BTW.  Thx.

    I also noticed that you can control click the link in the IDE and that also works.


    Search Documentation

    SerialPort Info

    Multics - An OS ahead of its time.

     "Those who use Application.DoEvents have no idea what it does

        and those who know what it does never use it."    former MSDN User JohnWein


    Fars as I can tell as I used "funky" before seems its a  WebExceptionStatus.ProtocolError. My hunch is this may be something to do with IIS.

    If you really want to play through this the following might help, minor mods to the original code.

    Public Function GetResponse(ByRef url As String) As String
        Dim req As WebRequest = WebRequest.Create(url)
    
        Try
            Using response As WebResponse = req.GetResponse()
                '
            End Using
    
        Catch e As WebException
            If e.Status = WebExceptionStatus.ProtocolError Then
                Dim resp As HttpWebResponse = TryCast(e.Response, HttpWebResponse)
                Dim response As WebResponse = e.Response
    
                If resp IsNot Nothing AndAlso resp.StatusCode = HttpStatusCode.NotFound Then
                    Console.WriteLine("We will hit here but can continue with -> Not found")
                Else
                    Throw
                End If
                Using reader As New StreamReader(response.GetResponseStream())
                    Dim text As String = reader.ReadToEnd()
                    Return text
                End Using
            Else
                Throw
            End If
        End Try
    
        Return ""
    
    End Function


    Please remember to mark the replies as answers if they help and unmarked them if they provide no help, this will help others who are looking for solutions to the same or similar problem. Contact via my Twitter (Karen Payne) or Facebook (Karen Payne) via my MSDN profile but will not answer coding question on either.

    NuGet BaseConnectionLibrary for database connections.

    StackOverFlow
    profile for Karen Payne on Stack Exchange

    • Marked as answer by dbasnett Thursday, February 27, 2020 3:11 PM
    Thursday, February 27, 2020 2:18 PM
    Moderator
  • I found that by adding this

      WEBreqst.UserAgent = ".NET Framework Test Client"

    before getting the response fixed it.  Damn.


    Search Documentation

    SerialPort Info

    Multics - An OS ahead of its time.

     "Those who use Application.DoEvents have no idea what it does

        and those who know what it does never use it."    former MSDN User JohnWein

    • Marked as answer by dbasnett Thursday, February 27, 2020 3:36 PM
    Thursday, February 27, 2020 3:36 PM

All replies

  • Try this (it's funky) where an exception will be thrown then we end up in the catch which at least for me using your url I get the page back. Not something I would want in production as I'd consider this fragile.

    Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
        Console.WriteLine(GetResponse("https://revisor.mo.gov/main/Home.aspx"))
    End Sub
    Public Function GetResponse(ByRef url As String) As String
    
        Dim req As WebRequest = WebRequest.Create(url)
    
        Try
            Using response As WebResponse = req.GetResponse()
                '
            End Using
    
        Catch e As WebException
    
            Dim response As WebResponse = e.Response
    
            Using reader As New StreamReader(response.GetResponseStream())
                Dim text As String = reader.ReadToEnd()
                Return text
            End Using
    
        End Try
    
        Return ""
    
    End Function


    Please remember to mark the replies as answers if they help and unmarked them if they provide no help, this will help others who are looking for solutions to the same or similar problem. Contact via my Twitter (Karen Payne) or Facebook (Karen Payne) via my MSDN profile but will not answer coding question on either.

    NuGet BaseConnectionLibrary for database connections.

    StackOverFlow
    profile for Karen Payne on Stack Exchange

    Thursday, February 27, 2020 12:49 AM
    Moderator
  • Any idea why it is happening?  Your code does work BTW.  Thx.

    I also noticed that you can control click the link in the IDE and that also works.


    Search Documentation

    SerialPort Info

    Multics - An OS ahead of its time.

     "Those who use Application.DoEvents have no idea what it does

        and those who know what it does never use it."    former MSDN User JohnWein


    • Edited by dbasnett Thursday, February 27, 2020 2:13 PM
    Thursday, February 27, 2020 2:01 PM
  • Any idea why it is happening?  Your code does work BTW.  Thx.

    I also noticed that you can control click the link in the IDE and that also works.


    Search Documentation

    SerialPort Info

    Multics - An OS ahead of its time.

     "Those who use Application.DoEvents have no idea what it does

        and those who know what it does never use it."    former MSDN User JohnWein


    Fars as I can tell as I used "funky" before seems its a  WebExceptionStatus.ProtocolError. My hunch is this may be something to do with IIS.

    If you really want to play through this the following might help, minor mods to the original code.

    Public Function GetResponse(ByRef url As String) As String
        Dim req As WebRequest = WebRequest.Create(url)
    
        Try
            Using response As WebResponse = req.GetResponse()
                '
            End Using
    
        Catch e As WebException
            If e.Status = WebExceptionStatus.ProtocolError Then
                Dim resp As HttpWebResponse = TryCast(e.Response, HttpWebResponse)
                Dim response As WebResponse = e.Response
    
                If resp IsNot Nothing AndAlso resp.StatusCode = HttpStatusCode.NotFound Then
                    Console.WriteLine("We will hit here but can continue with -> Not found")
                Else
                    Throw
                End If
                Using reader As New StreamReader(response.GetResponseStream())
                    Dim text As String = reader.ReadToEnd()
                    Return text
                End Using
            Else
                Throw
            End If
        End Try
    
        Return ""
    
    End Function


    Please remember to mark the replies as answers if they help and unmarked them if they provide no help, this will help others who are looking for solutions to the same or similar problem. Contact via my Twitter (Karen Payne) or Facebook (Karen Payne) via my MSDN profile but will not answer coding question on either.

    NuGet BaseConnectionLibrary for database connections.

    StackOverFlow
    profile for Karen Payne on Stack Exchange

    • Marked as answer by dbasnett Thursday, February 27, 2020 3:11 PM
    Thursday, February 27, 2020 2:18 PM
    Moderator
  • I spent some time looking at IIS yesterday and couldn't see anything.  It is really odd because that link is to the home page and it occurs on the site numerous places.

    Search Documentation

    SerialPort Info

    Multics - An OS ahead of its time.

     "Those who use Application.DoEvents have no idea what it does

        and those who know what it does never use it."    former MSDN User JohnWein

    Thursday, February 27, 2020 2:34 PM
  • I spent some time looking at IIS yesterday and couldn't see anything.  It is really odd because that link is to the home page and it occurs on the site numerous places.

    Search Documentation

    SerialPort Info

    Multics - An OS ahead of its time.

     "Those who use Application.DoEvents have no idea what it does

        and those who know what it does never use it."    former MSDN User JohnWein

    If you search the web, tons of developers had this issue and resolve it using similar code I provided. So you chances of resolving this another way are slim to null.

    Please remember to mark the replies as answers if they help and unmarked them if they provide no help, this will help others who are looking for solutions to the same or similar problem. Contact via my Twitter (Karen Payne) or Facebook (Karen Payne) via my MSDN profile but will not answer coding question on either.

    NuGet BaseConnectionLibrary for database connections.

    StackOverFlow
    profile for Karen Payne on Stack Exchange

    Thursday, February 27, 2020 3:02 PM
    Moderator
  • What did you search for, I couldn't find similar issues.

    Thanks for all of your help.


    Search Documentation

    SerialPort Info

    Multics - An OS ahead of its time.

     "Those who use Application.DoEvents have no idea what it does

        and those who know what it does never use it."    former MSDN User JohnWein

    Thursday, February 27, 2020 3:11 PM
  • What did you search for, I couldn't find similar issues.

    Thanks for all of your help.


    Search Documentation

    SerialPort Info

    Multics - An OS ahead of its time.

     "Those who use Application.DoEvents have no idea what it does

        and those who know what it does never use it."    former MSDN User JohnWein

    For vb.net issues or C# issues I always search for C# then translate, also since my day job is a web developer I think in that environment so I Googled "C# status code 404" then HttpWebRequest.GetResponse for 404. Came up with links like this one (I did this at home, now at work so no history).

    https://stackoverflow.com/questions/1949610/how-can-i-catch-a-404


    Please remember to mark the replies as answers if they help and unmarked them if they provide no help, this will help others who are looking for solutions to the same or similar problem. Contact via my Twitter (Karen Payne) or Facebook (Karen Payne) via my MSDN profile but will not answer coding question on either.

    NuGet BaseConnectionLibrary for database connections.

    StackOverFlow
    profile for Karen Payne on Stack Exchange

    Thursday, February 27, 2020 3:29 PM
    Moderator
  • I found that by adding this

      WEBreqst.UserAgent = ".NET Framework Test Client"

    before getting the response fixed it.  Damn.


    Search Documentation

    SerialPort Info

    Multics - An OS ahead of its time.

     "Those who use Application.DoEvents have no idea what it does

        and those who know what it does never use it."    former MSDN User JohnWein

    • Marked as answer by dbasnett Thursday, February 27, 2020 3:36 PM
    Thursday, February 27, 2020 3:36 PM
  • I found that by adding this

      WEBreqst.UserAgent = ".NET Framework Test Client"

    before getting the response fixed it.  Damn.


    Search Documentation

    SerialPort Info

    Multics - An OS ahead of its time.

     "Those who use Application.DoEvents have no idea what it does

        and those who know what it does never use it."    former MSDN User JohnWein

    I would say both solutions are like walking on egg shells as if something changes in the Framework this can break it and BTW I did come across a similar solution but discarded it.

    Please remember to mark the replies as answers if they help and unmarked them if they provide no help, this will help others who are looking for solutions to the same or similar problem. Contact via my Twitter (Karen Payne) or Facebook (Karen Payne) via my MSDN profile but will not answer coding question on either.

    NuGet BaseConnectionLibrary for database connections.

    StackOverFlow
    profile for Karen Payne on Stack Exchange

    Thursday, February 27, 2020 4:11 PM
    Moderator
  • Trying to find out if it is an IIS setting but no luck so far.

    Search Documentation

    SerialPort Info

    Multics - An OS ahead of its time.

     "Those who use Application.DoEvents have no idea what it does

        and those who know what it does never use it."    former MSDN User JohnWein

    Thursday, February 27, 2020 4:16 PM
  • the page exists

    Actually not. The HTML page that we get back does not exist until IIS processes the ASP.Net code. When we navigate to the page the ASP.Net code is executed but when we use the Net.HttpWebRequest class to request the file (I assume) it does not get the processing. And we know that we cannot get the aspx page (input) returned.

    I am nearly certain that the Net.HttpWebRequest class would not execute the JavaScript either when we get HTML files.



    Sam Hobbs
    SimpleSamples.Info

    Thursday, February 27, 2020 5:21 PM
  • the page exists

    Actually not. The HTML page that we get back does not exist until IIS processes the ASP.Net code. When we navigate to the page the ASP.Net code is executed but when we use the Net.HttpWebRequest class to request the file (I assume) it does not get the processing. And we know that we cannot get the aspx page (input) returned.

    I am nearly certain that the Net.HttpWebRequest class would not execute the JavaScript either when we get HTML files.



    Sam Hobbs
    SimpleSamples.Info

    What does this mean?  You lost me.  The home page does exist.

    https://revisor.mo.gov/main/Home.aspx


    Search Documentation

    SerialPort Info

    Multics - An OS ahead of its time.

     "Those who use Application.DoEvents have no idea what it does

        and those who know what it does never use it."    former MSDN User JohnWein

    Thursday, February 27, 2020 5:41 PM
  • Do you know what an aspx file looks like? It typically looks very different from the HTML that is sent to the browser. What you see in the browser is not the aspx file, actually. And web developers depend on the fact that the ASP.Net code in the aspx file is never sent to the client (you); for one thing, security depends on that.


    Sam Hobbs
    SimpleSamples.Info

    Thursday, February 27, 2020 6:35 PM
  • You obviously do not want to believe me. Since Karen answered the question, I will let her explain further if you need further explanation.


    Sam Hobbs
    SimpleSamples.Info

    Thursday, February 27, 2020 6:37 PM
  • Do you know what an aspx file looks like? It typically looks very different from the HTML that is sent to the browser. What you see in the browser is not the aspx file, actually. And web developers depend on the fact that the ASP.Net code in the aspx file is never sent to the client (you); for one thing, security depends on that.


    Sam Hobbs
    SimpleSamples.Info

    Since I wrote every aspx file on the site in question I might know what it looks like. 

    Did you see that the addition of setting the user-agent changed the behavior?  Did you see that the original behavior only occurred on the home page? 

    If you want to have an attitude you should be specific and correct.


    Search Documentation

    SerialPort Info

    Multics - An OS ahead of its time.

     "Those who use Application.DoEvents have no idea what it does

        and those who know what it does never use it."    former MSDN User JohnWein

    Thursday, February 27, 2020 7:03 PM
  • If anything I said is wrong then please state the facts. Instead you have been replying with sarcasm. I said I would end the matter and let Karen explain yet you continue to push. Okay, well go ahead and have the last word, I sure doubt it will accomplish anything for me to continue.

    I will unsubscribe from this discussion and won't see any reply.



    Sam Hobbs
    SimpleSamples.Info

    Friday, February 28, 2020 1:55 AM
  • If anything I said is wrong then please state the facts. Instead you have been replying with sarcasm. I said I would end the matter and let Karen explain yet you continue to push. Okay, well go ahead and have the last word, I sure doubt it will accomplish anything for me to continue.

    I will unsubscribe from this discussion and won't see any reply.



    Sam Hobbs
    SimpleSamples.Info

    OK

    Search Documentation

    SerialPort Info

    Multics - An OS ahead of its time.

     "Those who use Application.DoEvents have no idea what it does

        and those who know what it does never use it."    former MSDN User JohnWein

    Friday, February 28, 2020 1:50 PM