httpWebRequest.GetResponse(): The underlying connection was closed: The connection was closed unexpectedly.

Question

I keep getting a System.Net.WebException thrown, precisely from req.GetResponse() and not GetResponseStream(), with this message: "The underlying connection was closed: The connection was closed unexpectedly."

Would any of the .NET experts help me and point out where I may have screwed up here:

Imports System.IO
Imports System.Net

Public Class Form1

    Private Sub Form1_Load(sender As System.Object, e As System.EventArgs) Handles MyBase.Load

        Dim req As HttpWebRequest
        Dim reader As StreamReader
        Try
            req = WebRequest.Create("http://www.bootlegger.com/")
            req.Timeout = 100000
            req.KeepAlive = False
            req.AllowAutoRedirect = True
            reader = New StreamReader(req.GetResponse.GetResponseStream) ' <-- System.Net.WebException is thrown here, precisely from req.GetResponse() and not GetResponseStream(), with this message: "The underlying connection was closed: The connection was closed unexpectedly."
            Dim HTMLcontent As String = reader.ReadToEnd()
            reader.Close()
            req.Abort()
        Catch ex As Exception
            MsgBox(ex.Message)
        End Try

    End Sub

End Class

I have been searching several sites and forums and happened to run into several posts similar to my issue, but some are whether left without answers/solutions or a suggested solution wouldn't work for some reason in my case!

Things that might be put in consideration:

- The link I'm using works just fine from any browser and responds with the HTML content even when attempted to be read using notepad or any other text editor.

- The code works with no conflicts at all when that particular link gets replaced by any other.

- I really need that particular link to work in my code.

- I'm using .NET Framework 4.0 (32-bit mode) on a Windows 7 Home Premium 64-bit machine.

- Take a look at the trace log from the System.Net library below.

Thanks in advance!

__________________________System.Net trace log______________________________

System.Net Verbose: 0 : [7644] WebRequest::Create(http://www.bootlegger.com/)
System.Net Verbose: 0 : [7644] HttpWebRequest#33111870::HttpWebRequest(http://www.bootlegger.com/#1658570617)
System.Net Information: 0 : [7644] Current OS installation type is 'Client'.
System.Net Information: 0 : [7644] RAS supported: True
System.Net Verbose: 0 : [7644] Exiting HttpWebRequest#33111870::HttpWebRequest()
System.Net Verbose: 0 : [7644] Exiting WebRequest::Create()     -> HttpWebRequest#33111870
System.Net Verbose: 0 : [7644] HttpWebRequest#33111870::GetResponse()
System.Net Information: 0 : [7644] Associating HttpWebRequest#33111870 with ServicePoint#27806816
System.Net Information: 0 : [7644] Associating Connection#66337667 with HttpWebRequest#33111870
System.Net Information: 0 : [7644] Connection#66337667 - Created connection from 127.0.0.1:57758 to 72.52.4.239:80.
System.Net Information: 0 : [7644] Associating HttpWebRequest#33111870 with ConnectStream#35489797
System.Net Information: 0 : [7644] HttpWebRequest#33111870 - Request: GET / HTTP/1.1

System.Net Information: 0 : [7644] ConnectStream#35489797 - Sending headers
{
Host: www.bootlegger.com
Connection: Close
}.
System.Net Error: 0 : [7644] Exception in the HttpWebRequest#33111870:: - The underlying connection was closed: The connection was closed unexpectedly.
System.Net Information: 0 : [7644] Associating HttpWebRequest#33111870 with ServicePoint#27806816
System.Net Information: 0 : [7644] Associating Connection#65677972 with HttpWebRequest#33111870
System.Net Information: 0 : [7644] Connection#65677972 - Created connection from 127.0.0.1:57892 to 72.52.4.239:80.
System.Net Information: 0 : [7644] Associating HttpWebRequest#33111870 with ConnectStream#8442299
System.Net Information: 0 : [7644] HttpWebRequest#33111870 - Request: GET / HTTP/1.1

System.Net Information: 0 : [7644] ConnectStream#8442299 - Sending headers
{
Host: www.bootlegger.com
Connection: Close
}.
System.Net Error: 0 : [7644] Exception in the HttpWebRequest#33111870:: - The underlying connection was closed: The connection was closed unexpectedly.
System.Net Error: 0 : [7644] Exception in the HttpWebRequest#33111870::GetResponse - The underlying connection was closed: The connection was closed unexpectedly.

Answer 1

Based on the Exception the Server is closing the connection.

So there could be a couple of issues:

1. Your request is invalid
2. Try keep alive on because it might just try to close the connection after one request as HTTP version 1.1 will reuse the connection. I don't think this is but it could be.

I mean looking at the lines below you make the Get Request and then try to send the Send the Headers but the connection is already closed. It seems as if your request is invalid. There could be something on the server though that is closing the connection for some reason.

System.Net Information: 0 : [7644] HttpWebRequest#33111870 - Request: GET / HTTP/1.1

System.Net Information: 0 : [7644] ConnectStream#35489797 - Sending headers
{
Host: www.bootlegger.com
Connection: Close
}.

Thanks,

Brad

Answer 2

Thanks Brad for your reply, but FYI my previous versions of this code had req.KeepAlive=true while others didn't have that line at all, and yet encountered the same problem! I also tried using HTTP 1.0 but that didn't solve the problem either.

My curious question is, why other methods of retrieving the HTML content succeed when using text editors or browsers?! What's with .NET itself that keeps throwing that exception at me?!

Thanks!