none
Can I download a webpage programatically from a site that uses SSL? (see below) RRS feed

  • Question

  •  I tried to download a paper from a science magazine "cell.com".  The link was a PDF: https://www.cell.com/neuron/pdf/S0896-6273(13)00231-6.pdf.   It can be viewed in any browser without an error messages.  I tried downloading it programmatically using standard code, and got the following error message:  "The request was aborted: Could not create SSL/TLS secure channel."   So I modified my code (which still does not work) by cutting and pasting code that I don't understand from stack-overflow (see below).   Can anyone tell me if the task is doable?

    Public Shared Function DownloadFileUsingAgent(ByVal strURL As String, ByVal fullname As String, ByRef errMessage As String) As Boolean
            Dim retval As Boolean = True
            Using wcx As WebClient = New WebClient()
                wcx.Headers.Add("User-Agent: BrowseAndDownload")
                ServicePointManager.Expect100Continue = True
                ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12
                ServicePointManager.ServerCertificateValidationCallback = New RemoteCertificateValidationCallback(AddressOf ValidateRemoteCertificate)
                Try
                    wcx.DownloadFile(strURL, fullname)
                Catch ex As Exception
                    retval = False
                    errMessage = ex.Message
                End Try
              Threading.Thread.Sleep(500)
      
            End Using
            Return retval
        End Function
    


    Tuesday, July 17, 2018 8:36 AM

Answers

  • You can't do it not with some Windows desktop solution trying to be in a SSL session with some site using SSL not without a SSL certificate that was given to you from the site to install on your machine. 

    All browsers are SSL compliant out the gate with built-in SSL technology.  

    • Marked as answer by Gidmaestro Tuesday, July 17, 2018 6:33 PM
    Tuesday, July 17, 2018 9:24 AM

All replies

  • You can't do it not with some Windows desktop solution trying to be in a SSL session with some site using SSL not without a SSL certificate that was given to you from the site to install on your machine. 

    All browsers are SSL compliant out the gate with built-in SSL technology.  

    • Marked as answer by Gidmaestro Tuesday, July 17, 2018 6:33 PM
    Tuesday, July 17, 2018 9:24 AM
  • This works for me on Windows 10 :

    ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 Or SecurityProtocolType.Tls12
    Dim sRemoteFileName As String = "https://www.cell.com/neuron/pdf/S0896-6273(13)00231-6.pdf"
    Dim sLocalFilename As String = "e:\\testpdf.pdf"
    Dim webreq As System.Net.HttpWebRequest = System.Net.HttpWebRequest.Create(sRemoteFileName)
    Dim cookieContainer As CookieContainer = New CookieContainer
    webreq.CookieContainer = cookieContainer
    Using webresp As System.Net.WebResponse = webreq.GetResponse
        Using respStream As IO.Stream = webresp.GetResponseStream
            Using fs As New IO.FileStream(sLocalFilename, FileMode.Create, FileAccess.Write)
                Dim buffer(2047) As Byte
                Dim nRead As Integer
                Do
                    nRead = respStream.Read(buffer, 0, buffer.Length)
                    fs.Write(buffer, 0, nRead)
                Loop Until nRead = 0
                respStream.Close()
                fs.Flush()
                fs.Close()
            End Using
            respStream.Close()
        End Using
        webresp.Close()
    End Using

    Tuesday, July 17, 2018 9:51 AM