The underlying connection was closed: An unexpected error occurred on a send. Security certificate issue. RRS feed

  • Question

  • I am trying to write a simple web crawler using C#.

    The program loads the seed url, downloads all assets, extracts all <a> tags form the html and performs previous steps for each extracted url.

    Behind the scenes the app uses HttpWebRequest class.

    The following is a part of code I use for downloading resources(simplified)

    public class Downloader { static Downloader() { ServicePointManager.ServerCertificateValidationCallback = CertValidate; } static bool CertValidate(object sender, System.Security.Cryptography.X509Certificates.X509Certificate target, System.Security.Cryptography.X509Certificates.X509Chain chain, System.Net.Security.SslPolicyErrors errors) { return true; } public byte[] DownloadResource(string url) { var request = (System.Net.HttpWebRequest)WebRequest.Create(url); /*

    Set user agent, accept, encoding etc...

    */ using (var response = request.GetResponse()) { using (var responseStream = response.GetResponseStream()) { using(var memoryStream = new MemoryStream()) { responseStream.CopyTo(memoryStream); return memoryStream.ToArray(); } } } } }

    However, if I try to download from a url, which I cannot paste, since Microsoft was not able to verify my account,

    I get an exception:

    "The underlying connection was closed: An unexpected error occurred on a send."

    I tried to modify the code in the static constructor and appended the following line:

    ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;

    Everything worked well and the resource was successfully downloaded. However, when I run the application for a while, it crashes randomly. Sometimes it works for 30 minutes and sometimes for 4 hours.   The event log shows something like this:

    Log Name: Application
    Source: Application Error
    Event ID: 1000
    Task Category: (100)
    Level: Error
    Keywords: Classic
    User: N/A
    Computer: pc
    Faulting application name: myApp.exe, version:, time stamp: 0x57693286
    Faulting module name: unknown, version:, time stamp: 0x00000000
    Exception code: 0xc0000005

    Also, I'm getting errors while downloading files from secure servers, which do not support tls1.2. 

    What is the correct way of handling the security protocol?

    .NET version 4.5.2

    OS: Win10, 8.1

    • Edited by VahagnGile Tuesday, June 28, 2016 3:03 PM
    Tuesday, June 28, 2016 2:53 PM

All replies

  • Hi VahagnGile,

    >>"What is the correct way of handling the security protocol?"

    You could make your application support all the security protocols. Code below is for your reference.

    ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12
        | SecurityProtocolType.Tls11
        | SecurityProtocolType.Tls
        | SecurityProtocolType.Ssl3;
    Best Regards,
    Li Wang

    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Wednesday, June 29, 2016 8:28 AM
  • Hi,

    thanks for the response.

    I tried it. The result is the same. The app crashes randomly and I can't catch the exception. The error seems to occur in clr.exe(sometimes the windows event viewer shows that error occurred there and the exception code is again 0xc0000005). I have installed log4net from nuget and tried to log the exception with global exception handler. The log is always empty.


    • Edited by VahagnGile Wednesday, June 29, 2016 2:59 PM
    Wednesday, June 29, 2016 2:57 PM
  • Added more verbose logging. Just before the application crash I get the following error:

    Unable to read data from the transport connection: Overlapped I/O operation is in progress.

    Thursday, June 30, 2016 10:03 AM