locked
WebClient.DownloadFile can't establish secure SSL/TLS channel RRS feed

  • Question

  • I cannot download a publicly available file from a government webserver using .Net framework 4.8 or .Net core 3.1. I am able to download the file using Chrome, Firefox, or curl.  

    WebClient.DownloadFile() worked for over a year running as an Azure app service. It ran successfully Sept. 11, 2020 and hasn't worked since. It no longer works on my Windows 10 Pro computer. I also tried HttpWebRequest/HttpWebResponse without luck. I suspect server or firewall rule is blocking the request due to missing Http header? I tried setting user-agent header and forcing the protocol to TLS v1.2 and TLS v1.3.  I also tried other ServicePointManager properties/overrides with no luck.

    WireShark shows client Hello sends TLSv1.2 cipher list. I matched several of these ciphers to the server's list, using SSLLabs server report. Server supports TLS v1.2 and TLS v1.3.  After Client Hello, Server responds with [ACK] and then [RST, ACK] and no Server Hello.

    I'm to the point I may create a c++ cli program and link to libcurl and call it from c# - all just to download a publicly available file.  See code for the url.

    private void DownloadTest()

    //ServicePointManager.Expect100Continue = true; 
    //ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12; 
    //ServicePointManager.ServerCertificateValidationCallback = (sender, cert, chain, sslPolicyErrors) => { return true; }; 

    try 

    using (var client = new WebClient()) 

    client.DownloadFile("https://cops.fas.gsa.gov/awards/awards_2021.csv", @"c:\awards_2021.csv"); 

    }
    catch (Exception ex) 

    MessageBox.Show(ex.ToString()); 

    }


    Wednesday, October 7, 2020 1:47 PM

Answers

  • Thank you for your help CoolDadTX and Costorix31!

    I installed a NuGet package called CurlThin that wraps curl.  Another package, CurlThin.Native simply contains dll's for libcurl and ssl.  It works with the government server for tls 1.2 and if I set my windows 10 registry to enable tls 1.3 it works with it too.  I haven't decrypted to see which cipher used for 1.2

    After installing both packages, you need to call CurlResources.Init() from your project code somewhere first.  This installs the needed dependency dll's and .crt file to your output directory.  You need to call this once for Debug and once for Release.  

    I first got a runtime error memory corrupted when trying to read the data stream. My machine is 64-bit and although I set my VS Solution Platform to build x64 configuration on my toolbar, VS did not build x64 until I set the Platform Target on the Build tab of project properties from Any CPU to x64.  

            public void DownloadFile(string url, string fileName)
            {
    
                // curl_global_init() with default flags.
                var global = CurlNative.Init();
    
                // curl_easy_init() to create easy handle.
                var easy = CurlNative.Easy.Init();
                try
                {
                    CurlNative.Easy.SetOpt(easy, CURLoption.URL, url);
    
                    // Added this statement to resolve SSL_CACERT return code. [Error]
                    CurlNative.Easy.SetOpt(easy, CURLoption.CAINFO, CurlResources.CaBundlePath);
    
                    //MessageBox.Show("Using MemoryStream");
                    var stream = new MemoryStream();
                    CurlNative.Easy.SetOpt(easy, CURLoption.WRITEFUNCTION, (data, size, nmemb, user) =>
                    {
                        var length = (int)size * (int)nmemb;
                        var buffer = new byte[length];
                        Marshal.Copy(data, buffer, 0, length);
                        stream.Write(buffer, 0, length);
                        return (UIntPtr)length;
                    });
    
                    string tmp;
                    var result = CurlNative.Easy.Perform(easy);
    
                    Console.WriteLine($"Result code: {result}.");
                    Console.WriteLine();
                    Console.WriteLine("Response body:");
                    Console.WriteLine(tmp = Encoding.UTF8.GetString(stream.ToArray()));
    
                    using (StreamWriter sw = new StreamWriter(fileName, false))
                    {
                        sw.Write(tmp);
                    }
                }
                catch (Exception ex)
                {
                    MessageBox.Show("Exception: " + ex.ToString());
                }
                finally
                {
                    easy.Dispose();
    
                    if (global == CURLcode.OK)
                    {
                        CurlNative.Cleanup();
                    }
                }
            }
    

    Monday, October 12, 2020 2:23 AM

All replies

  • Since you're using .NET 4.8/Core then it will use the default TLS protocols defined by the OS. You can get this master list using IISCrypto. For SSL to work the client sends a list of protocols it supports. The server compares that to the list of protocols it allows and returns back the protocol they will use (should be the most secure). The error you're getting indicates no protocols could be found that match. 

    The site in question supports TLS 1.3. Windows doesn't support TLS 1.3 until 1903 which wasn't released until Oct of 2019. Even in 1909 it doesn't show up yet as it was still experimental as of Aug of this year. You can turn it on but folks that have tried to use it indicate that Windows is missing the new encryption algorithms for TLS 1.3 so it didn't work anyway. 

    Normally a site supports TLS 1.2 and TLS 1.3 so it should just fall back to TLS 1.2 but it looks like either this URL is using a TLS 1.3 algorithm with TLS 1.2 (which won't work) or it doesn't support TLS 1.2 at all (which would be odd). You might need to contact them about TLS 1.2 support.

    I modified your code to explicitly use TLS 1.3 (it is not enabled by default on my 1909 OS) and ensure it compiled against .NET 4.8 (because the Tls13 option is available prior to that irrelevant of runtime). 

    ServicePointManager.SecurityProtocol |= SecurityProtocolType.Tls13;

    With that change in place it can now connect but because the encryption algorithm's don't line up it doesn't understand the downloaded data. In theory you can set the appropriate EncryptionPolicy but I'm not sure what that would be for TLS 1.3 and I doubt .NET has support for it yet.



    Michael Taylor http://www.michaeltaylorp3.net

    Wednesday, October 7, 2020 4:13 PM
  • Thanks for your effort Michael.  My apologies ...I forgot to include in my original post I tried forcing TLS 1.3 and the error changed to no common algorithm (cipher suite), implying secure TLS Channel made.  Oddly, the server and client show common algorithms when I compare the client Hello list and SSLLabs report on the server.  I also used PowerShell Get-TlsCipherSuite for sanity.

    Using WireShark, curl successfully downloads the file using TLS 1.2.  Obviously, there is some different in the way >net handles TLS 1.2 than curl.  It's very frustrating other tools work, but .net and .net core don't.

    Wednesday, October 7, 2020 4:36 PM
  • So you're able to download the file using TLS 1.2, what encryption algorithm is it using for that? 

    Michael Taylor http://www.michaeltaylorp3.net

    Wednesday, October 7, 2020 4:41 PM
  • I'm mistaken ...and a bit confused ...WireShark shows curl using TLSv1.3, but within the TLS 1.3 Transport Layer, I see a Handshake version TLS 1.0.  The Client & Server Hello is version TLS 1.2.  All this communication is noted at the highest level as TLSv1.3 Record Layer.   Apparently the 1.3 protocol allows using handshake and hello's from other versions?

    The Server replies to Client Hello with TCP ACK.

    Client then sends "Change Cipher Spec, Application Data". 

    The client then sends 5 TLSv1.3 Application Data packets and many TCP ACK's.

    The server responds with TLSv1.3 Server Hello, Change Cipher Spec, Application Data packet.  In this Hello message (noted as TLS v1.2), I see server chooses Cipher Suite TLS_AES_256_GCM_SHA384.  This Cipher Suite is noted with hex 0x1302 (decimal 4866).  Using PowerShell Get-TLSCipherSuite, I find this Cipher Suite by name, with CipherSuite value of 4866.  The protocol is 772 ...represents TLS 1.3 CipherSuite. 

    After this successful TLS 1.3 download, I decided to force curl to use TLS 1.2 and the file is successfully downloaded.  The client sends a Hello, server replies with TCP ACK.  Client sends Change Cipher Spec, Application Data".  From that point the packets are marked TLS 1.2 Application Data and it is encrypted, and I cannot decrypt.  There is no entry marked Server Hello with a visible cipher suite, so I presume it is encrypted in the application data.
    Wednesday, October 7, 2020 8:11 PM
  • Here's an interesting read on the TLS 1.2 vs 1.3 handshaking. What you're describing is the 1.2 process. 1.3, to speed things up, combines the calls together and that doesn't seem to be happening here.

    At this point I'm wondering if there is an issue with the server implementation rather than something in your code. It could be a cipher suit issue but if it was working before and stopped working now then they changed something on the server. You might want to contact the site owners about it.

    BTW here's the official list of supported TLS and Windows for your reference.


    Michael Taylor http://www.michaeltaylorp3.net

    Wednesday, October 7, 2020 10:01 PM
  • You're likely right about the server.  It is a US government server for the GSA.  I suspected it and reported the issue to them on Sept 14, and the manager says they have personnel issues and to expect a long turn around time.

    I enabled v1.3 on my Win10 machine and download failed with different error.  Wireshark showed it progressed much further but failed due to IO error - frame size or corrupted frame.  Apparently, MS just released a release candidate for .net 5.0 and that will allow .net core to support TLS 1.3 ("provided it's run on an OS that supports it").  First link is the IO error, 2nd link to the MS release.

    https://stackoverflow.com/questions/64212994/net-4-8-tls-1-3-issue-on-windows-10

    https://dotnet.microsoft.com/download/dotnet/5.0

    Thursday, October 8, 2020 2:04 AM
  • I'm to the point I may create a c++ cli program and link to libcurl and call it from c# - all just to download a publicly available file.  See code for the url.

    Or you could use libcurl directly in C# with p/invoke (or use wrappers)

    I  tested your file, it works too with curl (TLSv1.3 / TLS_AES_256_GCM_SHA384, ALPN h2)

    Then I tested in C++ with WinHTTP, which has always worked with any site... but failed for  the first time with your file (ERROR_WINHTTP_SECURE_CHANNEL_ERROR), even by trying various flags.

    Thursday, October 8, 2020 2:53 PM
  • Thank you for your help CoolDadTX and Costorix31!

    I installed a NuGet package called CurlThin that wraps curl.  Another package, CurlThin.Native simply contains dll's for libcurl and ssl.  It works with the government server for tls 1.2 and if I set my windows 10 registry to enable tls 1.3 it works with it too.  I haven't decrypted to see which cipher used for 1.2

    After installing both packages, you need to call CurlResources.Init() from your project code somewhere first.  This installs the needed dependency dll's and .crt file to your output directory.  You need to call this once for Debug and once for Release.  

    I first got a runtime error memory corrupted when trying to read the data stream. My machine is 64-bit and although I set my VS Solution Platform to build x64 configuration on my toolbar, VS did not build x64 until I set the Platform Target on the Build tab of project properties from Any CPU to x64.  

            public void DownloadFile(string url, string fileName)
            {
    
                // curl_global_init() with default flags.
                var global = CurlNative.Init();
    
                // curl_easy_init() to create easy handle.
                var easy = CurlNative.Easy.Init();
                try
                {
                    CurlNative.Easy.SetOpt(easy, CURLoption.URL, url);
    
                    // Added this statement to resolve SSL_CACERT return code. [Error]
                    CurlNative.Easy.SetOpt(easy, CURLoption.CAINFO, CurlResources.CaBundlePath);
    
                    //MessageBox.Show("Using MemoryStream");
                    var stream = new MemoryStream();
                    CurlNative.Easy.SetOpt(easy, CURLoption.WRITEFUNCTION, (data, size, nmemb, user) =>
                    {
                        var length = (int)size * (int)nmemb;
                        var buffer = new byte[length];
                        Marshal.Copy(data, buffer, 0, length);
                        stream.Write(buffer, 0, length);
                        return (UIntPtr)length;
                    });
    
                    string tmp;
                    var result = CurlNative.Easy.Perform(easy);
    
                    Console.WriteLine($"Result code: {result}.");
                    Console.WriteLine();
                    Console.WriteLine("Response body:");
                    Console.WriteLine(tmp = Encoding.UTF8.GetString(stream.ToArray()));
    
                    using (StreamWriter sw = new StreamWriter(fileName, false))
                    {
                        sw.Write(tmp);
                    }
                }
                catch (Exception ex)
                {
                    MessageBox.Show("Exception: " + ex.ToString());
                }
                finally
                {
                    easy.Dispose();
    
                    if (global == CURLcode.OK)
                    {
                        CurlNative.Cleanup();
                    }
                }
            }
    

    Monday, October 12, 2020 2:23 AM