none
HTTPS/SSL Performance issue RRS feed

  • Question

  • Hi,

    I have a service running under HTTPS with the following binding:

    <security mode="Transport">
        <
    transport clientCredentialType="None" />
    </
    security>

    It all works fine and my .Net client is able to call it.

    The first call to the service after an iisreset takes a good few seconds to go through, to be expected as everything is starting up.  Once started, if I fire calls through in fairly quick succession they all get serviced very quickly, normally subsecond (the service itself is fast and doesn't do anything too costly). 

    HOWEVER...if I leave the service alone for a few minutes and then send a request through it's taking around 16 seconds.

    Most of the settings I am using in IIS are defaults.  The main one I did change which I hoped would solve the problem was the Idle Time-out on the Application Pool used for the service.  This defaulted to 20 minutes but I set it to zero; my understanding is that this means the service worker process will never shut down due to being idle.  I have set an explicit recycle of the App Pool at 3am so it is refreshed daily at a quiet time.

    So where is the 16 seconds going if the service/worker process etc is already up and running?

    I have done a few things to try and investigate.  I first enabled WCF transport level logging on the client and server to see what that showed.  On my client machine, it shows the call to the service happening at, say 10:00:00.  And then the response coming back from the server at 10:00:16.

    On the server, however, it shows the call coming in at 10:00:15 and then the response going back at 10:00:15.700.

    So it looked like the call from the client didn't arrive into WCF until 15 seconds after it was sent!  (Please note I sync'd the clocks on the client and server machines!).  Please see here for a screenshot detailing the WCF trace findings: http://cid-d5ae6fb44f5307c6.skydrive.live.com/self.aspx/.Public/WCF%20SSL%20issue.svclog.jpg

    My next step was to use Microsoft Network Monitor to try and see what was happening with the actual network traffic.  This showed the conversation between the client and server starting up and a lot of SSL handshaking before the actual data was sent from the client to the server.   The strange thing I noticed was a long delay (approx 15 seconds) during the handshaking.

    Please see here for a screenshot of the Network Monitor results: http://cid-d5ae6fb44f5307c6.skydrive.live.com/self.aspx/.Public/WCF%20SSL%20issue.netmon.jpg

    What is interesting is these 2 entries:

    09:35:20.448 TCP:Flags=...A...., SrcPort=64920, DstPort=HTTPS(443), PayloadLen=0, Seq=2250966266, Ack=2744013545, Win=256 (scale factor 0x8) = 65536
    09:35:35.245 SSL:  Change Cipher Spec. Encrypted Handshake Message. Application Data.

    Note that there is a 15 second gap between them.  I'm assuming this is my missing 15 seconds.

    So, I hope somebody can help shed some light on this.  What's happening, and how to resolve it!?!?

    Thanks,

    Paul

     

     



    Thursday, March 4, 2010 1:57 PM

Answers

  • I've made some progress.

    Just to rule out the self-signed certificate being the cause I generated a certificate with my own RootCA with a name which matched the URL of my test service.  See instructions here:

    http://mitchdavisspace.spaces.live.com/blog/cns!5D25DB20E4AD9CB3!212.entry?wa=wsignin1.0&sa=853659396

    This enabled me to get rid of the ServerCertificateValidationCallback on the client.  I just had to import the RootCA I created into the client's Trusted Root Certification Authorities store.

    And voila, the problem has disappeared.  My first call now takes less than 1 second!

    So I can only assume the client was attempting to validate the server certificate BEFORE the ServerCertificateValidationCallback was executed, and it was this attempt to validate the certificate which was taking the 15 seconds.    (note that I put a line of debug in the callback and it definitely got executed right at the end of the 15 seconds).

    I thought maybe the client was trying to validate the certificate externally.

    So with that in mind I went back to network monitor.

    With my RootCA installed on the client I monitored the traffic.  I only saw calls from my client application to the server, exactly as you would expect.

    Then I deleted the RootCA from my client and ran it again.  The call took 15 seconds.   I checked network monitor in more detail and noticed the client application calling an IP address which was nothing to do with me:  213.199.149.157

    I did a whois lookup on that IP and it came up with "Microsoft London Internet Data Center"!!!!! So it definitely looks like the client was trying to validate the server certificate against some Microsoft service!!!!

    Cheeky in my opinion, and has caused me about a weeks worth of pain!!!!!!!!!

    I hope this post helps somebody........

    • Marked as answer by robinsonpr Tuesday, March 9, 2010 10:48 AM
    Tuesday, March 9, 2010 10:48 AM

All replies

  • Some more info on this....

    I created a new test service which doesn't do any work, along with a simple client to call it.  I put the service on my test server (Server2008), hosted under IIS7 with Transport security and a HTTPS URL.

    When calling from my development machine (XP Pro) it comes back pretty much instantly.  I checked using Network Monitor and it is doing all the SSL handshaking, all in microseconds.

    I then put the test client on another test server (Server2008).  Exactly the same config file etc.  This time the call took 15 seconds!!!

    So it would appear to be something network related?  The 2 test servers are on the same domain, sitting next to each other in the same rack! 

    Any suggestions on where to look for the problem? 

    Thanks,

    Paul
    Friday, March 5, 2010 9:52 AM
  • OK even more baffling......

    My colleague created a simple client in Java to call my test service.

    Guess what, I deployed it to the Server2008 box and it takes less than 1 second to call the service over SSL.  My .Net client takes 15 seconds!!!

    So to recap:

    - I have a test WCF service on Server A, secured with transport security.

    - I have a .Net WCF client on Dev machine, calling the service over https. Takes 1 second.

    - The same .Net WCF client on Server B, calling the service over https.  Takes 15 seconds.

    - I have a Java client also on Server B, calling the service over https.  Takes 1 second.

    What's going on!?!!?!?!?!?!

    I thought it was something to do with the network/servers but now I have a Java client talking to the service from the server without an issue it's looking like something to do with the WCF client.

    I ran Network Monitor on Server B and compared the results of the .Net call with the Java call.  They were both doing similar SSL handshaking, but the .Net trace showed a big 15 second pause before the call was sent to the server.

    Very puzzled now....


    And to top it off my Java buddy tells me he'll rewrite the service in Java to sort it out!!!

    Gaaah!!!!
    Friday, March 5, 2010 9:26 PM
  • Not that I have an answer, but this is driving me nuts too!

    My observations to date:

    1. If I don't use SSL to make the connection performance is always fast (which is indicative of what you're seeing - that the 15 seconds is related to SSL stuff).

    2. I only have the issue with servers that are in a F5 load balanced network.  The https runs fine on a dedicated server not going through the load balancer.

    3. We're running in a SiteMinder protected environment - don't believe it's related to that.

    4. I'm able to connect using ColdFusion (I installed the certificate into the Java certificate store) without any issues.  So your Java buddy is laughing at the both of us!

    I've done some research and I'm stumped at this point - I've heard all kinds of issues (proxy settings, DNS lookup, .NET going out to Microsoft certification revocation list server, etc.)

    Nothing has worked to date... hopefully someone can suggest a solution.

     

    Saturday, March 6, 2010 2:49 PM
  • Thanks for posting pryzj, and I'm glad I'm not the only person experiencing issues with WCF and SSL.  I am surprised I can't find more info on this, unless it's only very special circumstances which produce the issue.

    My network isn't load balanced.  BUT it is a set of virtual servers running under VMWare, I wonder if that could be anything to do with it.

    What I'm struggling with is getting any more information about where the 15 seconds is going.  Network Monitor didn't really show anything, other than a big gap between the "change cipher" and then sending the data to the server;  I just turned on verbose WCF tracing on the client and that doesn't really show anything either (other than the big delay!)

    Is anyone from Microsoft able to look into this?

    Monday, March 8, 2010 9:12 AM
  • One other point to note which may be significant...

    In this scenario I am in development and integration testing so am using a self-signed certificate.

    In order for my client code to ignore the certificate sent back from the server I have added this to my client call:

    ServicePointManager.ServerCertificateValidationCallback += delegate(object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors)
    {
        return true;
    };


    Could this be related to the issue? 

    Still doesn't explain why the call is 15 seconds going from server to server but subsecond going from my development PC to the server....



     

     

    Monday, March 8, 2010 11:55 AM
  • I don't think it's related to that primarily because I'm able use that same code on a separate server.  Perhaps an authentication issue?  For now, my code is running under the default ASPNET account.  Perhaps we need to be running in under different user account or restrict the type of authentication it's going to do?
    Monday, March 8, 2010 1:27 PM
  • My service is running under a local machine account.  It's not using any authentication, just anonymous.
    Monday, March 8, 2010 1:43 PM
  • You can check how the client to validate the service certificate here. I think that this validation should consume some time for execution in client code.

    Best regards,
    Riquel
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    Welcome to the All-In-One Code Framework! If you have any feedback, please tell us.
    Tuesday, March 9, 2010 8:30 AM
    Moderator
  • I've made some progress.

    Just to rule out the self-signed certificate being the cause I generated a certificate with my own RootCA with a name which matched the URL of my test service.  See instructions here:

    http://mitchdavisspace.spaces.live.com/blog/cns!5D25DB20E4AD9CB3!212.entry?wa=wsignin1.0&sa=853659396

    This enabled me to get rid of the ServerCertificateValidationCallback on the client.  I just had to import the RootCA I created into the client's Trusted Root Certification Authorities store.

    And voila, the problem has disappeared.  My first call now takes less than 1 second!

    So I can only assume the client was attempting to validate the server certificate BEFORE the ServerCertificateValidationCallback was executed, and it was this attempt to validate the certificate which was taking the 15 seconds.    (note that I put a line of debug in the callback and it definitely got executed right at the end of the 15 seconds).

    I thought maybe the client was trying to validate the certificate externally.

    So with that in mind I went back to network monitor.

    With my RootCA installed on the client I monitored the traffic.  I only saw calls from my client application to the server, exactly as you would expect.

    Then I deleted the RootCA from my client and ran it again.  The call took 15 seconds.   I checked network monitor in more detail and noticed the client application calling an IP address which was nothing to do with me:  213.199.149.157

    I did a whois lookup on that IP and it came up with "Microsoft London Internet Data Center"!!!!! So it definitely looks like the client was trying to validate the server certificate against some Microsoft service!!!!

    Cheeky in my opinion, and has caused me about a weeks worth of pain!!!!!!!!!

    I hope this post helps somebody........

    • Marked as answer by robinsonpr Tuesday, March 9, 2010 10:48 AM
    Tuesday, March 9, 2010 10:48 AM
  • So here's the problem I have:

    1. The server that I want to access via SSL is a third party site.  I grab the certificate from that site by opening IE and saving their certificate as a .CER file
    2. I then install the certificate into the server that I want to access the other site from.  I place the certificate in the Trusted Root Certification Authorities store (Local Computer).
    3. The problem I see is that the third party SSL certificate has an "Issued to: " of *.xyz.com (where xyz is the name of the real company).  However, I access the web site via a webrequest using https://www.xyz.com and based on the information above you stated that the names need to match.

    Couple of questions:
    1. Do you know if it will require an "Issued to:" of www.xyz.com?  The administrator of the server that I need to test this on will be out for a few days and I'm trying to get this information prepped for when he returns.
    2. Does the certificate absolutely have to be added to the Trusted Root CA store or can it be stored somewhere else and added programmatically at runtime of the webrequest?

    Thanks in advance for your response.

     

    • Proposed as answer by pryzj Thursday, March 11, 2010 7:04 PM
    Wednesday, March 10, 2010 5:47 AM
  • Just a followup - might seem obvious, but make sure you add all the certificates in the certificate chain to the store.  It makes a difference!
    • Proposed as answer by pryzj Thursday, March 11, 2010 7:07 PM
    Thursday, March 11, 2010 7:06 PM
  • 1. The certificate you mentioned of *.xyz.com is whats know as a "wildcard certificate".  If a client calls site1.xyz.com or site2.xyz.com they should BOTH validate against that wildcard cert.  So no, you hopefully shouldn't need a cert with an "Issued to:" of www.xyz.com  Although I'm not entirely sure if www.xyz.com qualifies against that certificate, you'd have to try it!

    2. There are possibly a few places in the cert store where you can add certificates to get it to work.  I've always used the Trusted Root CA store.   And as you already said, you'll certainly need a route to all certificates in the signing chain!

    Cheers,

    Paul.
    Thursday, March 11, 2010 9:45 PM
  • Thank you a lot, you've saved my day. Experienced the same issue on production site without internet connection. I've added our cert as root CA and it worked!<style></style>
    Wednesday, November 27, 2019 3:30 PM
  • Could use 

    openssl s_client -connect XXX.XXX.XXX.XXX:443 -showcerts

    command in order to get certificate

    <style><br _moz_dirty="" /></style>

    Wednesday, November 27, 2019 3:32 PM