none
atypical problem with updating SSL certificate for existing WCF service renders service inoperable RRS feed

  • Question

  • We have an existing WCF service that was running fine until the last SSL certificate expired. When the administrator updated the server with the new certificate our problems began. Now the service won't initialize. The exception is one that most people have when they haven't granted private key access to the user. Believe me we have spent over 8 hours on this problem. It is not the typical problem. The certificate appears to be valid. The access rights have been granted properly. It's been double triple quadruple checked and beyond.

    One of the steps needing to be performed when updating the certificate with a new one is to update the thumbprint id value you find in the WCF web.config file settings. Believe me, everything has been checked thoroughly and found to be exactly how it needs to be according to the directions.

    It was pointed out that we should attempt an https connection instead of net.tcp on some posts I read earlier. However, due to the service's requirements, we can only operate via net.tcp. This is because we are using some of the advanced features of WCF for transferring large chunks of data without timing out. We cannot use https as that option limits WCF down to operations that last at most 30 seconds (honestly that's what we have found to be reliable and nothing greater than that). 

    We now need to find out all the other issues that people have had using net.tcp on WCF services, when reaching this exception (I selected this from the event log on the server and changed the domain name referenced in the exception to protect the identity of the client since this is a public forum):

    Log Name:      Application
    Source:        System.ServiceModel 4.0.0.0
    Date:          2/19/2013 5:10:44 PM
    Event ID:      3
    Task Category: WebHost
    Level:         Error
    Keywords:      Classic
    User:          NETWORK SERVICE
    Computer:      WEB5
    Description:
    WebHost failed to process a request.
     Sender Information: System.ServiceModel.Activation.HostedHttpRequestAsyncResult/44852309
     Exception: System.ServiceModel.ServiceActivationException: The service '/cksknighted.svc' cannot be activated due to an exception during compilation.  The exception message is: It is likely that certificate 'CN=*.thedomainname.com, OU=Domain Control Validated, O=*.thedomainname.com' may not have a private key that is capable of key exchange or the process may not have access rights for the private key. Please see inner exception for detail.. ---> System.ArgumentException: It is likely that certificate 'CN=*.thedomainname.com, OU=Domain Control Validated, O=*.thedomainname.com' may not have a private key that is capable of key exchange or the process may not have access rights for the private key. Please see inner exception for detail. ---> System.Security.Cryptography.CryptographicException: Keyset does not exist

       at System.Security.Cryptography.Utils.CreateProvHandle(CspParameters parameters, Boolean randomKeyContainer)
       at System.Security.Cryptography.Utils.GetKeyPairHelper(CspAlgorithmType keyType, CspParameters parameters, Boolean randomKeyContainer, Int32 dwKeySize, SafeProvHandle& safeProvHandle, SafeKeyHandle& safeKeyHandle)
       at System.Security.Cryptography.RSACryptoServiceProvider.GetKeyPair()
       at System.Security.Cryptography.X509Certificates.X509Certificate2.get_PrivateKey()
       at System.ServiceModel.Security.SecurityUtils.EnsureCertificateCanDoKeyExchange(X509Certificate2 certificate)
       --- End of inner exception stack trace ---
       at System.ServiceModel.Security.SecurityUtils.EnsureCertificateCanDoKeyExchange(X509Certificate2 certificate)
       at System.ServiceModel.Security.ServiceCredentialsSecurityTokenManager.CreateLocalSecurityTokenProvider(RecipientServiceModelSecurityTokenRequirement recipientRequirement)
       at System.ServiceModel.Security.ServiceCredentialsSecurityTokenManager.CreateSecurityTokenProvider(SecurityTokenRequirement requirement)
       at System.ServiceModel.Channels.SslStreamSecurityUpgradeProvider.CreateServerProvider(SslStreamSecurityBindingElement bindingElement, BindingContext context)
       at System.ServiceModel.Channels.SslStreamSecurityBindingElement.BuildServerStreamUpgradeProvider(BindingContext context)
       at System.ServiceModel.Channels.ConnectionOrientedTransportChannelListener..ctor(ConnectionOrientedTransportBindingElement bindingElement, BindingContext context)
       at System.ServiceModel.Channels.TcpChannelListener..ctor(TcpTransportBindingElement bindingElement, BindingContext context)
       at System.ServiceModel.Channels.TcpTransportBindingElement.BuildChannelListener[TChannel](BindingContext context)
       at System.ServiceModel.Channels.Binding.BuildChannelListener[TChannel](Uri listenUriBaseAddress, String listenUriRelativeAddress, ListenUriMode listenUriMode, BindingParameterCollection parameters)
       at System.ServiceModel.Description.DispatcherBuilder.MaybeCreateListener(Boolean actuallyCreate, Type[] supportedChannels, Binding binding, BindingParameterCollection parameters, Uri listenUriBaseAddress, String listenUriRelativeAddress, ListenUriMode listenUriMode, ServiceThrottle throttle, IChannelListener& result, Boolean supportContextSession)
       at System.ServiceModel.Description.DispatcherBuilder.BuildChannelListener(StuffPerListenUriInfo stuff, ServiceHostBase serviceHost, Uri listenUri, ListenUriMode listenUriMode, Boolean supportContextSession, IChannelListener& result)
       at System.ServiceModel.Description.DispatcherBuilder.InitializeServiceHost(ServiceDescription description, ServiceHostBase serviceHost)
       at System.ServiceModel.ServiceHostBase.InitializeRuntime()
       at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout)
       at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
       at System.ServiceModel.ServiceHostingEnvironment.HostingManager.ActivateService(String normalizedVirtualPath)
       at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath)
       --- End of inner exception stack trace ---
       at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at System.ServiceModel.Activation.HostedHttpRequestAsyncResult.End(IAsyncResult result)
     Process Name: w3wp
     Process ID: 11604

    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="System.ServiceModel 4.0.0.0" />
        <EventID Qualifiers="49154">3</EventID>
        <Level>2</Level>
        <Task>5</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2013-02-20T01:10:44.000000000Z" />
        <EventRecordID>1191739</EventRecordID>
        <Channel>Application</Channel>
        <Computer>WEB5</Computer>
        <Security UserID="S-1-5-20" />
      </System>
      <EventData>
        <Data>System.ServiceModel.Activation.HostedHttpRequestAsyncResult/44852309</Data>
        <Data>System.ServiceModel.ServiceActivationException: The service '/cksknighted.svc' cannot be activated due to an exception during compilation.  The exception message is: It is likely that certificate 'CN=*.thedomainname.com, OU=Domain Control Validated, O=*.thedomainname.com' may not have a private key that is capable of key exchange or the process may not have access rights for the private key. Please see inner exception for detail.. ---&gt; System.ArgumentException: It is likely that certificate 'CN=*.thedomainname.com, OU=Domain Control Validated, O=*.thedomainname.com' may not have a private key that is capable of key exchange or the process may not have access rights for the private key. Please see inner exception for detail. ---&gt; System.Security.Cryptography.CryptographicException: Keyset does not exist

       at System.Security.Cryptography.Utils.CreateProvHandle(CspParameters parameters, Boolean randomKeyContainer)
       at System.Security.Cryptography.Utils.GetKeyPairHelper(CspAlgorithmType keyType, CspParameters parameters, Boolean randomKeyContainer, Int32 dwKeySize, SafeProvHandle&amp; safeProvHandle, SafeKeyHandle&amp; safeKeyHandle)
       at System.Security.Cryptography.RSACryptoServiceProvider.GetKeyPair()
       at System.Security.Cryptography.X509Certificates.X509Certificate2.get_PrivateKey()
       at System.ServiceModel.Security.SecurityUtils.EnsureCertificateCanDoKeyExchange(X509Certificate2 certificate)
       --- End of inner exception stack trace ---
       at System.ServiceModel.Security.SecurityUtils.EnsureCertificateCanDoKeyExchange(X509Certificate2 certificate)
       at System.ServiceModel.Security.ServiceCredentialsSecurityTokenManager.CreateLocalSecurityTokenProvider(RecipientServiceModelSecurityTokenRequirement recipientRequirement)
       at System.ServiceModel.Security.ServiceCredentialsSecurityTokenManager.CreateSecurityTokenProvider(SecurityTokenRequirement requirement)
       at System.ServiceModel.Channels.SslStreamSecurityUpgradeProvider.CreateServerProvider(SslStreamSecurityBindingElement bindingElement, BindingContext context)
       at System.ServiceModel.Channels.SslStreamSecurityBindingElement.BuildServerStreamUpgradeProvider(BindingContext context)
       at System.ServiceModel.Channels.ConnectionOrientedTransportChannelListener..ctor(ConnectionOrientedTransportBindingElement bindingElement, BindingContext context)
       at System.ServiceModel.Channels.TcpChannelListener..ctor(TcpTransportBindingElement bindingElement, BindingContext context)
       at System.ServiceModel.Channels.TcpTransportBindingElement.BuildChannelListener[TChannel](BindingContext context)
       at System.ServiceModel.Channels.Binding.BuildChannelListener[TChannel](Uri listenUriBaseAddress, String listenUriRelativeAddress, ListenUriMode listenUriMode, BindingParameterCollection parameters)
       at System.ServiceModel.Description.DispatcherBuilder.MaybeCreateListener(Boolean actuallyCreate, Type[] supportedChannels, Binding binding, BindingParameterCollection parameters, Uri listenUriBaseAddress, String listenUriRelativeAddress, ListenUriMode listenUriMode, ServiceThrottle throttle, IChannelListener&amp; result, Boolean supportContextSession)
       at System.ServiceModel.Description.DispatcherBuilder.BuildChannelListener(StuffPerListenUriInfo stuff, ServiceHostBase serviceHost, Uri listenUri, ListenUriMode listenUriMode, Boolean supportContextSession, IChannelListener&amp; result)
       at System.ServiceModel.Description.DispatcherBuilder.InitializeServiceHost(ServiceDescription description, ServiceHostBase serviceHost)
       at System.ServiceModel.ServiceHostBase.InitializeRuntime()
       at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout)
       at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
       at System.ServiceModel.ServiceHostingEnvironment.HostingManager.ActivateService(String normalizedVirtualPath)
       at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath)
       --- End of inner exception stack trace ---
       at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at System.ServiceModel.Activation.HostedHttpRequestAsyncResult.End(IAsyncResult result)</Data>
        <Data>w3wp</Data>
        <Data>11604</Data>
      </EventData>
    </Event>

    Wednesday, February 20, 2013 5:09 PM

Answers

  • We found the problem after much hacking and research! There is a known bug in WCF that apparently Microsoft has allowed to continue for some time now

    Talk about upsetting! This took our team about 48 hours of manpower to solve. We solved our dilemma first by hacking, then we researched with extra knowledge we gleaned while hacking to find that post there.

    The problem is Microsoft software doesn't work right. In our case, with the certificates that GoDaddy is now issuing out, all domains are now coming with extra domains tagged on for free, like "www". So if you purchase a certificate for xyz.com and you plan to use strictly for WCF using net.tcp transport then you too will encounter this bug. You will then need to modify the client .config file and replace one of the xyz.com entries with www.xyz.com.

    Pain, pain, pain. Microsoft, please fix your code!

    • Marked as answer by RavingLunatic Thursday, February 21, 2013 8:52 PM
    Thursday, February 21, 2013 8:52 PM

All replies

  • As a followup, the website's app pool is using the identity Network Service at the moment and that identity has been granted access to the private keys for the SSL certificate. We have attempted each of these other identities: System, Local Service, IIS_USRS and AppPoolIdentity and also granted each identity read and write access for the certificate.

    Also, the certificate installed was a *.pfx file if that is meaningful in any way. I know we attempted to export private keys and import them again and when attempting to do that we were not given the option to export private keys. The popup wizard window reads:

    Private keys are password protected. If you want to export the private key with the certificate, you must type a password on a later page.

    Do you want to export the private key with the certificate?

    Yes, export the private key
    No, do not export the private key

    Note: The associated private key cannot be found. Only the certificate can be exported.

    So, something is haywire, but where exactly? The certificate appears to be valid otherwise.

    I think we have stumbled into a bug.

    Wednesday, February 20, 2013 6:19 PM
  • Additionally, we are using the same certificate for another site and it works fine there for https. No detectable problems. Also we have researched the issue with exporting the private key and the leading cause of that (overwritten profile of Administrator account) is not our issue.
    Wednesday, February 20, 2013 6:27 PM
  • Hi, I am confused on what is your issue now from your description, please explain in a simplified way, thanks.
    Thursday, February 21, 2013 10:12 AM
  • We found the problem after much hacking and research! There is a known bug in WCF that apparently Microsoft has allowed to continue for some time now

    Talk about upsetting! This took our team about 48 hours of manpower to solve. We solved our dilemma first by hacking, then we researched with extra knowledge we gleaned while hacking to find that post there.

    The problem is Microsoft software doesn't work right. In our case, with the certificates that GoDaddy is now issuing out, all domains are now coming with extra domains tagged on for free, like "www". So if you purchase a certificate for xyz.com and you plan to use strictly for WCF using net.tcp transport then you too will encounter this bug. You will then need to modify the client .config file and replace one of the xyz.com entries with www.xyz.com.

    Pain, pain, pain. Microsoft, please fix your code!

    • Marked as answer by RavingLunatic Thursday, February 21, 2013 8:52 PM
    Thursday, February 21, 2013 8:52 PM
  • Hi, I am confused on what is your issue now from your description, please explain in a simplified way, thanks.
    We just figured out what the problem really is by hacking. I just posted the problem and solution in this thread in case you too have this problem, it is a bug in Microsoft code.
    Thursday, February 21, 2013 8:54 PM