none
WCF System.ServiceModel.Security.SecurityNegotiationException over NetTCP RRS feed

  • Question

  • Hi all - I'm having an issue that I can't identify.  Any help is appreciated.


    We have an ASP.Net site that calls a SOAP web service.  This web service, in turn, calls a WCF service over a Net TCP binding.  Each of these runs in their own app pool, but each app pool is running under the same domain account.
    The web site is on one server, and the services are on another - the services are both on the same box.

    We have 3 environments: Dev, PreProd, and Production. 
    Dev and Production are running Windows Server 2008; PreProd is running Windows Server 2012 R2 (yes, I know they aren't the same.  This is outside of my control).  We're running .NET 4.6.1

    Dev and Prod have multiple services on each tier, with a load balancer.

    So, on to my problem:  

    We recently changed the binding on the WCF service to send Windows credentials.  

    In the Dev environment, things are working fine.  However, in PreProd and Prod, we are getting an exception when the SOAP web service is trying to call the WCF service.  The exception is:

    System.ServiceModel.Security.SecurityNegotiationException: The server has rejected the client credentials. ---> System.Security.Authentication.InvalidCredentialException: The server has rejected the client credentials. ---> System.ComponentModel.Win32Exception: The logon attempt failedat System.Net.Security.NegoState.ProcessReceivedBlob()
    at System.Net.Security.NegoState.StartSendBlob()
    at System.Net.Security.NegoState.StartSendBlob()
    at System.Net.Security.NegoState.ProcessAuthentication()
    at System.Net.Security.NegotiateStream.AuthenticateAsClient()
    at System.ServiceModel.Channels.WindowsStreamSecurityUpgradeProvider.WindowsStreamSecurityUpgradeInitiator.OnInitiateUpgrade()
    --- End of inner exception stack trace ---



    When I look in the Event Viewer, I see a Logon failure in the Security Event Log.  In our custom application Event log, I see this:

    ---------------------------

    General Information

    *********************************************
    Additional Info:

    ExceptionManager.MachineName: <machine name>
    ExceptionManager.TimeStamp: 9/16/2016 5:24:02 PM
    ExceptionManager.FullName: Microsoft.ApplicationBlocks.ExceptionManagement, Version=1.0.0.0, Culture=neutral, PublicKeyToken=4567ec46a5647aa4
    ExceptionManager.AppDomainName: /LM/W3SVC/1/ROOT/<App Pool>
    ExceptionManager.ThreadIdentity:
    ExceptionManager.WindowsIdentity: <service account>

     

    1) Exception Information
    *********************************************
    Exception Type: System.ServiceModel.Security.SecurityNegotiationException
    Message: The server has rejected the client credentials.
    Data: System.Collections.ListDictionaryInternal
    TargetSite: Void HandleReturnMessage(System.Runtime.Remoting.Messaging.IMessage, System.Runtime.Remoting.Messaging.IMessage)
    HelpLink: NULL
    Source: mscorlib
    HResult: -2146233087
     

    StackTrace Information
    *********************************************

    Server stack trace:
       at System.ServiceModel.Channels.WindowsStreamSecurityUpgradeProvider.WindowsStreamSecurityUpgradeInitiator.OnInitiateUpgrade(Stream stream, SecurityMessageProperty& remoteSecurity)
       at System.ServiceModel.Channels.StreamSecurityUpgradeInitiatorBase.InitiateUpgrade(Stream stream)
       at System.ServiceModel.Channels.ConnectionUpgradeHelper.InitiateUpgrade(StreamUpgradeInitiator upgradeInitiator, IConnection& connection, ClientFramingDecoder decoder, IDefaultCommunicationTimeouts defaultTimeouts, TimeoutHelper& timeoutHelper)
       at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.SendPreamble(IConnection connection, ArraySegment`1 preamble, TimeoutHelper& timeoutHelper)
       at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.DuplexConnectionPoolHelper.AcceptPooledConnection(IConnection connection, TimeoutHelper& timeoutHelper)
       at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
       at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
       at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
       at System.ServiceModel.Channels.LayeredChannel`1.OnOpen(TimeSpan timeout)
       at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
       at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
       at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
       at System.ServiceModel.Channels.ServiceChannel.CallOpenOnce.System.ServiceModel.Channels.ServiceChannel.ICallOnce.Call(ServiceChannel channel, TimeSpan timeout)
       at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
       at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
       at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
       at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
       at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)


    Exception rethrown at [0]:
       at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
       at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type
       <Rest of stack trace that calls the service>
     

    2) Exception Information
    *********************************************
    Exception Type: System.Security.Authentication.InvalidCredentialException
    Message: The server has rejected the client credentials.
    Data: System.Collections.ListDictionaryInternal
    TargetSite: Void ProcessReceivedBlob(Byte[], System.Net.LazyAsyncResult)
    HelpLink: NULL
    Source: System
    HResult: -2146233087
     

    StackTrace Information
    *********************************************
       at System.Net.Security.NegoState.ProcessReceivedBlob(Byte[] message, LazyAsyncResult lazyResult)
       at System.Net.Security.NegoState.StartReceiveBlob(LazyAsyncResult lazyResult)
       at System.Net.Security.NegoState.CheckCompletionBeforeNextReceive(LazyAsyncResult lazyResult)
       at System.Net.Security.NegoState.StartSendBlob(Byte[] message, LazyAsyncResult lazyResult)
       at System.Net.Security.NegoState.CheckCompletionBeforeNextSend(Byte[] message, LazyAsyncResult lazyResult)
       at System.Net.Security.NegoState.ProcessReceivedBlob(Byte[] message, LazyAsyncResult lazyResult)
       at System.Net.Security.NegoState.StartReceiveBlob(LazyAsyncResult lazyResult)
       at System.Net.Security.NegoState.CheckCompletionBeforeNextReceive(LazyAsyncResult lazyResult)
       at System.Net.Security.NegoState.StartSendBlob(Byte[] message, LazyAsyncResult lazyResult)
       at System.Net.Security.NegoState.ProcessAuthentication(LazyAsyncResult lazyResult)
       at System.Net.Security.NegotiateStream.AuthenticateAsClient(NetworkCredential credential, ChannelBinding binding, String targetName, ProtectionLevel requiredProtectionLevel, TokenImpersonationLevel allowedImpersonationLevel)
       at System.Net.Security.NegotiateStream.AuthenticateAsClient(NetworkCredential credential, String targetName, ProtectionLevel requiredProtectionLevel, TokenImpersonationLevel allowedImpersonationLevel)
       at System.ServiceModel.Channels.WindowsStreamSecurityUpgradeProvider.WindowsStreamSecurityUpgradeInitiator.OnInitiateUpgrade(Stream stream, SecurityMessageProperty& remoteSecurity)


    3) Exception Information
    *********************************************
    Exception Type: System.ComponentModel.Win32Exception
    NativeErrorCode: -2146893044
    ErrorCode: -2147467259
    Message: The logon attempt failed
    Data: System.Collections.ListDictionaryInternal
    TargetSite: NULL
    HelpLink: NULL
    Source: NULL
    HResult: -2147467259

    ---------------------------

    The service accounts have all been registered as SPNs.


    I have found many instances of this exception on the internet, but nothing seems to apply to this situation.  

    I initially thought that this was something to do with differences in the OS, but Dev and Production are the same, but the problem exists in Prod but not Dev.

    Does anyone have any suggestions that might help us track down this problem?


    Thanks in advance.

    Saturday, September 17, 2016 3:27 AM

Answers

  • Thanks Edward.

    It turns out it was the Loopback check: we added Windows authentication to the WCF endpoint and are referring to the second service through the load balancer address. 

    So, when the first service called the second, and the load balancer sent the call to the same machine, the server treated it like a loopback call, and rejected it.  Basically, what is described here:

    https://support.microsoft.com/en-us/kb/896861

    We decided to add a BackConnectionHostNames entry as described in Method 1 in the article above.  Since we've done this, we haven't had any errors.

    Thanks,

    Phil

    • Marked as answer by PhilMarInOhio Friday, September 23, 2016 1:57 PM
    Friday, September 23, 2016 1:57 PM

All replies

  • Hi PhilMarlnOhio,

    Based on your description, there are three environments and two servers. Do you mean that there are two servers in each environments? How did you implement web service to call WCF service in Production? How did you migrate your service from Dev to Prod? Did you add service reference in Prod?

    >> The service accounts have all been registered as SPNs.

    I suggest you check whether you used a duplicate SPN or you register SPN correctly, you could refer the link below:

    # WCF on intranet with windows authentication: Kerberos or NTLM (Part 1)

    https://blogs.msdn.microsoft.com/tiche/2011/07/12/wcf-on-intranet-with-windows-authentication-kerberos-or-ntlm-part-1/

    Best Regards,

    Edward


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.


    Monday, September 19, 2016 5:37 AM
  • Thanks for the reply, Edward.  Sorry that I didn't include enough details.

    There are three environments, correct.  Each environment consists of a web tier, and an app tier.

    For Dev and Prod, each tier contains two servers.  For PreProd, each tier contains a single server.

    For our prod services, we have a separate build.  We didn't add service references; we have hand-coded proxy classes that inherit from System.ServiceModel.ClientBase.  Bindings, Endpoints and Behaviors are all done through configuration, not code.

    I will check out that link tomorrow when I get into the office.  One other thing I will check is whether this is a loopback issue.  I was able to reproduce this in our Dev environment; for the calls that failed, I see the SOAP service on machine A calling the WCF service on machine B.  For calls that succeed, it doesn't leave the box.  The client endpoint uses the 'cluster' address to specify the service.  Each node in the cluster exposes the service, and the routing from server to client is handled by the load balancer.

    Thanks,

    Phil

    Tuesday, September 20, 2016 1:02 AM
  • Hi Phil,

    Thanks for more information.

    Could you call service cross Dev and prod? If you could, I would suggest you make some more tests to narrow down this issue.

    1. Call Dev service from prod client
    2. Call Prod service from Dev client

    Best Regards,

    Edward


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.


    Tuesday, September 20, 2016 3:02 AM
  • Thanks Edward.

    It turns out it was the Loopback check: we added Windows authentication to the WCF endpoint and are referring to the second service through the load balancer address. 

    So, when the first service called the second, and the load balancer sent the call to the same machine, the server treated it like a loopback call, and rejected it.  Basically, what is described here:

    https://support.microsoft.com/en-us/kb/896861

    We decided to add a BackConnectionHostNames entry as described in Method 1 in the article above.  Since we've done this, we haven't had any errors.

    Thanks,

    Phil

    • Marked as answer by PhilMarInOhio Friday, September 23, 2016 1:57 PM
    Friday, September 23, 2016 1:57 PM
  • Hi Phil,

    Thanks for sharing. It would be much helpful if others meet this issue.

    Best Regards,

    Edward


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.


    Saturday, September 24, 2016 4:20 AM
  • We just added the BackConnectionHostNames for the very same issue and this solved the problem for us!

    Thanks for sharing!

    Thursday, July 6, 2017 8:14 PM