none
WCF in load balanced environment with performance RRS feed

  • General discussion

  • I am a new to WCF, and recently happened that I was placed into a ongoing project which involves wcf services.

    As the project has taken into the last stages we are facing performances issues(500 error on the server) and

    the release is nearing by.So during the performance runs --

    getting this exception message:-

    "Cannot find the negotiation state for the context 'uuid-XXXXXXX" in the Tracelog.Exception information: 

        Exception type:
    SecurityNegotiationException 

        Exception message:
    Secure channel cannot be opened because security negotiation with the remote
    endpoint has failed. This may be due to absent or incorrectly specified
    EndpointIdentity in the EndpointAddress used to create the channel. Please
    verify the EndpointIdentity specified or implied by the EndpointAddress
    correctly identifies the remote endpoint. 



    Server stack trace: 

       at
    System.ServiceModel.Security.IssuanceTokenProviderBase`1.DoNegotiation(TimeSpan
    timeout)

       at
    System.ServiceModel.Security.SspiNegotiationTokenProvider.OnOpen(TimeSpan
    timeout)

       at
    System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)

       at
    System.ServiceModel.Security.SecurityProtocol.OnOpen(TimeSpan timeout)

       at
    System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)

       at
    System.ServiceModel.Channels.SecurityChannelFactory`1.ClientSecurityChannel`1.OnOpen(TimeSpan
    timeout)

       at
    System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)

       at
    System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)

       at
    System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)

       at
    System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan
    timeout, CallOnceManager cascade)

       at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan
    timeout)

       at
    System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway,
    ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)

       at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage
    methodCall, ProxyOperationRuntime operation)

       at
    System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)



    Exception rethrown at [0]: 

       at
    System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg,
    IMessage retMsg)

       at
    System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData&
    msgData, Int32 type)

       at
    GenericAdapter.SearchSVCRef.ISearch.GetRenewCount()

       at
    GenericAdapter.SearchSVCRef.SearchClient.GetRenewCount()

       at
    GenericAdapter.ContentServices.SearchAdapter.GetRenewCount()

       at
    Intel.CM.ReferenceLibraryUI.BusinessObjects.Adapter.UserInfo.GetDocRenewCount()

       at
    Intel.CM.ReferenceLibraryUI.Web.SiteMaster.RefreshToolbar()

       at
    Intel.CM.ReferenceLibraryUI.Web.SiteMaster.refLibToolbar_PreRender(Object
    sender, EventArgs e)

       at
    System.Web.UI.Control.PreRenderRecursiveInternal()

       at
    System.Web.UI.Control.PreRenderRecursiveInternal()

       at
    System.Web.UI.Control.PreRenderRecursiveInternal()

       at
    System.Web.UI.Control.PreRenderRecursiveInternal()

       at
    System.Web.UI.Control.PreRenderRecursiveInternal()

       at
    System.Web.UI.Control.PreRenderRecursiveInternal()

       at
    System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint,
    Boolean includeStagesAfterAsyncPoint)



    The
    request for security token has invalid or malformed elements.

       at
    System.ServiceModel.Security.SecurityUtils.ThrowIfNegotiationFault(Message
    message, EndpointAddress target)

       at System.ServiceModel.Security.SspiNegotiationTokenProvider.GetNextOutgoingMessageBody(Message
    incomingMessage, SspiNegotiationTokenProviderState sspiState)



    The following are the respective settings of service and client (client code)

    Service config :-

    <binding name="secureWSBinding">                   

     <readerQuotas maxDepth="2147483647" maxStringContentLength="2147483647"  maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647" />        

              <reliableSession enabled="false"/>                    <security mode="TransportWithMessageCredential">         

                   <message clientCredentialType="" establishSecurityContext="false" negotiateServiceCredential="false"/>     

                   </security>                </binding>

    Client code snippet:-

    WSHttpBinding WSBinding = new WSHttpBinding();
                Uri SVCUri = default(Uri);
                EndpointIdentity SvcUPN = default(EndpointIdentity);
                if (EnvName == null)
                    EnvName = "DEV";
                
                    SVCUri = new Uri("https://XXXX/SystemMessageService.svc?wsdl"); ----> [uri hidden]
                    SvcUPN = EndpointIdentity.CreateUpnIdentity("YYYe@aXXX.com"); ---[identity hidden]
              
                AddressHeader header = AddressHeader.CreateAddressHeader("apiKey", "ns", InitGenericAdapter.APIKey);
                EndpointAddress SvcURL = new EndpointAddress(SVCUri, SvcUPN, header);
                WSHttpSecurity WSSecurityMode = new WSHttpSecurity();
                WSSecurityMode.Mode = SecurityMode.TransportWithMessageCredential;
                WSSecurityMode.Message.ClientCredentialType = MessageCredentialType.Windows;
                WSSecurityMode.Message.EstablishSecurityContext = false;
                WSSecurityMode.Message.NegotiateServiceCredential = false;
                WSBinding.Security = WSSecurityMode;
                ObjSMClient = new SystemMessageClient(WSBinding, SvcURL);

    Any help on this is much appreciated

    Friday, September 30, 2016 11:10 AM

All replies

  • 1. Is this an issue that occurs during load testing only, i.e. is this an intermittent issue ? And there are times when this exact set up works correctly? 

    2. You have mentioned this is a load balanced server?  And exactly the same config is used on all servers acting behind the load balancer?

    3. Is the each server behind the load balancer working correctly.?

    4. Are you able to provide the server WSDL and server endpoint config?

    This sort of problem is usually related to the sever , client contract/ bindings mismatch 






    • Edited by lanax Saturday, October 1, 2016 6:56 AM
    Saturday, October 1, 2016 4:21 AM
  • Please find the answer below each question

    1. Is this an issue that occurs during load testing only, i.e. is this an intermittent issue ? And there are times when this exact set up works correctly? 

    Ans)  Not intermittent during high load testing  . But "YES" looks intermittent for load not too high or normal. Set up works exactly many times correctly.

    2. You have mentioned this is a load balanced server?  And exactly the same config is used on all servers acting behind the load balancer?

    Ans) Having same configurations across all the servers in the load balanced environment.

    3. Is the each server behind the load balancer working correctly.?

    Ans) Yes

    4. Are you able to provide the server WSDL and server endpoint config?

    Ans) If you are referring to the (.svc) link for WSDL, it is working fine and able to provide the endpoint configs to the client (as was mentioned above [at the start of disccussion] in the client code) . If you are not referring to the (.svc) link , please enlighten me

    This sort of problem is usually related to the sever , client contract/ bindings mismatch

    Not a question of MISMATCH

    Please post your views (If possible sample code) as the delivery is nearing by.
    • Edited by Rex Muduli Monday, October 3, 2016 5:24 AM refined the first questions' answer
    Monday, October 3, 2016 3:04 AM
  • Hi,

    Fundamentally, this issues has got to do with an invalid security token. There are many ways in which this can happen from ticket expiry, to  incorrect SPN configuration  not suited for load balancing environments and how the load balancer topology is set up..

    From the stack trace  it looks like the WCF request originates from a Web page. 

    1. Is the Web page that is behind the load balancer (LB) ? And do all the web apps behind the LB invoke the same WCF server? Or Are the requests to the WCF server also routed through the LB/

    2.  You might have to isolate the source of the problem by load testing the WCF service separately with and without the LB ( applied only if WCF requests are routed through LB), but without going through the web app forms . Your aim is to  figure out the problem area (WCF service or WCF Service in combination with LB and if via LB then which instances ).  If you have multiple WCF services behind LB, you need to individually load test all of them.

    3. Looks the WCF service is using windows authentication. Are you using Kerberos or NtLM? If Kerberos, is the SPN configured correctly if the requests to your WCF service is behind an LB. For SPN configured with LB https://blogs.technet.microsoft.com/askds/2011/08/09/kerberos-and-load-balancing/ 

    4.  The WCF requests originating from the web page with windows authentication. In your web app are you using impersonation or delegation of the end user or is the credentials the app pool identity to connect to the WCF service

    5. WSDL  is the contract that defines the protocol that the client must comply to use the service.  So you might want to post the WSDL content and the endpoint configuration for the server and client ( everything between <System.ServiceModel></System.ServiceModel> in your configurations. The config you have posted is only the binding configuration, but the behaviour and endpoint configuration are also required to complete the security settings.

    6. Some basic checks, are you making sure that the WCF client is disposed off correctly? See sample code here https://coding.abel.nu/2012/02/using-and-disposing-of-wcf-clients/

    7. Also what is the percentage of these failures under load test?

    Also include your load balancer topology to show how the requests are routed from the load testing client all the way to the WCF service, include details of the Host and service instances.

    You really have to load test each component in isolation to narrow down the source of your problem.


    you might also want to trace security audits https://msdn.microsoft.com/en-au/library/ff647243.aspx
    • Edited by lanax Tuesday, October 4, 2016 4:12 AM
    Tuesday, October 4, 2016 3:54 AM