locked
Security Token Problem RRS feed

  • Question

  • Hi,

    I am getting the following error when load testing my service:-

    System.ServiceModel.Security.MessageSecurityException: An unsecured or incorrectly secured fault was received from the other party. See the inner FaultException for the fault code and detail. ---> 
    System.ServiceModel.FaultException: The security context token is expired or is not valid. The message was not processed.

    I have establishSecurityContext=false on the binding and there are no time delays involved that could cause the token to expire. I have posted to the load test forum, but am also posting here just in case anyone has seen this before. This service works fine outside the context of a VS load test (so far that I have seen!).

    What could be causing this? I didn't even think I would have an SCT as I have configured it not to use one?!?

    Cheers
    Dan

    Tuesday, January 20, 2009 12:53 PM

Answers

  • Does your load balancer support sticky sessions?
    If not then the problem is probably that some requests go to one server and some to another.
    Note that even though you logically send one request there are a few infrastructure messages that are sent.
    Try to set negotiateServerCredential to false to have a one shot kerberos (this requires your server to run from a domain account).
    You can also use sticky sessions.


    http://webservices20.blogspot.com/
    WCF Security, Performance And Testing Blog
    • Marked as answer by Dan0001 Wednesday, January 21, 2009 4:54 PM
    Tuesday, January 20, 2009 3:33 PM

All replies

  • Hi Dan,

    Could you provide more details about what binding and configuration you are using for the service ?

    Thanks
    Pablo.
    Pablo Cibraro - http://weblogs.asp.net/cibrax
    Tuesday, January 20, 2009 1:55 PM
  • Hi Pablo,

    Here is the binding I am using on the client:-

    <binding name="WSHttpBinding_IReports_IReports4" closeTimeout="00:01:00"

    openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:10:00"

    bypassProxyOnLocal="false" transactionFlow="false" hostNameComparisonMode="StrongWildcard"

    maxBufferPoolSize="524288" maxReceivedMessageSize="100000000"

    messageEncoding="Text" textEncoding="utf-8" useDefaultWebProxy="true"

    allowCookies="false">

    <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"

    maxBytesPerRead="4096" maxNameTableCharCount="16384" />

    <reliableSession ordered="true" inactivityTimeout="00:10:00"

    enabled="false" />

    <security mode="Message">

    <transport clientCredentialType="Windows" proxyCredentialType="None"

    realm="" />

    <message clientCredentialType="Windows" negotiateServiceCredential="true"

    algorithmSuite="Default" establishSecurityContext="false" />

    </security>

    </binding>


    This is the binding on the service side:-

            <binding name="WSHttpBinding_IReports" closeTimeout="00:30:00"
              openTimeout="00:30:00" receiveTimeout="00:30:00" sendTimeout="00:30:00"
              bypassProxyOnLocal="false" transactionFlow="false" hostNameComparisonMode="StrongWildcard"
              maxBufferPoolSize="524288" maxReceivedMessageSize="5000000"
              messageEncoding="Text" textEncoding="utf-8" useDefaultWebProxy="true"
              allowCookies="false">
              <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"
                maxBytesPerRead="4096" maxNameTableCharCount="16384" />
              <reliableSession ordered="true" inactivityTimeout="00:20:00"
                enabled="false" />
              <security mode="Message">
                <transport clientCredentialType="Windows" proxyCredentialType="None"
                  realm="" />
                <message clientCredentialType="Windows" negotiateServiceCredential="true"
                  algorithmSuite="Default" establishSecurityContext="false" />
              </security>
            </binding>


    Do you need any other configuration information? Let me know and I will post it asap :)

    Cheers
    Dan


    Tuesday, January 20, 2009 2:03 PM
  • Hi Dan,

    That's quite weird, your configuration seems to be ok. (You have also disabled secure conversation). Could you enable the WCF trace to see if we can find more details about the error in there ?. (And see also if the secure conversation messages are actually being exchanged).
    That can be done with this section in the configuration file,

    <system.diagnostics>
      <sources>
       <source name="System.ServiceModel.MessageLogging" switchValue="Error">
        <listeners>
         <add name="ServiceModelMessageLoggingListener" />
        </listeners>
       </source>
       <source name="System.ServiceModel" switchValue="Error" propagateActivity="true">
        <listeners>
         <add name="ServiceModelTraceListener" />
        </listeners>
       </source>
      </sources>
      <sharedListeners>
       <add initializeData="messages.svclog" type="System.Diagnostics.XmlWriterTraceListener, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" name="ServiceModelMessageLoggingListener" traceOutputOptions="Timestamp">
       </add>
       <add initializeData="tracelog.svclog" type="System.Diagnostics.XmlWriterTraceListener, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" name="ServiceModelTraceListener" traceOutputOptions="Timestamp">
       </add>
      </sharedListeners>
      <trace autoflush="true" />
    </system.diagnostics>

    Regards,
    Pablo.
    Pablo Cibraro - http://weblogs.asp.net/cibrax
    Tuesday, January 20, 2009 2:36 PM
  • Dan

    When negotiateServiceCredential is "true" there is an SCT even if you do not use a secure session.
    Are you using a web farm? If so you should recheck for time differences between the servers (including daylight saving option).
    And as Pablso said the trace log would give further information.

    http://webservices20.blogspot.com/
    WCF Security, Performance And Testing Blog
    Tuesday, January 20, 2009 2:51 PM
  • ok, I got some more info!

    I turned on tracingand got the following trace items with warnings or errors (I used the service config utility instead of the above xml fragment just in case I lost anything - Pablo, let me know if it needed to be exactly as above and I will run the test again with it):-

    While processing message 1, it did the standard stuff of opening a channel and then threw an exception on the process rstr issue action:-

    <E2ETraceEvent xmlns="http://schemas.microsoft.com/2004/06/E2ETraceEvent">

    <System xmlns="http://schemas.microsoft.com/2004/06/windows/eventlog/system">

    <EventID>131075</EventID>

    <Type>3</Type>

    <SubType Name="Warning">0</SubType>

    <Level>4</Level>

    <TimeCreated SystemTime="2009-01-20T14:46:08.2375375Z" />

    <Source Name="System.ServiceModel" />

    <Correlation ActivityID="{677dfad5-6800-487b-b82e-2c76b8fe51df}" />

    <Execution ProcessName="w3wp" ProcessID="4348" ThreadID="5" />

    <Channel />

    <Computer>MYSERVICEHOST</Computer>

    </System>

    <ApplicationData>

    <TraceData>

    <DataItem>

    <TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Warning">

    <TraceIdentifier>http://msdn.microsoft.com/en-GB/library/System.ServiceModel.Diagnostics.ThrowingException.aspx</TraceIdentifier>

    <Description>Throwing an exception.</Description>

    <AppDomain>/LM/W3SVC/1346661509/ROOT/SDMASCAMPAIGNSHAPE-2-128769363675500331</AppDomain>

    <Exception>

    <ExceptionType>System.ServiceModel.Security.SecurityNegotiationException, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>

    <Message>Cannot find the negotiation state for the context 'uuid-954e94c0-6e42-4816-b53f-bc75e18a9534-2'.</Message>

    <StackTrace>

    at System.ServiceModel.Security.NegotiationTokenAuthenticator`1.ProcessRequestCore(Message request)

    at System.ServiceModel.Security.NegotiationTokenAuthenticator`1.NegotiationHost.NegotiationSyncInvoker.Invoke(Object instance, Object[] inputs, Object[]&amp; outputs)

    at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc&amp; rpc)

    at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc&amp; rpc)

    at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(MessageRpc&amp; rpc)

    at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage3(MessageRpc&amp; rpc)

    at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage2(MessageRpc&amp; rpc)

    at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage1(MessageRpc&amp; rpc)

    at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)

    at System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(RequestContext request, Boolean cleanThread, OperationContext currentOperationContext)

    at System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(RequestContext request, OperationContext currentOperationContext)

    at System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(IAsyncResult result)

    at System.ServiceModel.Dispatcher.ChannelHandler.OnAsyncReceiveComplete(IAsyncResult result)

    at System.ServiceModel.Diagnostics.Utility.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)

    at System.ServiceModel.AsyncResult.Complete(Boolean completedSynchronously)

    at System.ServiceModel.Channels.InputQueue`1.AsyncQueueReader.Set(Item item)

    at System.ServiceModel.Channels.InputQueue`1.Dispatch()

    at System.ServiceModel.Channels.InputQueue`1.OnDispatchCallback(Object state)

    at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke2()

    at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.OnSecurityContextCallback(Object o)

    at System.Security.SecurityContext.Run(SecurityContext securityContext, ContextCallback callback, Object state)

    at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke()

    at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ProcessCallbacks()

    at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.CompletionCallback(Object state)

    at System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ScheduledOverlapped.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)

    at System.ServiceModel.Diagnostics.Utility.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)

    at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)

    </StackTrace>

    <ExceptionString>System.ServiceModel.Security.SecurityNegotiationException: Cannot find the negotiation state for the context 'uuid-954e94c0-6e42-4816-b53f-bc75e18a9534-2'.</ExceptionString>

    </Exception>

    </TraceRecord>

    </DataItem>

    </TraceData>

    </ApplicationData>

    </E2ETraceEvent>


    I then got this message directly afterwards:-

    <E2ETraceEvent xmlns="http://schemas.microsoft.com/2004/06/E2ETraceEvent">

    <System xmlns="http://schemas.microsoft.com/2004/06/windows/eventlog/system">

    <EventID>458760</EventID>

    <Type>3</Type>

    <SubType Name="Warning">0</SubType>

    <Level>4</Level>

    <TimeCreated SystemTime="2009-01-20T14:46:08.2375375Z" />

    <Source Name="System.ServiceModel" />

    <Correlation ActivityID="{677dfad5-6800-487b-b82e-2c76b8fe51df}" />

    <Execution ProcessName="w3wp" ProcessID="4348" ThreadID="5" />

    <Channel />

    <Computer>MYSERVICEHOST</Computer>

    </System>

    <ApplicationData>

    <TraceData>

    <DataItem>

    <TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Warning">

    <TraceIdentifier>http://msdn.microsoft.com/en-GB/library/System.ServiceModel.Security.SecurityNegotiationProcessingFailure.aspx</TraceIdentifier>

    <Description>Service security negotiation processing failure.</Description>

    <AppDomain>/LM/W3SVC/1346661509/ROOT/SDMASCAMPAIGNSHAPE-2-128769363675500331</AppDomain>

    <ExtendedData xmlns="http://schemas.microsoft.com/2006/08/ServiceModel/NegotiationTokenAuthenticatorTraceRecord">

    <NegotiationTokenAuthenticator>System.ServiceModel.Security.SpnegoTokenAuthenticator</NegotiationTokenAuthenticator>

    <AuthenticatorListenUri>http://myservicehost:8001/SDMASCAMPAIGNSHAPE/Reports.svc</AuthenticatorListenUri>

    <Exception>System.ServiceModel.Security.SecurityNegotiationException: Cannot find the negotiation state for the context 'uuid-954e94c0-6e42-4816-b53f-bc75e18a9534-2'.

    at System.ServiceModel.Security.NegotiationTokenAuthenticator`1.ProcessRequestCore(Message request)

    at System.ServiceModel.Security.NegotiationTokenAuthenticator`1.NegotiationHost.NegotiationSyncInvoker.Invoke(Object instance, Object[] inputs, Object[]&amp; outputs)</Exception>

    </ExtendedData>

    </TraceRecord>

    </DataItem>

    </TraceData>

    </ApplicationData>

    </E2ETraceEvent>


    On the next activity, a receive bytes activity, there was a throwing exception warning while processing message 2, exception type:-

    System.ServiceModel.Security.Tokens.SecurityContextTokenValidationException, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089

    with message:-

    The SecurityContextSecurityToken with context-id=urn:uuid:06650bfd-89c8-47af-9f25-a232d589b036 (key generation-id=) is not registered.

    and then a throwing exception error, exception type:-

    System.ServiceModel.Security.MessageSecurityException, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089

    with message:-
    Message security verification failed.

    I understand now that I get an SCT even without secure session (thanks Yaron!) and I am using network load balancing with a two server farm.

    It is
     
    amazingly intermittent - with 7 consecutive load test runs I got PPFPPPF!

    I can email the full trace file if necessary (I am having a nightmare with this text editor ;-)

    I also noticed the following in the security event log and am not sure if it is relevant as I get this even when everything works fine and the account in question (TeamTestService) is a local admin (I promoted him from standard user as a grasping at straws option!).

    Event Type: Failure Audit
    Event Source: Security
    Event Category: Object Access
    Event ID: 560
    Date:  20/01/2009
    Time:  14:43:03
    User:  MYSERVICEHOST\TeamTestService
    Computer: MYSERVICEHOST
    Description:
    Object Open:
      Object Server: SC Manager
      Object Type: SC_MANAGER OBJECT
      Object Name: ServicesActive
      Handle ID: -
      Operation ID: {0,223715460}
      Process ID: 480
      Image File Name: C:\WINDOWS\system32\services.exe
      Primary User Name: MYSERVICEHOST$
      Primary Domain: MYSERVICEDOMAIN
      Primary Logon ID: (0x0,0x3E7)
      Client User Name: TeamTestService
      Client Domain: MYSERVICEHOST
      Client Logon ID: (0x0,0xD48E881)
      Accesses: READ_CONTROL
       Connect to service controller
       Enumerate services
       Query service database lock state
       
      Privileges: -
      Restricted Sid Count: 0
      Access Mask: 0x20015


    For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

    • Proposed as answer by Dev-Admin Sunday, March 4, 2012 12:47 PM
    Tuesday, January 20, 2009 3:13 PM
  • Fforgot to add, I checked the server times and they are the same.
    Tuesday, January 20, 2009 3:14 PM
  • Does your load balancer support sticky sessions?
    If not then the problem is probably that some requests go to one server and some to another.
    Note that even though you logically send one request there are a few infrastructure messages that are sent.
    Try to set negotiateServerCredential to false to have a one shot kerberos (this requires your server to run from a domain account).
    You can also use sticky sessions.


    http://webservices20.blogspot.com/
    WCF Security, Performance And Testing Blog
    • Marked as answer by Dan0001 Wednesday, January 21, 2009 4:54 PM
    Tuesday, January 20, 2009 3:33 PM
  • it's just plain old (but cheap!) NLB in windows server 2003 ;-)

    sticky sessions definitely makes sense, i will try negotiateServerCredential and also separately, i will suspend one of the hosts of the cluster and see if that helps. i have a feeling that both IIS instances run using network service though, does that qualify as a domain account as i think it is "seen by the network" as the computer account?
    Tuesday, January 20, 2009 3:46 PM
  • I think this should work.

    Anyway suspending one of the hosts can focus us on the problem.

    http://webservices20.blogspot.com/
    WCF Security, Performance And Testing Blog
    Tuesday, January 20, 2009 5:15 PM
  • Sorry I think I am getting lost again!

    "this requires your server to run from a domain account"

    Does Network Service qualify as a domain account here?

    The reason I ask is that I am running into a problem when setting negotiateServiceCredential to false - if I do this then I have to specify the identity element and the service principal name in it, but it has the machine name in it by default and not the DNS name for the NLB cluster so half the requests fail. If I put the DNS name in it, then it fails authentication as it is an unrecognised SPN and I found out recently the hard way that I cannot add the same SPN (i.e. the DNS name) to more than one machine account as that is a big no-no (not that it actually stops you from creating the duplicate SPN, which is annoying!).

    I am a real beginner when it comes to Kerberos so apologies for stating the obvious :)

    If I can get this working with Network Service I will be happy, but it seems as though it is not possible even though from my limited understanding it is actually a domain account.

    Also, I can continue my testing by suspending one of the hosts (will be able to confirm the problem later today), but it got me thinking - what are the real pros and cons of using negotiateServiceCredential (i.e. sessions) and turning on sticky sessions versus turning it off and having no sessions. I can think of the obvious, scalability with the latter versus convenience of the former, but I imagine there are many hidden surprises!



    Wednesday, January 21, 2009 8:57 AM
  • If you have sicky sessions enabled then you should also turn on secure conversation. This will enable to use a symmetric encryption which is faster. I think turning off secure sessions and turning on negotiation will give same effect. There are other considerations for the negotiations, among them interoperability, authentication mode (ntlm vs. kerberos) and more. I think it is better to open a new thread on this.
    http://webservices20.blogspot.com/
    WCF Security, Performance And Testing Blog
    Wednesday, January 21, 2009 10:47 AM
  •  Ok cool, I can do a new thread for the authentication considerations, but I still don't understand about the domain account bit and the potential need for SPNs (see beginning of last post from me today above) - is it ok to continue this question in this thread?
    Wednesday, January 21, 2009 11:00 AM
  • Dan

    I found this:

    In case of Kerberos credentials, setting negotiateServiceCredential="false" results in on-shot Kerberos or sometimes called Kerberos direct. For this mode to work the service must be running under machine SPN (NetworkService, LocalService accounts etc) and client must specify the SPN in the endpoint identity.


    So the answer is that you should be able to use network service.

    http://webservices20.blogspot.com/
    WCF Security, Performance And Testing Blog
    Wednesday, January 21, 2009 2:08 PM
  • Hi Dan,

    I think I ran into the same issue some time ago, it happens when you try to enable Secure conversations in web farms. The solution in that case is to use cookie SCTs, see this post for more information http://weblogs.asp.net/cibrax/archive/2008/07/29/wcf-issue-with-secure-conversation-in-web-farms.aspx

    Regards,
    Pablo.
    Pablo Cibraro - http://weblogs.asp.net/cibrax
    Wednesday, January 21, 2009 2:10 PM
  • Yaron, thanks for the link! I think though having done some more tests that it will not help me all the way. I think this is what is happening from what I have tested today:-

    I use a DNS cluster name in the endpoint address so I immediately have a problem with the value of the SPN element. I can put host/machine1name, but then it does fail on the requests that get forwarded to machine2 (in the cluster). I replaced machine1name with DNS name, but of course that isn't recognised by windows auth as it doesn't really exist. I want to register the dns record as an spn against both the machine accounts (as both services are running under Network Service), but that is not possible as you can only register the spn against one account. So I ended up creating a domain service account for both servers in the cluster to run the app pool that hosts the service and registered the spn against that. I then turned off sticky sessions and set negotiateServiceCredential to false and could finally add in an identity-spn element with the dns record (host/dnsname). It seems to be working so far, but what a pain!

    Pablo, it looks as if the SCT cookie is specifically designed for this scenario (have not seen it before!). I will have to give that a try as well.

    For now, I think I will go back to sessions and turn sticky sessions back on as it seems the most stable (and I want to leave work today at a normal time :)

    Thanks guys for all your help, I have definitely learned a huge amount from this problem! As I am going with the sticky sessions solution I will mark that as the answer even though the cookie one works too, hope that is ok with you :)

    Cheers
    Dan

    Wednesday, January 21, 2009 4:53 PM
  • Does your load balancer support sticky sessions?
    If not then the problem is probably that some requests go to one server and some to another.
    Note that even though you logically send one request there are a few infrastructure messages that are sent.
    Try to set negotiateServerCredential to false to have a one shot kerberos (this requires your server to run from a domain account).
    You can also use sticky sessions.


    http://webservices20.blogspot.com/
    WCF Security, Performance And Testing Blog
    So I know it's been 10 years ... but can you explain this further? I am using a WCF service with a MembershipProvider. Moved to a load balancer that doesn't support sticky sessions and am getting this error. I don't understand where a session is required because I thought this was all a single request but if we turn the load balancer off, it works.
    Tuesday, January 8, 2019 2:30 PM