AppFabric Caching Exception "The service does not allow you to log on anonymously"

Question

I have a cache cluster with 2 hosts. I can successfully start the cluster but when I run my IIS 7 app I get the following error:

ErrorCode<ERRCA0017>:SubStatus<ES0001>:There is a temporary failure. Please retry later.

When I look at get-cachehost I see that the service on my lead host is down. From the lead host I look at the log in Event Viewer -> Application and Services Logs -> Microsoft -> Windows -> Application Server-System Service -> Admin and see the following:

<em>AppFabric Caching service crashed with exception {System.Runtime.CallbackException: Async Callback threw an exception. ---> System.IdentityModel.Tokens.SecurityTokenValidationException: The service does not allow you to log on anonymously.
  at System.ServiceModel.Security.SecurityUtils.ValidateAnonymityConstraint(WindowsIdentity identity, Boolean allowUnauthenticatedCallers)
  at System.ServiceModel.Channels.WindowsStreamSecurityUpgradeProvider.WindowsStreamSecurityUpgradeAcceptor.CreateClientSecurity(NegotiateStream negotiateStream, Boolean extractGroupsForWindowsAccounts)
  at System.ServiceModel.Channels.WindowsStreamSecurityUpgradeProvider.WindowsStreamSecurityUpgradeAcceptor.OnAcceptUpgrade(Stream stream, SecurityMessageProperty& remoteSecurity)
  at System.ServiceModel.Channels.StreamSecurityUpgradeAcceptorBase.AcceptUpgrade(Stream stream)
  at Microsoft.ApplicationServer.Caching.VelocityStreamSecurityUpgradeAcceptor.AcceptUpgrade(Stream stream)
  at System.ServiceModel.Channels.InitialServerConnectionReader.UpgradeConnection(IConnection connection, StreamUpgradeAcceptor upgradeAcceptor, IDefaultCommunicationTimeouts defaultTimeouts)
  at System.ServiceModel.Channels.ServerSessionPreambleConnectionReader.ServerFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
  at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
  at Microsoft.ApplicationServer.Caching.WcfServerChannel.Listen(IAsyncResult result)
  at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
  --- End of inner exception stack trace ---
  at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
  at System.Runtime.InputQueue`1.AsyncQueueWaiter.Set(Boolean itemAvailable)
  at System.Runtime.InputQueue`1.CompleteWaiters(Boolean itemAvailable, IQueueWaiter[] waiters)
  at System.Runtime.InputQueue`1.EnqueueAndDispatch(Item item, Boolean canDispatchOnThisThread)
  at System.Runtime.InputQueue`1.EnqueueAndDispatch(T item, Action dequeuedCallback, Boolean canDispatchOnThisThread)
  at System.ServiceModel.Channels.InputQueueChannelAcceptor`1.EnqueueAndDispatch(TChannel channel, Action dequeuedCallback)
  at System.ServiceModel.Channels.ConnectionOrientedTransportManager`1.OnHandleServerSessionPreamble(ServerSessionPreambleConnectionReader serverSessionPreambleReader, ConnectionDemuxer connectionDemuxer)
  at System.ServiceModel.Channels.ConnectionDemuxer.OnSessionPreambleKnown(ServerSessionPreambleConnectionReader serverSessionPreambleReader)
  at System.ServiceModel.Channels.ServerSessionPreambleConnectionReader.ContinueReading()
  at System.ServiceModel.Channels.ServerSessionPreambleConnectionReader.StartReading(Action`1 viaDelegate, TimeSpan receiveTimeout)
  at System.ServiceModel.Channels.ConnectionDemuxer.OnDuplexConnection(IConnection connection, Action connectionDequeuedCallback, Int64 streamPosition, Int32 offset, Int32 size, TimeSpan timeout)
  at System.ServiceModel.Channels.ConnectionDemuxer.OnConnectionModeKnownCore(ConnectionModeReader modeReader, Boolean isCached)
  at System.ServiceModel.Channels.ConnectionModeReader.ReadCallback(Object state)
  at System.ServiceModel.Channels.SocketConnection.AsyncReadCallback(Boolean haveResult, Int32 error, Int32 bytesRead)
  at System.Runtime.Fx.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
  at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)}. Check debug log for more information</em>

There is no firewall between these machines and I know they are communication because I can start and stop the cluster from either machine.

Here is the relevant portions of the cache config:

 

<configuration>
- <configSections>
 <section name="dataCache" type="Microsoft.ApplicationServer.Caching.DataCacheSection, Microsoft.ApplicationServer.Caching.Core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
 </configSections>
- <dataCache size="Small">
- <caches>
- <cache consistency="StrongConsistency" name="default">
- <policy>
 <eviction type="Lru" />
 <expiration defaultTTL="10" isExpirable="true" />
 </policy>
 </cache>
 </caches>
- <hosts>
 <host replicationPort="22236" arbitrationPort="22235" clusterPort="22234" hostId="1517955733" size="2048" leadHost="true" account="EXT\machine1$" cacheHostName="AppFabricCachingService" name="r52brweb04" cachePort="22233" />
 <host replicationPort="22236" arbitrationPort="22235" clusterPort="22234" hostId="1350565914" size="2048" leadHost="false" account="EXT\machine2$" cacheHostName="AppFabricCachingService" name="r52brweb05" cachePort="22233" />
 </hosts>
- <advancedProperties>
- <securityProperties>
- <authorization>
 <allow users="IIS APPPOOL\App-1" />
 <allow users="IIS APPPOOL\App-2" />
 </authorization>
 </securityProperties>
 </advancedProperties>
 </dataCache>
 </configuration>

 

Answer 1

It looks like you're allowing the IIS AppPool identities App-1 and App-2 access to your cache cluster. I haven't tried this yet in my own tests. In my tests, I've just used the machine account for the web server. So, for example, if your web server was WebServer1 on the DOMAIN1 domain, then you would run Grant-CacheAllowClientAccount for DOMAIN1\WebServer1$.

Can you try granting access to the machine account of the web server(s) and see if that helps?

If not, are you using AppFabric caching for the session state provider? Or are you calling the APIs directly? If you're still having problems, maybe copy out the relevant sections of the web.config (for session state provider) or explain what APIs you're calling.

Thanks.

Jason Roth

Answer 2

Thanks for the reply Jason. I will give this a try. We are not using this for session state, the web applications are actually caching service responses and we are calling the API's directly. I thought since the web app would be doing this as the app pool identity, those would be the accounts that need access. I will try it with the machine account and let you know if that resolves this.

Answer 3

I gave this a shot but I get eh following error:

Server channel security authorization failed for client {IIS APPPOOL\ePayments-Unit-2}. To grant access, use Grant-CacheAllowedClientAccount command.

My web applications are running as app pool identity.

Answer 4

Thanks for the additional info and test. I'd like to find time to run a test on this to see if I can reproduce the problem. But first, could you please try one more thing. I want to see if the issue is some how related to App Pool identities or whether there is something else going on. Can you temporarily change the identity of your application pools to Network Service? You could also try a domain account. For Network Service, grant the machine account access to the cluster. For the domain account, obviously grant that account access to the cluster. Restart the application pool both times and try your API calls. If these work, then there is something specific about using app pool identities that is related to the problem. If they do not work, then we've eliminated that variable.

Note that for testing purposes, you might be able to work around this issue by temporarily disabling security:

Set-CacheClusterSecurity -ProtectionLevel None -SecurityMode None

Then once we figure this out, you can set it back to the default:

Set-CacheClusterSecurity -ProtectionLevel EncryptAndSign -SecurityMode Transport

Let me know the results and thanks for your patience as we look at this.

Jason Roth

Answer 5

You probably are using ApplicationPoolIdentity for your app pool which is not supported. Only Windows authentication is supported. You also need to change the identity for application pool to NetworkService alongwith the above change.

Agreed this is a bug as the exception is not handled in service.

Answer 6

Hello Rahul,

the solution is not completely clear, can you please add more details? You say that ApplicationPoolIdentity is not supported and that only Windows Authentication is supported instead, then you say that the application pool should be changed to use NetworkService.

What exactly needs to be changed? I mean, the client ASP.NET application accessing the AppFabric cache should be configured to use NetworkService as its application pool account, but what does this have to do with Windows Authentication?

Thanks