Cluster routinely stops RRS feed

  • Question

  • My AppFabric cluster (SQL server configuration) routines shutdowns after some period of inactivity. I see this both on my local development cluster and an actual deployment environment. Is this normal behavior? I look in the event viewer but nothing obvious jumps out.

    Friday, July 23, 2010 3:00 PM

All replies

  • Hi. I just checked a test AppFabric Cache Cluster and it has been running for about a week now with no problems. I don't believe that you should just see it shut down. You should be aware though, that the AppFabric Caching Service is not set to automatically start after reboot. You have to start it with Start-CacheCluster or Start-CacheHost in PowerShell. So if you reboot the caching servers, you will see the cluster as down. Finally, there may be caching events in the event log that you haven't found yet. I'm pasting a screenshot below to show where to look. Look in both the Admin and Operational logs under:

    • Applications and Services Logs
    • Microsoft
    • Windows
    • Application Server-System Services

    Jason Roth

    Tuesday, July 27, 2010 5:09 PM
  • Thank you very much for the heads up on the alternate location in the event viewer. I do see a crash related to the external store. This only happens around midnight based on the logs so I am not sure what could be going on in the test environment. I will have to investigate some more.


    AppFabric Caching service crashed.{Lease with external store expired: Microsoft.Fabric.Federation.ExternalRingStateStoreException: Renew lease failed ---> Microsoft.Fabric.Data.ExternalStoreException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> Microsoft.ApplicationServer.Caching.ConfigStoreException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

    at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)

    at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()

    at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)

    at System.Data.SqlClient.TdsParserStateObject.ReadSni(DbAsyncResult asyncResult, TdsParserStateObject stateObj)

    at System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()

    at System.Data.SqlClient.TdsParser.ConsumePreLoginHandshake(Boolean encrypt, Boolean trustServerCert, Boolean& marsCapable)

    at System.Data.SqlClient.TdsParser.Connect(ServerInfo serverInfo, SqlInternalConnectionTds connHandler, Boolean ignoreSniOpenTimeout, Int64 timerExpire, Boolean encrypt, Boolean trustServerCert, Boolean integratedSecurity)

    at System.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, SqlConnection owningObject)

    at System.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, Boolean redirectedUserInstance, SqlConnection owningObject, SqlConnectionString connectionOptions, TimeoutTimer timeout)

    at System.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(SqlConnection owningObject, TimeoutTimer timeout, SqlConnectionString connectionOptions, String newPassword, Boolean redirectedUserInstance)

    at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, Object providerInfo, String newPassword, SqlConnection owningObject, Boolean redirectedUserInstance)

    at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection)

    at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnection owningConnection, DbConnectionPool pool, DbConnectionOptions options)

    at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject)

    at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject)

    at System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection owningObject)

    at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)

    at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)

    at System.Data.SqlClient.SqlConnection.Open()

    at Microsoft.ApplicationServer.Caching.SqlServerCustomProvider.BeginTransaction()

    --- End of inner exception stack trace ---

    at Microsoft.ApplicationServer.Caching.SqlServerCustomProvider.BeginTransaction()

    at Microsoft.ApplicationServer.Caching.ExternalConfigStore.BeginTransaction()

    --- End of inner exception stack trace ---

    at Microsoft.ApplicationServer.Caching.ExternalConfigStore.BeginTransaction()

    at Microsoft.Fabric.Data.ExternalStoreAuthority.UpdateNode(NodeInfo nodeInfo, TimeSpan timeout)

    --- End of inner exception stack trace ---

    at Microsoft.Fabric.Data.ExternalStoreAuthority.UpdateNode(NodeInfo nodeInfo, TimeSpan timeout)

    at Microsoft.Fabric.Federation.SiteNode.PerformExternalRingStateStoreOperations(Boolean& canFormRing, Boolean isInsert, Boolean isJoining)}

    Tuesday, July 27, 2010 7:48 PM
  • So are you using SQL Server for your configuration store provider (System.Data.SqlClient provider)? It looks like in the callstack that an attempt is being made to connect with SQL Server. I'm not sure why or even if this would cause a crash if the connection could not be made. But you might want to look at your SQL Server error log to see if anything was happening on the SQL Server at the same time as the AppFabric caching service crash. If you find something, let me know. I'll also see if I can find anything from the clues in your callstack.


    Jason Roth

    Tuesday, July 27, 2010 9:55 PM