Configuraton services do not goes into faulted state & retries when it cannot connect to repository database.

Question

Hi All,

I have a configuration service running on one machine & repository database is on other machine.

when the services cannot be able to connect to repository database & throws an SQL Exception due to any dependency issue or network latency related issue.

The services were not gone into faulted state & its retry mechanism for re-starting services will not come into effect.

it just lock the exception & starts successfully. Why services are not re-tries so that if in one attempt it fails it should retry again after some time delay?

exception is getting logged into "Application event log" as 

****************************************************************

Event Type: Error

Event Source: Configuration Management Service

Event Category: None

Event ID: 0

User: N/A

Computer: RPGENSER

Description:

The application is currently unable to initialize from it's SQL Server Configuration configuration database, and is unable to start. Please ensure that (a) You have correctly installed the SQL Server database for the configuration database---see the StockTrader installation document; and (b) your connection strings in your config file (web.config for Web applictions and/or exe.config for the Service Host Console applications) are pointing to the correct configuration databases with proper login credentials. The exception is: 

System.Data.SqlClient.SqlException: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)

   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection)

   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)

   at System.Data.SqlClient.TdsParser.Connect(ServerInfo serverInfo, SqlInternalConnectionTds connHandler, Boolean ignoreSniOpenTimeout, Int64 timerExpire, Boolean encrypt, Boolean trustServerCert, Boolean integratedSecurity, SqlConnection owningObject)

   at System.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, Boolean ignoreSniOpenTimeout, Int64 timerExpire, SqlConnection owningObject)

   at System.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(String host, String newPassword, Boolean redirectedUserInstance, SqlConnection owningObject, SqlConnectionString connectionOptions, Int64 timerStart)

   at System.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(SqlConnection owningObject, SqlConnectionString connectionOptions, String newPassword, Boolean redirectedUserInstance)

   at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, Object providerInfo, String newPassword, SqlConnection owningObject, Boolean redirectedUserInstance)

   at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection)

   at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnection owningConnection, DbConnectionPool pool, DbConnectionOptions options)

   at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject)

   at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject)

   at System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection owningObject)

   at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)

   at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)

   at System.Data.SqlClient.SqlConnection.Open()

   at ConfigService.ServiceConfiguration.DALSQLServer.Configuration.Open(String connString)

   at ConfigService.ServiceConfigurationHelper.ServiceConfigHelper.InitializeConfigFromDatabase(Boolean primaryHost, List`1& connectionPointList, List`1& connectedServiceList, List`1& connectedConfigServiceList, List`1& hostedServiceList, List`1& serviceHostList, String connString)

The Service host is NOT STARTED.  Please exit.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

********************************************************************

The services are actually in faulted state & do not work as the repository database setting object is missing.

Why services are not in faulted state?

Is there any way by which it retries for such SQL exception.

Waiting your reply

Thanks & Regards,

Amit.Saraf

Answer 1

Hi All,

Further investigating this issue in configuration service open source code in version 2.04

we found that the errors are getting logged but the services was not put into faulted state.

If your hosting service is available on separate physical system and repository database is available on separate physical system and your hosting service cannot be able to connect to this repository databases due to  any network related issue or due to some service dependency issue then services should be placed into faulted state so that thy will retry one more time after some specific time delay.

But in actual open source code the errors are getting logged from 

Assembly:- ServiceConfigurationHelper

Code file:-ServiceConfigHelper_MainMethods.cs

method name:-InitializeConfigFromDatabase

In this method the try_catch block the error is getting logged in catch block & the exception is bubbled back to

Assembly:- ServiceHostShellConsoleBase

code file:- ShellServiceConsoleBase.cs

method name:- startNTService

Where the exception is handled but no action was taken against it.

so i simply try to modified some code & try to generate a possible fault exception in open source code 

Assembly:- ServiceConfigurationHelper

code file:-ServiceConfigHelper_MasterHosts.cs

methode name :-MasterServiceSelfHost

catch block of above method is modified as 

 catch(Exception ex)

 //<Amit>:-tried to throw but not come into effect.

{

       started = false;

     ConfigUtility.writeConsoleMessage("\n raising a fault exception & restarting services ", EventLogEntryType.Error, true, settingsInstance);

       //throw;

  //<Amit>:-tried to trow but not come into effect.

     throw new FaultException(new FaultReason(ex.Message));

  }

Fault event was sucesssfully fired but services were not restarted & no loges found in event log related to this.

IS there any possible modification required into open source code that will restart the entire configuration process from scratch.{ which also involves hitting to repository database & also restarting config,node & nodeDC services}.

Thanks & Regards,

Amit.Saraf