Connection Pooling randomly throwing COM exceptions

Question

 

We randomly get the following exceptions:

[COMException (0x80070006): The handle is invalid. (Exception from HRESULT: 0x80070006 (E_HANDLE))]
   System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo) +0
   System.Runtime.InteropServices.Marshal.ThrowExceptionForHR(Int32 errorCode) +34
   System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection owningObject) +636
   System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection) +82
   System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory) +105
   System.Data.SqlClient.SqlConnection.Open() +111
   System.Data.Common.DbDataAdapter.FillInternal(DataSet dataset, DataTable[] datatables, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior) +121
   System.Data.Common.DbDataAdapter.Fill(DataSet dataSet, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior) +137
   System.Data.Common.DbDataAdapter.Fill(DataSet dataSet) +86

and

[UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))]
   System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo) +0
   System.Runtime.InteropServices.Marshal.ThrowExceptionForHR(Int32 errorCode) +34
   System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection owningObject) +636
   System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection) +82
   System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory) +105
   System.Data.SqlClient.SqlConnection.Open() +111
   System.Data.Common.DbDataAdapter.FillInternal(DataSet dataset, DataTable[] datatables, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior) +121
   System.Data.Common.DbDataAdapter.Fill(DataSet dataSet, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior) +137
   System.Data.Common.DbDataAdapter.Fill(DataSet dataSet) +86

Before the Access Denied exception is thrown, the user is prompted for credentials - none work, not even admin.

 

Also, occasionally, pooling will throw a SemaphoreFullException:

[SemaphoreFullException: Adding the given count to the semaphore would cause it to exceed its maximum count.]
   System.Threading.Semaphore.Release(Int32 releaseCount) +1853063
   System.Data.ProviderBase.DbConnectionPool.PutNewObject(DbConnectionInternal obj) +54
   System.Data.ProviderBase.DbConnectionPool.DeactivateObject(DbConnectionInternal obj) +228
   System.Data.ProviderBase.DbConnectionPool.PutObject(DbConnectionInternal obj, Object owningObject) +265
   System.Data.ProviderBase.DbConnectionInternal.CloseConnection(DbConnection owningObject, DbConnectionFactory connectionFactory) +97
   System.Data.SqlClient.SqlConnection.Close() +117
   System.Data.Common.DbDataAdapter.QuietClose(IDbConnection connection, ConnectionState originalState) +13
   System.Data.Common.DbDataAdapter.FillInternal(DataSet dataset, DataTable[] datatables, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior) +285
   System.Data.Common.DbDataAdapter.Fill(DataSet dataSet, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior) +137
   System.Data.Common.DbDataAdapter.Fill(DataSet dataSet) +86

 

Obviously, if I turn off pooling, I don't get these exceptions - but performance suffers noticeably.

 

These exceptions don't always happen. The web app will run fine for hours before an exception is thrown. It could also be as short as the next request. . It randomly occurs and in random places in the code. Once thrown, every request afterwards throws the "Access Denied" exception.

Using basic connection string: "server=xxx.xxx.xxx.xxx;database=xxx;uid=xxx;pwd=xxx;app=xxx".

 

I have no idea how to go about troubleshooting this. Any help or ideas would be greatly appreciated.

Answer 1

This exception is being thrown from the pool's syncronization primitives, and should not occur in normal circumstances.  A serious configuration or installation error might cause them.  It could also be a bug in the pool itself -- the randomness of time before hitting the problem would be consistent with a race condition, although something this far into the core of the pooler should have shown up before now.  I'll look into the this possibility.

If this isn't a production machine, I suggest trying to eliminate the config/installation problem by re-installing the .Net Framework (there are probably less drastic measures -- anyone who knows one is urged to chime in :-).

Is there anything else you can tell about your code or environment that might be significant?  Does the problem happen only under a heavy load?  Is your app multi-threaded and sharing state between threads, either explicitly or via sharing connections between host threads such as ASP.Net?

One method to attack the problem would be to use ETW tracing.  See this link for details: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnadonet/html/tracingdataaccess.asp

Answer 2

While it is on a production machine, it is in its own application pool and only a handful of people are using it (usability testing). It is also on my local develeopment machine (XP) and I get the same exceptions, albeit much less frequently. Therefore, I do not believe it to be an installation/configuration issue - althought I won't dismiss any suggestions of something to try. I will re-install the .net framework during the next downtime we have for that box.

I'm not sure what would be significant, but....

I wrote a connection string provider library the keep connection string information in either the registry or in a heavily protected database. The password is encrytped with some custom encryption.

I wrote a facade class around SqlConnection to make working with the connection string provider easier for the other developers. They only need to know the database alias. I also used this facade class to do custom serialization (serializes the database alias) so that the ManagedConnection:SqlConnection can be serialized with the objects that use them. My money is on this class causing the problem, although I don't have a clue how.

The ManagedConnection class constructors create a single SqlConnection object that is used for the life of the object. Each of the methods that use the SqlConnection uses a try/catch/finally block to Open/Close the connection properly.

    [Serializable]
    public sealed class ManagedConnection : ISerializable
    {
        private SqlConnection conn;
        private string alias, appName;
        private ConnectionStringFormat format;

        public ManagedConnection(string DatabaseAlias, ConnectionStringFormat Format)
        {
            alias = DatabaseAlias;
            format = Format;
            conn = new SqlConnection(ConnectionStringProvider.GetConnectionString(alias, format));
        }

        public ManagedConnection(string DatabaseAlias, ConnectionStringFormat Format, string ApplicationName)
        {
            alias = DatabaseAlias;
            appName = ApplicationName;
            format = Format;
            conn = new SqlConnection(ConnectionStringProvider.GetConnectionString(alias, format) + "; app=" + appName);
        }

        //deserialization constructor
        private ManagedConnection(SerializationInfo info, StreamingContext context)
        {
            alias = info.GetString("Alias");
            format = (ConnectionStringFormat)info.GetValue("Format", typeof(ConnectionStringFormat));
            appName = info.GetString("AppName");
            conn = new SqlConnection(ConnectionStringProvider.GetConnectionString(alias, format) + ";app=" + appName);
        }

        public ConnectionState ConnectionState
        {
            get { return conn.State; }
        }

        public string DatabaseAlias
        {
            get { return alias; }
        }

        public string ApplicationName
        {
            get { return appName; }
        }

        public ConnectionStringFormat ConnectionStringFormat
        {
            get { return format; }
        }

        // serialization method
        [SecurityPermissionAttribute(SecurityAction.Demand, SerializationFormatter = true)]
        public void GetObjectData(SerializationInfo info, StreamingContext context)
        {
            info.AddValue("Alias", alias);
            info.AddValue("Format", format);
            info.AddValue("AppName", appName);
        }

        public DataSet ExecuteProcedure_DataSet(string SPName, List<SqlParameter> SqlParameters)
        {
            SqlCommand cmd;
            SqlDataAdapter sda;
            DataSet ds = new DataSet();

            cmd = conn.CreateCommand();
            cmd.CommandType = CommandType.StoredProcedure;
            cmd.CommandText = SPName;

            if(SqlParameters != null)
                foreach (SqlParameter p in SqlParameters)
                    cmd.Parameters.Add(p);

            sda = new SqlDataAdapter(cmd);

            try
            {
                sda.Fill(ds);

                return ds;
            }

            catch (Exception ex)
            {
                string parmValues = string.Empty;

                if (SqlParameters != null)
                    foreach (SqlParameter p in cmd.Parameters)
                        parmValues += "\r\n" + p.ParameterName + " = " + (p.Value == null ? "null" : p.Value.ToString());

                throw new Exception("\r\nStored Procedure: " + cmd.CommandText + parmValues + "\r\n", ex);
            }

            finally
            {
                sda.Dispose();
                cmd.Parameters.Clear();
                cmd.Dispose();
                conn.Close();
            }
        }

Answer 3

Sorry, forgot to answer your other questions...

The problem happens with only 1 user (me) running it. There isn't much of a load yet. The application is a trouble ticket system that integrates our billing system. The application connects up to 8 different databases on two different servers. Tickets and the billing system information are represented as objects that are collections of objects that are collections of objects. In addition, I wrote a custom security library to enable applying permissions on the various objects/functions. Each permission check is a hit against the security database. It's nothing for a single http request to generate 50 hits to various databases for permission checks, building the current ticket, building the current billing customer/plan.

The app itself isn't multi-threaded. Isn't being a web app implicitly multi-threaded?

I'm not sure what you mean by "via sharing connections between host threads such as ASP.net". Is a host thread the worker process thread or the thread for a particular request? Is there only one connection pool for the entire application or does each request get its own pool? Even if they all share the same pool, I seem to remember reading somewhere that each connection in the pool is not only associated with a connection string, but the user/identity/whatever as well.

I'm relatively new to multi-threading and discussions about it are still above my head. Should I do some kind of synchonizing on the ManagedConnection objects? I don't think the same ManagedConnection object would be used by more than one thread at a time, but I don't understand enough to know for certain.

 

Answer 4

Ok, I no longer think it's my code causing the problem.

I've been running with pooling turned off for the ManagedConnection connections. However, I have the SessionState mode set to SqlServer and pooling is not disabled. I just got the access denied exception:

Server Error in '/' Application.
--------------------------------------------------------------------------------

Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))

ASP.NET is not authorized to access the requested resource. Consider granting access rights to the resource to the ASP.NET request identity. ASP.NET has a base process identity (typically {MACHINE}\ASPNET on IIS 5 or Network Service on IIS 6) that is used if the application is not impersonating. If the application is impersonating via <identity impersonate="true"/>, the identity will be the anonymous user (typically IUSR_MACHINENAME) or the authenticated request user.

To grant ASP.NET access to a file, right-click the file in Explorer, choose "Properties" and select the Security tab. Click "Add" to add the appropriate user or group. Highlight the ASP.NET account, and check the boxes for the desired access.

Source Error:

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below. 

Stack Trace:

[UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))]
   System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo) +0
   System.Runtime.InteropServices.Marshal.ThrowExceptionForHR(Int32 errorCode) +34
   System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection owningObject) +636
   System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection) +82
   System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory) +105
   System.Data.SqlClient.SqlConnection.Open() +111
   System.Web.SessionState.SqlStateConnection..ctor(SqlPartitionInfo sqlPartitionInfo) +79

[HttpException (0x80004005): Unable to connect to SQL Server session database.]
   System.Web.SessionState.SqlSessionStateStore.ThrowSqlConnectionException(SqlConnection conn, Exception e) +227
   System.Web.SessionState.SqlStateConnection..ctor(SqlPartitionInfo sqlPartitionInfo) +349
   System.Web.SessionState.SqlSessionStateStore.GetConnection(String id, Boolean& usePooling) +282
   System.Web.SessionState.SqlSessionStateStore.DoGet(HttpContext context, String id, Boolean getExclusive, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actionFlags) +119
   System.Web.SessionState.SqlSessionStateStore.GetItemExclusive(HttpContext context, String id, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actionFlags) +50
   System.Web.SessionState.SessionStateModule.GetSessionStateItem() +114
   System.Web.SessionState.SessionStateModule.BeginAcquireState(Object source, EventArgs e, AsyncCallback cb, Object extraData) +1019
   System.Web.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +90
   System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +155

 

--------------------------------------------------------------------------------
Version Information: Microsoft .NET Framework Version:2.0.50727.42; ASP.NET Version:2.0.50727.42

Answer 5

I did ETW tracing, but I didn't notice anything obviously wrong. I don't know what I'm looking for, but here's a trace of a succesful call to the stored procedure Folder_Load:

System.Data, "<ds.DataSet.DataSet|API> 5#"
System.Data, "<ds.DataTableCollection.DataTableCollection|INFO> 5#  dataSet=5"
System.Data, "<sc.SqlCommand.set_CommandText|API> 28#  '"
System.Data, "
System.Data, "'"
System.Data, "<sc.SqlCommand.set_Connection|API> 28#  3#"
System.Data, "<sc.SqlCommand.set_CommandType|API> 28#  4{ds.CommandType}"
System.Data, "<sc.SqlCommand.set_CommandText|API> 28#  '"
System.Data, "Folder_Load"
System.Data, "'"
System.Data, "enter_01 <comm.DbDataAdapter.Fill|API> 5#  dataSet"
System.Data, "enter_02 <comm.DbDataAdapter.Fill|API> 5#  dataSet  startRecord  maxRecords  srcTable  command  behavior=16{ds.CommandBehavior}"
System.Data, "enter_03 <sc.SqlConnection.Open|API> 3#"
System.Data,, "<sc.TdsParser.Connect|SEC> SQL authentication"
System.Data.SNI, "ObtainIDa 8# <SNI_Conn::SNI_Conn|ID|SNI> 00E27938{.}"
System.Data.SNI, "ObtainIDa 9# <Tcp::Tcp|ID|SNI> 00E27A30{.} created by 8#{SNI_Conn}"
System.Data.SNI, "UpdateIDa 9# <Tcp::Open|ID|SNI> connection: 0EA08060{ProtElem}"
System.Data.SNI, "UpdateIDa 8# <SNIOpenEx|ID|SNI> connection string: '127.0.0.1'"
System.Data.SNI, "ObtainIDa 10# <Ssl::Ssl|ID|SNI> 00E2AF78{.} created by 8#{SNI_Conn}"
System.Data.SNI, "RecycleIDa 10# <Ssl::~Ssl|ID|SNI>"
System.Data, "<prov.DbConnectionFactory.CreatePooledConnection|RES|CPOOL> 1#  Pooled database connection created."
System.Data, "leave_03"
System.Data, "<sc.SqlCommand.ExecuteReader|INFO> 28#  Command executed as RPC."
System.Data.SNI, "ObtainIDa 11# <SNI_Packet::SNI_Packet|ID|SNI> 00E27420{.}"
System.Data, "enter_03 <comm.DataAdapter.Fill|API> 5#  dataSet  srcTable  dataReader  startRecord  maxRecords"
System.Data, "<ds.DataTable.DataTable|API> 5#"
System.Data, "<ds.DataColumn.DataColumn|API> 9#  columnName='FolderID'  expr='(null)'  type=1{ds.MappingType}"
System.Data, "<ds.DataColumn.DataColumn|API> 10#  columnName='FolderName'  expr='(null)'  type=1{ds.MappingType}"
System.Data, "<ds.DataColumn.DataColumn|API> 11#  columnName='FolderAlias'  expr='(null)'  type=1{ds.MappingType}"
System.Data, "enter_04 <ds.DataTableCollection.Add|API> 5#  table=5"
System.Data, "<ds.DataTableCollection.RegisterName|INFO> 5#  name='Table'  tbNamespace=''"
System.Data, "leave_04"
System.Data, "enter_04 <ds.DataTable.BeginLoadData|API> 5#"
System.Data, "<ds.DataTable.SuspendIndexEvents|Info> 5#  0"
System.Data, "enter_05 <ds.DataSet.set_EnforceConstraints|API> 5#  0{bool}"
System.Data, "leave_05"
System.Data, "leave_04"
System.Data, "enter_04 <sc.SqlDataReader.Read|API> 5#"
System.Data, "leave_04"
System.Data, "enter_04 <ds.DataTable.LoadDataRow|API> 5#  fAcceptChanges=1{bool}"
System.Data, "<ds.DataRow.set_RBTreeNodeId|INFO> 5#  value=1"
System.Data, "enter_05 <ds.DataRow.AcceptChanges|API> 5#"
System.Data, "leave_05"
System.Data, "leave_04"
System.Data, "enter_04 <sc.SqlDataReader.Read|API> 5#"
System.Data, "leave_04"
System.Data, "enter_04 <ds.DataTable.EndLoadData|API> 5#"
System.Data, "<ds.DataTable.RestoreIndexEvents|Info> 5#  1"
System.Data, "enter_05 <ds.DataSet.set_EnforceConstraints|API> 5#  1{bool}"
System.Data, "enter_06 <ds.DataSet.EnableConstraints|INFO> 5#"
System.Data, "leave_06"
System.Data, "leave_05"
System.Data, "leave_04"
System.Data, "enter_04 <sc.SqlDataReader.NextResult|API> 5#"
System.Data, "enter_05 <sc.SqlDataReader.Read|API> 5#"
System.Data, "leave_05"
System.Data, "leave_04"
System.Data, "leave_03"
System.Data, "enter_03 <sc.SqlDataReader.Close|API> 5#"
System.Data, "leave_03"
System.Data, "enter_03 <sc.SqlConnection.Close|API> 3#"
System.Data, "leave_03"
System.Data, "leave_02"
System.Data, "leave_01"
System.Data, "enter_01 <sc.SqlConnection.Close|API> 3#"
System.Data, "leave_01"

and here's a trace where the UnauthorizedAccessException is thrown:

System.Data, "<ds.DataSet.DataSet|API> 12893#", 0, 0
System.Data, "<ds.DataTableCollection.DataTableCollection|INFO> 12893#  dataSet=12893", 0, 0
System.Data, "<sc.SqlCommand.set_CommandText|API> 14279#  '", 0, 0
System.Data, "p", 0, 0
System.Data, "'", 0, 0
System.Data, "<sc.SqlCommand.set_Connection|API> 14279#  623#", 0, 0
System.Data, "<sc.SqlCommand.set_CommandType|API> 14279#  4{ds.CommandType}", 0, 0
System.Data, "<sc.SqlCommand.set_CommandText|API> 14279#  '", 0, 0
System.Data, "Folder_Load", 0, 0
System.Data, "'", 0, 0
System.Data, "enter_01 <comm.DbDataAdapter.Fill|API> 12893#  dataSet", 0, 0
System.Data, "enter_02 <comm.DbDataAdapter.Fill|API> 12893#  dataSet  startRecord  maxRecords  srcTable  command  behavior=16{ds.CommandBehavior}", 0, 0
System.Data, "enter_03 <sc.SqlConnection.Open|API> 623#", 0, 0
System.Data, "leave_03", 0, 0
System.Data, "enter_03 <sc.SqlConnection.Close|API> 623#", 0, 0
System.Data, "leave_03", 0, 0
System.Data, "leave_02", 0, 0
System.Data, "leave_01", 0, 0
System.Data, "enter_01 <sc.SqlConnection.Close|API> 623#", 0, 0
System.Data, "leave_01", 0, 0

I notice that the connection was immediately closed in the 2nd trace, but nothing that indicates (to me) why.

Does this help?

Answer 6

Just a note that re-installing the .net 2.0 framework did not fix the problem.

So.. If it's not an installation/configuration issue, that brings it back to my code.

But since I get the same exceptions thrown when using the session state (mode = SqlServer), how can it be my code?

A bug in the framework? Then why does it appear that I'm the only one experiencing this evil?

The ETW trace doesn't seem to show anything that helps.

 

Are you still reading this Alazel? What else can I try? What other information do you need? I don't understand why this is happening, but I really need to get this resolved soon as this app needs to go live.

Please, any help is appreciated.

Anyone?

 

Answer 7

*bump*

Ok, nobody knows what this is.

What would you do to troubleshoot?

 

Answer 8

We are also experiencing this problem.

It appears to be a race in the DbConnectionPool.

There are two ways that this problem occurs:
1. a call to DbConnectionPool.CleanupCallback(Object)
2. a call to DbConnectionPool.PutNewObject(Object)

It would seem that the PutNewObject is a more likely candidate for a problem since it seems that it just does a semaphore Release with no apparent matching WaitOne.

Here are two stack traces that show the problem:
=============
I. First StackTrace
=============
System.Threading.SemaphoreFullException was unhandled
  Message="Adding the given count to the semaphore would cause it to exceed its maximum count."
  Source="System"
  StackTrace:
       at System.Threading.Semaphore.Release(Int32 releaseCount)
       at System.Data.ProviderBase.DbConnectionPool.CleanupCallback(Object state)
       at System.Threading._TimerCallback.TimerCallback_Context(Object state)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading._TimerCallback.PerformTimerCallback(Object state)

===============
II. Second Stack trace
===============
<TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Critical">
  <TraceIdentifier>http://msdn.microsoft.com/TraceCodes/System/ActivityTracing/2004/07/Reliability/Exception/Unhandled</TraceIdentifier>
  <Description>Unhandled exception</Description>
  <AppDomain>/LM/W3SVC/1/Root/vbps-1-127983026559926507</AppDomain>
  <Exception>
    <ExceptionType>System.Threading.SemaphoreFullException, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
    </ExceptionType>
    <Message>Adding the given count to the semaphore would cause it to exceed its maximum count.</Message>
    <StackTrace>   at System.Threading.Semaphore.Release(Int32 releaseCount)
      at System.Data.ProviderBase.DbConnectionPool.PutNewObject(DbConnectionInternal obj)
      at System.Data.ProviderBase.DbConnectionPool.DeactivateObject(DbConnectionInternal obj)
      at System.Data.ProviderBase.DbConnectionPool.PutObject(DbConnectionInternal obj, Object owningObject)
      at System.Data.ProviderBase.DbConnectionInternal.CloseConnection(DbConnection owningObject, DbConnectionFactory connectionFactory)
      at System.Data.SqlClient.SqlConnection.Close()
      at Jsi.Vbip.Sql.SqlMethodSupport.InvokeCommand(SqlCommand command, Boolean waitForConnection, Boolean useCommonParameters, Type returnType) in C:\new development\Vbip\Core\SqlMethodGenerator.cs:line 188
      at Jsi.Vbip.ISqlMethodsImplementation.UpdateFileLastAccessed(Guid FileID, SqlDateTime LastAccessed)
      at Jsi.Vbip.PacketStore.RepositoryFile.Close() in C:\new development\Vbip\PacketStore\RepositoryFile.cs:line 293
      at Jsi.Vbip.PacketStore.RepositoryFileManager.CheckExpiry(Object state) in C:\new development\Vbip\PacketStore\RepositoryFileManager.cs:line 587
      at System.Threading._TimerCallback.TimerCallback_Context(Object state)
      at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
      at System.Threading._TimerCallback.PerformTimerCallback(Object state)
    </StackTrace>

   <ExceptionString>System.Threading.SemaphoreFullException: Adding the given count to the semaphore would cause it to exceed its maximum count.
    at System.Threading.Semaphore.Release(Int32 releaseCount)
    at System.Data.ProviderBase.DbConnectionPool.PutNewObject(DbConnectionInternal obj)
    at System.Data.ProviderBase.DbConnectionPool.DeactivateObject(DbConnectionInternal obj)
    at System.Data.ProviderBase.DbConnectionPool.PutObject(DbConnectionInternal obj, Object owningObject)
    at System.Data.ProviderBase.DbConnectionInternal.CloseConnection(DbConnection owningObject, DbConnectionFactory connectionFactory)
    at System.Data.SqlClient.SqlConnection.Close()
    at Jsi.Vbip.Sql.SqlMethodSupport.InvokeCommand(SqlCommand command, Boolean waitForConnection, Boolean useCommonParameters, Type returnType) in C:\new development\Vbip\Core\SqlMethodGenerator.cs:line 188
    at Jsi.Vbip.ISqlMethodsImplementation.UpdateFileLastAccessed(Guid FileID, SqlDateTime LastAccessed)
    at Jsi.Vbip.PacketStore.RepositoryFile.Close() in C:\new development\Vbip\PacketStore\RepositoryFile.cs:line 293
    at Jsi.Vbip.PacketStore.RepositoryFileManager.CheckExpiry(Object state) in C:\new development\Vbip\PacketStore\RepositoryFileManager.cs:line 587
    at System.Threading._TimerCallback.TimerCallback_Context(Object state)
    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
    at System.Threading._TimerCallback.PerformTimerCallback(Object state)
    </ExceptionString>
  </Exception>
</TraceRecord>

Answer 9

alazela wrote:

....  It could also be a bug in the pool itself -- the randomness of time before hitting the problem would be consistent with a race condition, although something this far into the core of the pooler should have shown up before now.  I'll look into the this possibility. ...

Have you found anything?

 

Answer 10

I've opened up a case with Microsoft.   No progress on creating a test case.   The application can be modified to reproduce the problem in around 10 minutes (usually between 12 and 18 minutes) but it's pretty hard to send our entire application to Microsoft as it's an enterprise app with a rather difficult setup.

If you can make a test case that causes the exception, it sounds like Microsoft would fix it.

Answer 11

Sorry for the delayed response -- I've been away for a couple of weeks.

I wasn't able to track down the problem in the source nor re-produce the symptoms with simple cases.  Today I'll look through the traces you've posted to see if that helps me find anything.


VbipAcc1, can you post the case # you raised so I can see if anyone else at MS has examined the issue and found anything out?

Answer 12

Can you post the connection string you are using?  There are a couple of places that could match the call stacks given, and this might help track it down.  I don't need actual values for sensitive stuff like password, username & server, but I would like to see what options you are specifying and values for the non-sensitive options.

One possibility:  could the app be openning many connections in parallel?  The connection pool throttles connection attempts via a semaphore, and you can get exception call stacks something like this if there is a timeout while waiting for the semaphore.

Answer 13

server=xxx.xxx.xxx.xxx;uid=xxx;pwd=xxx;database=xxx

My app opens a lot of connections sequentially. It would be nothing for a single request to generate 10-30 connections to using 3 databases (2 of which are on the same server).

Each database has its own DAL-like class to access it. Each instance creates a single SqlConnection object that it uses for the lifetime of the object (web app, so until the end of the request?). At first, I would leave that SqlConnection object open for the lifetime of the DAL object thinking to avoid any overhead of opening/closing connections continuously. After the exceptions were thrown I started doing research. Common thought appears to be to let the connection pool handle everything. So, I changed to the code to call Open/Close on the (still) single SqlConnection object as needed. The exceptions did not go away. I then started posting on various forums and newsgroups to no avail. I've gotten nothing helpful.

After posting here, I decided to not use pooling and disabled it. The exceptions are no longer thrown from my code. However, I occasionally (rarely) get the exceptions from the session state (I have mode = SqlServer and the connection string is like above).

What else can I do to help you help me?

 

Answer 14

Can you open a case with CSS?  They'll have the tools and personel to dig into this fairly quickly and help you capture the information we'll need solve the problem.  This appears almost certainly to be a serious bug in System.Data code at this point, but I'm unable to pin it down with the information I've got so far.

Answer 15

It appears to be a very fine race in the ADO.Net code.  A suggestion given by MS is to change the connection string to include: "Min Pool Size=1".  This should prevent the race from happening.

Answer 16

It appears that the System.Threading.SemaphoreFullException is happening as a result of a very fine race condition.

A suggestion from MS is to change the database connection string to include: "Min Pool Size=1".  I am going to try it on my reproducable case that I have in house.

Answer 17

We are experiencing the exact same thing here with our application.

Basically, an .aspx  form submits and processes data with about 9 connections opened and closed between 3 databases.

We get all 3 of the errors you mentioned. Even with Pooling=False in the Connection String we were still able to generate an
error.  Also, we get errors with a very light load of users 2-10 people max. 

I have searched all over to no answer. Did you ever solve this issue? We have a ticket open with Microsoft to fix this
but so far they seem to be stumped.  It's been 3 days now.  This is a very important app that needs to be live again.

Please let me know if you have the answer or workaround...I'm going to try the last suggestion mentioned
of setting Min Pool Size = 1.

Thanks, Tad

Answer 18

Sorry I haven't posted in a while, but I had to move on - the app couldn't wait around.

I ended up leaving pooling disabled and creating 1 connection per database (lazily) and leaving it open for the duration of the request. Kind of how it's commonly done in classic ASP.

After turning off pooling, I would still get the error a couple more times and then no more - I haven't had any since.

 

Answer 19

I also get the SemaphoreFullException exception at irregular times.  My code is very simple, essentially something like:

Private Shared _pool As Semaphore
_pool = New Semaphore(2, 2)

<WebMethod()> _
Public Sub HelloWorld()
            _pool.WaitOne()
            <Call VB6 component>
            _pool.Release()
End Function

Every now and then, _pool.Release() throws SemaphoreFullException .

I can not see a programming error in my code, and suspect an error in the Semaphore class..

I do not use ADO or any database conenction.  I do have some references to single threaded VB6 COM components, but I do not see how that could influence.

Best regards, Bjorn Sigurd

Answer 20

...found the solution myself:  I had the declaration

Private Shared _pool As Semaphore

and was performing the call

_pool = New Semaphore(2, 2)

in the Constructor / Public Sub New() for the Web Service.  When changing to

If mobjSemaphore Is Nothing Then
         _pool = New Semaphore(2, 2)
End If

everything seems to be ok.

Sorry about that....

Answer 21

We too are facing the same problem. We get both the COM Exception and Semaphore Exception atleast once a day. Ours is a windows application. We are using SQL Express 2005 as database. We are using Data Access Application Blocks and we are using DatabaseFactory.CreateDatabase to get reference to a Database object and are calling either ExecuteNonQuery or ExecuteDataSet methods on it to call stored procedures. The problem occurs in a method of the code that saves each user action to local database. Its called "Journal Entry". So every button click is saved to the local database by calling a stored procedure available in local database(SQL Express). I believe DAAB automatically handles connection pooling. But as suggested it could be a bug in connection pooling or DAAB code. Could any one help us. We have deployed the application on site and the application crashes multiple times a day on site.

Thanks,
Siva

Answer 22

One thing to watch out for when using some versions of DAAB in a multi-threaded applications -- you can end up using the same connection from multiple threads, especially if you set up some of the objects as static fields. Multi-thread use of ADO.Net connections will definitely causes random problems, including failures to properly release semaphores.

Answer 23

So are you saying that there are some versions of DAAB that do not have this problem in a multi-threaded application? I'm experiencing the SemaphoreFullException problem while using the SqlHelper class from DAAB version 2.0 (most of the functions in that class are defined as Static).

 

Is there a more recent version of this class that doesn't have this problem, or is there a replacement that does not invovle restructuring my entire application?

Answer 24

Hello, I have excatly the same problem. Did you succeed to solve the problem ? Thanks for your response. Ines

Answer 25

I filed a bug internally to take a look at this, once I get this investigated I will post back, tx!

Answer 26

We started getting the same error today. We have not made any code changes in a while. We did upgrade our SQL Server SP1 to CU1 last week.. that is the only change we made.

Answer 27

Just a follow up folks.  I've been investigating this one for a while, creating stress tests and modifying internal timers to increase the likelyhood that the code paths in question would trigger something but thus far have not been able to replicate.  Testing on a quad amd64 machine and giving it lots of stress.

From code examination alone I don't see any issues yet the code looks very solid.

As a response to :It would seem that the PutNewObject is a more likely candidate for a problem since it seems that it just does a semaphore Release with no apparent matching WaitOne.

Actually the PutObject is releasing a semaphore ref gains in a call to WaitForMultipleObjectsEx in DbConnection.GetConnection, this call passes in array of handles one of which is the PoolSemaphore in question, so this code is ok, I verified this.

I have some theories on how to replicate this one however, will post back once I have a repro.  If anyone has a nice repro let me know.

Answer 28

Also if anyone is hitting this consistently on their web site etc let me know and I can help you capture dumps for offline analysis (I can analyze, this will help).

Answer 29

Hi,

Currently I am facing this problem.
Did addition of "Min Pool Size=1" to the connection string solved the problem ?

Answer 30

Hello,

I am experiencing this same behavior. The most common error I see is [COMException (0x80070006): The handle is invalid. (Exception from HRESULT: 0x80070006 (E_HANDLE))]. However, I have also ran accross this error [SemaphoreFullException: Adding the given count to the semaphore would cause it to exceed its maximum count.] , but less frequently. 

I have the Min Pool Size = 1 in my connection string and I still get his error. Hs anyone found a way to solve this issue? The only way I've found to restore the client once they get the 0x80070006 exception is to recycle the application pool in IIS, which seems to be a rather drastic and broad reaching approach. Any help would be much appreciated as this is running in a live environment and I cannot replicate the behavior in my test environment.

Answer 31

Just as a reply to above post, I examined this code very closely and verified we don't have a problem here. Note the semaphore is add-ref'd earlier by a call to WaitForMultipleObjects, so there is not a WaitOne/Release mismatch here.

Answer 32

Hi folks, just an update.

I finally got a few customers to send me dumps with this particular exception and I uncovered one possible root cause.

If you are familiar with using Windows debugger (WinDbg) and the sos debugger extension, you can check this yourself against your own web site.   Easiest way to catch the dump is use DebugDiag tool (download here ->
http://www.microsoft.com/downloads/details.aspx?FamilyID=28bd5941-c458-46f1-b24d-f60151d875a3&displaylang=en)

With DebugDiag, you can configure it to monitor your web site and capture full dumps when a first chance System.Threading.SemaphoreFullException is raised.

Once you have the dump, here is how you proceed:

Open dump in Windbg (download Debugging Tools for Windows http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx).

Load sos debugger extension:

0:011> .load D:\Windows\Microsoft.NET\Framework\v2.0.50727\sos.dll

Run !dso command, then find first DbConnectionPool on stack:

0:099> !dso
OS Thread Id: 0x1111 (99)
ESP/REG  Object   Name
04000000 30000000 System.Threading.SemaphoreFullException
…
04000200 04000000 System.Data.ProviderBase.DbConnectionPool

Dump this (!do )

0:099> !do 04000000
Name: System.Data.ProviderBase.DbConnectionPool
MethodTable: 6523f088
EEClass: 6515c8e0
Size: 100(0x64) bytes
 (C:\WINDOWS\assembly\GAC_32\System.Data\2.0.0.0__b77a5c561934e089\System.Data.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name

…
...     2c ...l+PoolWaitHandles  0 instance 05000000 _waitHandles
…

Dump _waitHandles:

0:011> !do 05000000
Name: System.Data.ProviderBase.DbConnectionPool+PoolWaitHandles
MethodTable: 652410b4
EEClass: 65170bd8
Size: 52(0x34) bytes
 (C:\WINDOWS\assembly\GAC_32\System.Data\2.0.0.0__b77a5c561934e089\System.Data.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
…
...       14 ...reading.Semaphore  0 instance 06000000 _poolSemaphore
…

Now examine _poolSemaphore:

0:011> !do 06000000
Name: System.Threading.Semaphore
MethodTable: 7a5e3018
EEClass: 7a479418
Size: 24(0x18) bytes
 (C:\WINDOWS\assembly\GAC_MSIL\System\2.0.0.0__b77a5c561934e089\System.dll)
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
…
...        c        System.IntPtr  1 instance     123c waitHandle
…

Notice this has a waitHandle, this is the actual semaphore handle, examine this:

0:011> !handle 123c f
Handle 0000123c
  Type          Event
…

Note this handle should be a semaphore handle in normal operating circumstances but what I found is the handle is corrupted, it's now an Event handle! (or some other random sort of handle for example a file handle).

This can happen if something inside the same process inadvertently closes the wrong handle.  Windows will recycle the handle and the next caller to open a handle will get the same handle value.    When we use this handle with semaphore functions it reports E_HANDLE or potentially SemaphoreFullException when ReleaseSemaphore is called on the handle (this is what .NET Semaphore class raises when ReleaseSemaphore fails for ANY reason).

Hence the root cause of this problem I believe is something inside the process inadvertently calling CloseHandle on our internally held handles.  This invalidates _poolSemaphore and all calls into pool fail (until pool is cleared).

To debug further you can use Application Verifier and track handles to capture the caller who is calling CloseHandle.

To do this you download Application Verifier and enable handle tracking for w3wp.exe.  Then attach Windbg to running w3wp.exe process and run !htrack -enable to enable handle tracking.  Then capture another dump when SemaphoreFullException occurs and run !htrack to see who closed the handle (the call stacks are stored in the dump and tracked when CloseHandle is called).

I was thinking we could harden our pool handles by calling DuplicateHandle to increase handle ref on our side, but in general this would not solve the general problem that customers need to resolve anyway.  The general problem is code inside the process inadvertently closing handles it does not own, this is a very serious problem that will cause other things in the process to abnormally fail so you need to get to the root cause of this problem in any case, even if I harden my pool handles.

Answer 33

This was a beast to debug. Here's a few things I learned:

1. Handles are reused. Therefore, if you have an IntPtr to a handle and it gets closed, your IntPtr is still (or will be soon) pointing to a valid handle.

2. If you call LogonUser (from the advapi32.dll) twice for the same user, your handle returned from the first call is automatically closed (if it wasn't already). If you call CloseHandle on the IntPtr returned from the first call again, you will screw things up as your IntPtr may now be pointing to a registry handle or a file handle or an SQL connection handle or it may even be the same value as the one returned from the second LogonUser call, etc.

3. If you close an SQL connection handle, you will get one of the above post's three errors. The actual error depends upon whatever state the pool happens to be in. To debug one of the above three errors, search for all uses of CloseHandle in your code -- one of those is likely your culprit -- one of those is likely closing a handle that is no longer pointing to the handle/token you thought it was.

4. The WindowsIdentity token/handle constructor duplicates the handle internally. Therefore, you can and should CloseHandle your token from LogonUser immediately after you construct the WindowsIdentity object. There are some rumors that you need to duplicate your handle before passing it into that constructor, but that issue was fixed in .Net 2.0.

5. Do not dispose of the WindowsIdentity object yourself. The Timer class grabs the current one and holds onto it for later use. A number of .Net classes use the Timer class internally, including the SqlClient connection pool manager. Rather, you should cache them so that you only have one for each user, not one per webservice call, etc. You don't want to burden the authentication server anyway -- you should be caching WindowsIdentity objects to avoid network traffic. For more info on this problem see http://winterdom.com/2008/12/crashingyourprocesswithtimers

Answer 34

Adding "Min Pool Size=1" should exacerbate the issue as there is now guaranteed to be at least one active SQL connection handle that can be closed incorrectly through the CloseHandle call. A single threaded environment with connection pooling disabled should not see any of the above errors as it's unlikely the code does anything like this:

using(new SqlConnection) { CloseHandle(a handle/token I thought was open but really wasn't); }