locked
TFS Warehouse / Coverage Analysis is failing RRS feed

  • Question

  • Hello everyone!

    I am running with TFS 2015.3, and SQL Server 2016 SP1, in separate servers.

    Recently, I change our production TFS Database from 32 bit to 64 bit (for updating purposes), and I believe we loose some database configuration and permissions.

    This week (after the change), we are having some issues with some Jobs, related to Warehouse and Coverage Analysis:

    The error that is shown is:

    Microsoft.TeamFoundation.Framework.Server.DatabaseOperationTimeoutException: TF246018: The database operation exceeded the timeout limit and has been cancelled. Verify that the parameters of the operation are correct. ---> System.Data.SqlClient.SqlException: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception: The wait operation timed out --- End of inner exception stack trace --- at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry) at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry) at System.Data.SqlClient.SqlCommand.ExecuteNonQuery() at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.ExecuteCommand(ExecuteType executeType, CommandBehavior behavior) --- End of inner exception stack trace --- at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.TranslateException(Int32 errorNumber, SqlException sqlException, SqlError sqlError) at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.TranslateException(SqlException sqlException) at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.MapException(SqlException ex, QueryExecutionState queryState) at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.HandleException(Exception exception) at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.ExecuteCommand(ExecuteType executeType, CommandBehavior behavior) at Microsoft.VisualStudio.Services.CircuitBreaker.Command.<>c__DisplayClass7_0.<Execute>b__0() at Microsoft.VisualStudio.Services.CircuitBreaker.CommandAsync.<Execute>d__28.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.VisualStudio.Services.CircuitBreaker.Command.Execute(Action run, Action fallback) at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.ExecuteCommandWithBaseCircuitBreaker(ExecuteType executeType, CommandBehavior behavior) at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.ExecuteNonQuery(Boolean bindReturnValue) at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.ExecuteNonQuery() at Microsoft.TeamFoundation.TestManagement.Server.BuildCoverage.Update(TestManagementRequestContext context, Int32 coverageChangeId) at Microsoft.TeamFoundation.TestManagement.Server.Jobs.CoverageAnalyzer.AnalyzeBuild(BuildInfo buildInfo, String& resultMessage) at Microsoft.TeamFoundation.TestManagement.Server.Jobs.CoverageAnalyzer.Run(String& resultMessage) at Microsoft.TeamFoundation.TestManagement.Server.Jobs.CoverAnJob.Run(IVssRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime queueTime, String& resultMessage)

    And also I got this one for another job:

    [Test Management Warehouse Sync]: ---> MakeDataChanges() result=DataChangesPending. ---> MakeDataChanges() result=DataChangesPending. ---> MakeDataChanges() result=DataChangesPending. ---> Microsoft.TeamFoundation.Warehouse.WarehouseException: TF221122: An error occurred running job Test Management Warehouse Sync for team project collection or Team Foundation server Product. ---> Microsoft.TeamFoundation.Framework.Server.DatabaseOperationCanceledException: TF53001: The database operation was canceled by an administrator. ---> System.Data.SqlClient.SqlException: A severe error occurred on the current command. The results, if any, should be discarded. Operation cancelled by user. at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry) at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry) at System.Data.SqlClient.SqlCommand.ExecuteNonQuery() at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.ExecuteCommand(ExecuteType executeType, CommandBehavior behavior) --- End of inner exception stack trace --- at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.TranslateException(SqlException sqlException) at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.MapException(SqlException ex, QueryExecutionState queryState) at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.HandleException(Exception exception) at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.ExecuteCommand(ExecuteType executeType, CommandBehavior behavior) at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.ExecuteNonQuery(Boolean bindReturnValue) at Microsoft.TeamFoundation.Warehouse.WarehouseBatchedDataAccessComponent.ExecuteNonQuery() at Microsoft.TeamFoundation.TestManagement.Warehouse.TeamTestWarehouseAdapter.SaveBatchesToWarehouse() at Microsoft.TeamFoundation.TestManagement.Warehouse.TeamTestWarehouseAdapter.ProcessResults(WarehouseRowVersionQueryDelegate query, String highWaterMarkProperty, String processLimitProperty) at Microsoft.TeamFoundation.TestManagement.Warehouse.TeamTestWarehouseAdapter.MakeDataChanges() at Microsoft.TeamFoundation.Warehouse.WarehouseSyncJobExtension`1.MakeDataChanges(IVssRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, String& resultMessage) at Microsoft.TeamFoundation.Warehouse.WarehouseSyncJobExtension`1.RunInternal(IVssRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime queueTime, String& resultMessage) at Microsoft.TeamFoundation.Warehouse.WarehouseJobExtension.Run(IVssRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime queueTime, String& resultMessage) --- End of inner exception stack trace ---


    I am guessing this is due to a Database error, but I can't get to the bottom of it. I am using the TFS for automated Builds, and I am not able to see the Code Coverage for the most recent builds. I don't know if it is related. I went to the Application Tier, and Start a Rebuild for the Reporting services, and that works for some of the Build (the Code Coverage appears), but for the newer builds, it is not happening as expected (I run the Rebuild and still the Code Coverage is not shown).

    Do you have any ideas on what is going on with those jobs? How can I solve this?

    Thank you so much in advance. If you need any further information, please let me know.

    Saturday, September 16, 2017 12:41 AM

All replies

  • I forgot to say, and it may be important: At the start of the week, we had some DNS problems with the Domain Controller (already solved by our Helpdesk team), so I don't know if this error can be related to it, and how could I know that? 

    Thanks!


    • Edited by LDCONEJO Saturday, September 16, 2017 5:04 AM
    Saturday, September 16, 2017 5:03 AM
  • Hi LDCONEJO,

    Do you enable code coverage in your build definition? Could you please create a new build definition, could you see code coverage data?

    Best Regards


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Monday, September 18, 2017 6:46 AM
  • Hello Jingzhu.

    Yes, everything is enable.

    For example, after I have that issue, I restart both servers, last Saturday. When they turned on, everything started working fine (the Code Coverage Job run correctly). But today, without any change or different configuration in the servers, the error start appearing again, and I can't see the Code Coverage again.

    Why it is acting that weird? Have any idea?

    Thursday, September 21, 2017 4:50 PM
  • Look at this, for example:

    It was working smoothly, but suddenly, the job starts failing. And we didn't make any change. I am not sure on what to do to make it run consistently.

    Always for the same error:

    Microsoft.TeamFoundation.Framework.Server.DatabaseOperationTimeoutException: TF246018: The database operation exceeded the timeout limit and has been cancelled. Verify that the parameters of the operation are correct. ---> System.Data.SqlClient.SqlException: Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception: The wait operation timed out
       --- End of inner exception stack trace ---
       at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
       at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
       at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
       at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption)
       at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
       at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
       at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
       at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
       at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.ExecuteCommand(ExecuteType executeType, CommandBehavior behavior)
       --- End of inner exception stack trace ---
       at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.TranslateException(Int32 errorNumber, SqlException sqlException, SqlError sqlError)
       at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.TranslateException(SqlException sqlException)
       at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.MapException(SqlException ex, QueryExecutionState queryState)
       at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.HandleException(Exception exception)
       at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.ExecuteCommand(ExecuteType executeType, CommandBehavior behavior)
       at Microsoft.VisualStudio.Services.CircuitBreaker.Command.<>c__DisplayClass7_0.<Execute>b__0()
       at Microsoft.VisualStudio.Services.CircuitBreaker.CommandAsync.<Execute>d__28.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at Microsoft.VisualStudio.Services.CircuitBreaker.Command.Execute(Action run, Action fallback)
       at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.ExecuteCommandWithBaseCircuitBreaker(ExecuteType executeType, CommandBehavior behavior)
       at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.ExecuteNonQuery(Boolean bindReturnValue)
       at Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.ExecuteNonQuery()
       at Microsoft.TeamFoundation.TestManagement.Server.BuildCoverage.Update(TestManagementRequestContext context, Int32 coverageChangeId)
       at Microsoft.TeamFoundation.TestManagement.Server.Jobs.CoverageAnalyzer.AnalyzeBuild(BuildInfo buildInfo, String& resultMessage)
       at Microsoft.TeamFoundation.TestManagement.Server.Jobs.CoverageAnalyzer.Run(String& resultMessage)
       at Microsoft.TeamFoundation.TestManagement.Server.Jobs.CoverAnJob.Run(IVssRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime queueTime, String& resultMessage)

    Any clue?

    Friday, September 22, 2017 1:37 PM
  • Hi LDCONEJO,

    Please check this thread, could the solution in answer reply work for yours?

    https://social.msdn.microsoft.com/Forums/vstudio/en-US/ece6c454-75b7-4a0c-b68a-89ad27093a68/tf246018-the-database-operation-exceeded-the-timeout-limit-and-has-been-cancelled-verify-that-the?forum=tfsgeneral

    If this could not work, please refer to steps in the middle of the link below:

    https://dankovalkov.wordpress.com/2017/02/03/tfs-2017-resolving-tf221122-tf246018-tf400307-and-tf400797-issues/

    Best Regards


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Sunday, September 24, 2017 3:21 AM