Incremental Analysis Database Sync Occasional Failure TF221122 : TF255040

Question

I'm moving a Tfs Project collection from one server to another.  In getting Reporting Services to run I'm seeing a problem as witnessed using the AdminReportPack Cube Status report.

Since midnight yesterday I've seen two full syncs succeed, 11 incrementals succeed and 5 incrementals fail.  I've installed and patched everything I can think of on the AT and DT servers.  I see the assembly referenced on the server and I can't figure out why it sometimes fails to load the assembly via reflection.

Has anyone seen this before and/or can think of troubleshooting steps?

[Incremental Analysis Database Sync]:
Microsoft.TeamFoundation.Server.WarehouseException: TF221122: An error occurred running job Incremental Analysis Database Sync for team project collection or Team Foundation server TEAM FOUNDATION.
Microsoft.TeamFoundation.Server.WarehouseException: TF255040: Install SQL Reporting Services or at a minimum SQL Client Connectivity Tools on the application tier to ensure Analysis Services object model is present for warehouse processing.
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.AnalysisServices, Version=10.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91' or one of its dependencies. The system cannot find the file specified.
at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection)
at System.Reflection.Assembly.Load(String assemblyString)
at Microsoft.TeamFoundation.Warehouse.AnalysisDatabaseSyncJobExtension.CheckAnalysisServicesInstalled()
--- End of inner exception stack trace ---
at Microsoft.TeamFoundation.Warehouse.AnalysisDatabaseSyncJobExtension.CheckAnalysisServicesInstalled()
at Microsoft.TeamFoundation.Warehouse.AnalysisDatabaseSyncJobExtension.RunInternal(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime queueTime, String& resultMessage)
at Microsoft.TeamFoundation.Warehouse.WarehouseJobExtension.Run(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime queueTime, String& resultMessage)
--- End of inner exception stack trace ---

Answer 1

For what it's worth, I've googled "Microsoft.TeamFoundation.Warehouse.AnalysisDatabaseSyncJobExtension.CheckAnalysisServicesInstalled"  and found two other instances of people having this problem.  One person said it "went away"  without any explanation and another person said it "went away" and attributed it to the "infrastructure guy added more RAM".

Something is really weird here in that the error only occurs occasionally.   Reporting Services seems to otherwise be working.  I see current data eventually making it into reports that use the OLAP cube.

Answer 2

I used .NET reflector to examine the implementation of AnalysisDatabaseSyncJobExtension.CheckAnalysisServicesInstalled() and the code looks like this:

...

string assemblyString = "Microsoft.AnalysisServices, Version=10.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91";

...

Assembly.Load(assemblyString);

...

So based on that I wrote a small console app that loads that assembly and it succeeded over and over as I have that assembly in the GAC.

Answer 3

Hi Chris, 

Thanks for your post.

If you installed DT on another machine, have you installed the SQL Client Connectivity Tools on your AT? If not, please try to install it. Please refer to the answer reply in this post: http://social.msdn.microsoft.com/Forums/pl-PL/tfssetup/thread/e7c81903-6645-40bd-b970-4957cd20de04. 

If this issue still occurred, you can try to recreate the Warehouse database and Analysis database for your TFS 2010, then rebuild. Please refer to this article: http://mohamedradwan.wordpress.com/2010/10/29/how-to-recreate-the-data-warehosue-and-the-analysis-service-for-tfs/.         

---

John Qiao [MSFT]
MSDN Community Support | Feedback to us

Answer 4

Hi Chris, 

If misunderstood anything, please describe your question in more detail and we will be able to provide the better responses.

---

John Qiao [MSFT]
MSDN Community Support | Feedback to us

Answer 5

Hi John,

 Sorry, I took a mental break from this problem.  Yes, according to the MS SQL Server 2008 Setup Discovery Report I have the Client Tools connectivity 1033 Enterprise Edition 10.51.2500.0 components installed on the AT.

My current situation is I can't get the cube to build at all.  I get this error all the time now.

Rebuilding the analysis cube is exactly what I'm trying to do.  I can get the warehouse to process via web service invocation but the cube fails with the error given.  I'm using the AdminReportPack Cube Status report to monitor this.

I'm completly stuck here.  I'm on the AT using Enterprise Manager and I can connect to the SSAS on the DT layer and I can see this assembly on my hard drive but for the life of me this cube just won't build.   I don't know what else to try.

Thanks,

Chris

Answer 6

I've also noticed that if I tear it all down and reconfigure it in the console the first time I try to build the cube  the job will run for about 75 - 90 seconds and then give the following error.  If I run the job again it goes back to giving an error message about the client tools.

[Full Analysis Database Sync]:
AnalysisDatabaseProcessingType=Full, needCubeSchemaUpdate=True.
Microsoft.TeamFoundation.Server.WarehouseException: TF221122: An error occurred running job Full Analysis Database Sync for team project collection or Team Foundation server TEAM FOUNDATION.
Microsoft.TeamFoundation.Server.WarehouseException: Failed to Process Analysis Database 'Tfs_Analysis'.
Microsoft.AnalysisServices.ConnectionException: The connection either timed out or was lost.
System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.IO.BufferedStream.Read(Byte[] array, Int32 offset, Int32 count)
at Microsoft.AnalysisServices.DimeRecord.ForceRead(Stream stream, Byte[] buffer, Int32 length)
at Microsoft.AnalysisServices.DimeRecord.ReadHeader()
at Microsoft.AnalysisServices.DimeReader.ReadRecord()
at Microsoft.AnalysisServices.TcpStream.GetResponseDataType()
--- End of inner exception stack trace ---
at Microsoft.AnalysisServices.XmlaClient.EndRequest()
at Microsoft.AnalysisServices.XmlaClient.SendExecuteAndReadResponse(Boolean skipResults, Boolean throwIfError)
at Microsoft.AnalysisServices.AnalysisServicesClient.CommitTransaction()
at Microsoft.AnalysisServices.Server.CommitTransaction()
at Microsoft.TeamFoundation.Warehouse.TFSOlapProcessComponent.ProcessOlap(AnalysisDatabaseProcessingType processingType, WarehouseChanges warehouseChanges, Boolean lastProcessingFailed, Boolean cubeSchemaUpdateNeeded)
--- End of inner exception stack trace ---
at Microsoft.TeamFoundation.Warehouse.TFSOlapProcessComponent.ProcessOlap(AnalysisDatabaseProcessingType processingType, WarehouseChanges warehouseChanges, Boolean lastProcessingFailed, Boolean cubeSchemaUpdateNeeded)
at Microsoft.TeamFoundation.Warehouse.AnalysisDatabaseSyncJobExtension.RunInternal(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime queueTime, String& resultMessage)
at Microsoft.TeamFoundation.Warehouse.WarehouseJobExtension.Run(TeamFoundationRequestContext requestContext, TeamFoundationJobDefinition jobDefinition, DateTime queueTime, String& resultMessage)
--- End of inner exception stack trace ---

Answer 7

Hi Chris, 

Thanks for your reply.

Before you move the Collection to anther server, the Reporting Services works fine on this another server? 

You said that “reconfigure it in console”, do you mean that recreated the warehouse and analysis databases suing TFS Admin Console, but still can’t rebuild correctly?

I suggest you can try to reconfigure the Reporting Service for your another TFS Server, How to manually configure a report server for TFS 2010, please refer to: http://msdn.microsoft.com/en-us/library/dd578652.aspx and this video: http://www.youtube.com/watch?v=LRrVAsD7fPY. After configured Report Server for TFS 2010, then launch TFS 2010 Admin Console>>Application Tier>>Reporting to configure, please refer to: http://msdn.microsoft.com/en-us/library/ms404869.aspx#RestoreWarehouse.

 

This issue requires more troubleshooting and I can’t reproduce this scenario on my machine. If it is urgent issue, I suggest you to contact a Professional Support Service at http://support.microsoft.com/common/international.aspx?RDPATH=gp;en-us;offerprophoneto gain more support on this case.   

---

John Qiao [MSFT]
MSDN Community Support | Feedback to us

Answer 8

Yes, the reporting works for this collection fine on the previous server. In fact I just dropped the warehouse and cube and was able to rebuild it from scratch and green up in the cube status report with fresh data for the warehouse and cube.  It's just when I attempt this procedure on the new server I get the various error messages.   Very wierd.

I'll review the information in your links and see if it helps.  Otherwise we will be opening a ticket with MSFT to get us through this transition.

Thanks,
Chris