locked
Warehouse Troubleshooting Guide RRS feed

  • General discussion

  • This will be a living document and as we troubleshoot issues, we will add to the query and document. It will culminate in a consistency checker tool which automates the checking and ultimately other functions like kicking the warehouse etc... Feedback is welcome and I will update this data.

    Symptoms:

    The data in the reports is stale and doesn’t seem to ever get updated.
    The reports error out.

    An error has occurred during report processing. (rsProcessingAborted)

    Background:

    Data in the operational stores (Work Item Tracking, Version Control, Team Build, Integration Services) is pulled into a relational database (TfsWarehouse) and then into an OLAP cube on a scheduled basis. The reports are rendered by querying the cube. There is a Windows Service names TfsServerScheduler which calls the warehouse web service every hour (by default) to initiate the processing of the warehouse. The cube and reports should be available while this process continues in the background.

    Troubleshooting:
    Preparation Steps:

    You should have admin access to the application tier and data tier. The database and cubes are on the data tier. In Beta 3, the reporting services site and reports are on the data tier but in the final RTM version of Team Foundation, it is on the application tier.

    On the application tier, turn off IE friendly error handling so you can see messages. In IE, Tools, Internet Options, Advanced, Uncheck HTTP Friendly Errors

    Step 1: Is the TFSServerScheduler windows service running?

    Right click on My Computer, Manage. Select Services and Applications and then Services. Scroll down to the TFS Server Scheduler and ensure it is running.

    This windows service is what signals the warehouse web service to go aggregate data and process the cube. In the final RTM version of Team Foundation, there is one other case where the warehouse is signaled - the completion of the project creation wizard. That means that within a few minutes, the warehouse should be processed and the reports should be available.

    To find out when was the last time the scheduler called the warehouse to process, browse to %ProgramFiles%\Microsoft Team Foundation Server 2005\TfsServerScheduler. The xml file in that directory will contain the last time warehouse processing was attempted.

    Step 2: Did processing pull data over into the warehouse?

    Data is pulled from all of the operational systems into a database named TfsWarehouse. From there, it is processed into the cube.

    To find out what amount of data in the operational systems made it into the TfsWarehouse database, open up SQL Manager and execute:

    use TfsWarehouse
    select Setting as [Last Cube Processing] from _WarehouseConfig where ID = 'LastProcessedTime'

    -- total revisions in the relational warehouse
    use TfsWarehouse
    select top 1 __LastUpdatedTime as [LastVCWHWrite] from dbo.[Code Churn] with (nolock) order by __LastUpdatedTime desc
    select
    top 1 __LastUpdatedTime as [LastWITWHWrite] from dbo.[Work Item History] with (nolock) order by __LastUpdatedTime desc

    use TfsWarehouse
    select count(*) as [WH Revisions] from [dbo].[Work Item History] with (nolock)
    where [Record Count] <> -1

    use TfsWorkitemTracking
    select count(*) as [WIT Revisions] from [dbo].[WorkitemsLatestAndWere] with (nolock)

    use TfsWarehouse
    select max ([Changeset ID]) as [WH Changeset] from [dbo].Changeset with (nolock)
    where DimensionMemberActive = 1

    use TfsVersionControl
    select max(ChangeSetId) as [VC Changeset] from tbl_Changeset with (nolock)

    -- identities
    use TfsWarehouse
    select Property_Value as [Warehouse Identity Id] from _PropertyBag
    where Property_Key = 'CSS Identity Sequence Id'

    use TfsIntegration
    select max(sequence_Id) as [Integration Identity Id] from tbl_security_identity_cache

    -- structure
    use TfsWarehouse
    select Property_Value as [Warehouse Structure Id] from _PropertyBag
    where Property_Key = 'CSS Structure Sequence Id'

    use TfsIntegration
    select max(sequence_Id) as [Integration Structure Id] from tbl_nodes with (nolock)

    This should let you how up to date your warehouse is. If no data is present, or data is severely latent, either the scheduler is not signaling processing or there is a problem in the warehouse aggregating data. Please refer to Gathering Error Data below.

    So, let's go over the output. Here's sample output from our dogfood server with comments after each output block:

    -- This shows the last time the cube was successfully processed. If
    -- the scheduler is initiating processing successfully,
    -- it should be less than an hour

    Last Cube Processing
    -----------------------
    11/18/2005 11:45:33 AM

    -- Last code churn entry written in the warehouse
    LastVCWHWrite
    -----------------------
    2005-11-22 11:23:15.803
    -- Last workitem revision written into the warehouse
    LastWITWHWrite
    -----------------------
    2005-11-19 22:13:44.047
    -- The number of workitem revisions in the workitem tracking
    -- operational store
    WH Revisions
    ------------
    463754
    -- The number of workitem revisions that made it into the warehouse
    WIT Revisions
    -------------
    463754
    -- The last Changeset that was processed into the warehouse
    WH Changeset
    ------------
    12702
    -- The last changeset in the Source Control operational store
    VC Changeset
    ------------
    12702
    -- The user/group change number that made it into the warehouse
    -- (each change which can be one or more user/group changes gets a
    --sequence id)
    Warehouse Identity Id
    ----------------------
    5667
    -- The user/group change number in the operational store
    Integration Identity Id
    -----------------------
    5667
    -- The area/iteration change number in the warehouse
    Warehouse Structure Id
    -----------------------
    990
    -- The area/iteration change number in the operational store
    Integration Structure Id
    ------------------------
    990

    Step 3: Manually processing the warehouse

    If there are problems, it is possible to manually process the cube. After manually processing the cube, you can repeat step 2 to validate data was moved over and then if there's problems, refer to the Gathering Error Data section below for the time range of the processing.

    Log onto the application tier. Browse to http://localhost:8080/Warehouse/v1.0/warehousecontroller.asmx

    Click on GetWarehouseStatus, and then click the Invoke button. Wait for it to be idle.

    It should return ProcessingAdapters (pulling data), ProcessingOlap (processing the cube), or Idle. If you have the final RTM version of Team Foundation, you may see it repeat since it does ProcessingAdapters (schema modifications), ProcessingOlap (schema modifications), ProcessingAdapters (pulling data), ProcessingOlap (processing the cube), and then idle.

    Browse to http://localhost:8080/Warehouse/v1.0/warehousecontroller.asmx

    Click on Run, and then click the Invoke button. This will start the processing of the warehouse. In Beta3, you will see a blank screen - in RTM, it will return true or false as to whether is started processing (asynchronously).

    You can repeat GetWarehouseStatus to monitor the processing.

    To verify, you can repeat step 2.

    Step 4: Browse to the reports
    Each report has the time the warehouse last processed. It should be a recent time.

    Report Generated: 2005/11/23 12:55:37 by domain\user; Last Warehouse Update: 2005/11/23 12:15:51

    The data should be current and less than an hour.

    For troubleshooting, you will likely want to disable caching for the report:
    view the report in IE (http://<server>/Reports/ ** then click on < project name > then < report >)
    click Properties -> Execution,
    Select "Do not cache temporary copies of this report",
    Apply
    Click View

    **In Beta3, <server> is the data tier - in RTM it is the application tier.

    Gathering Error Data

    Go to the Application Event Log (right click my computer, manage) and see if there are any errors (red x) from the source 'Warehouse'. The error and error stack in that event log can help identify the problem.

    If all of the steps are leading to errors, gathering a trace which a support specialist or product team member can analyze will help diagnose the issue.

    - Use a tool to capture debug output
    http://www.sysinternals.com/ntw2k/freeware/debugview.shtml, is a popular tool
    dbmon is another popular windows tool

    - Enabling tracing
    %ProgramFiles%\Microsoft Team Foundation Server 2005\Web Services\web.config:
    Change: <add name="General" value="1" /> to <add name="General" value="4" />

    Do Step 3 above

    Known Beta 3 Issues:

    1. Getting this? Are you on a non-english locale?
    Detailed Message: Cube processing runtime error: \r\nMicrosoft.TeamFoundation.Warehouse.WarehouseException: OLE DB error: OLE DB or ODBC error: The conversion of a char data type to a datetime data type resulted in an out-of-range datetime value.; 22007.
    at Microsoft.TeamFoundation.Warehouse.OlapCreator.ProcessOlap(Boolean schemaUpdated, UpdateStatusStore updateStatus)
    at Microsoft.TeamFoundation.Warehouse.AdapterScheduler.RunCubeProcess()


    See:
    http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=139746&SiteID=1

    2. Are iterations or area nodes not showing up in reports? Did you happen to delete a project and then recreate it by the same name?

    The following query will help you identify if you have a deleted project in the warehouse:


    SELECT TP.*

    FROM [TFSWarehouse].[dbo].[Team Project] as TP

    JOIN

    (SELECT

    'vstfs:///Classification/TeamProject/' + CAST(project_id AS VARCHAR(36)) AS Uri

    FROM [TFSIntegration].[dbo].[tbl_projects_tombstone]) as dTP

    ON (TP.[Project Uri] = dTP.[Uri])

    Renaming the deleted project will fix your problem. You can run the following to patch your warehouse:

    UPDATE [TFSWarehouse].[dbo].[Team Project]

    SET [Team Project] = ('Deleted Project - ' + TP.[Team Project] + ' (' + CAST(dTP.id AS VARCHAR(36)) + ')')

    FROM [TFSWarehouse].[dbo].[Team Project] as TP

    JOIN

    (SELECT

    'vstfs:///Classification/TeamProject/' + CAST(project_id AS VARCHAR(36)) AS Uri,

    project_id as Id

    FROM [TFSIntegration].[dbo].[tbl_projects_tombstone]) as dTP

    ON (TP.[Project Uri] = dTP.[Uri])

    3. Scenario Details and Related Workitem Reports show no data: The data should be in the warehouse if you run the queries above but there was an issue in the Beta 3 code which caused the data from time to time not to get into the cube during incremental processing. The workaround is to process the cube fully:
    http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=139746&SiteID=1

    4.  OLAP Schema can become out of sync:  Because of the processing order and not persisting a schema dirty bit, the OLAP schema could become out of sync.

    Symptoms when viewing reports from the AT (The dimension 'xxx' was not found in the cube):

    similar to ...

    An error has occurred during report processing. (rsProcessingAborted)
    Query execution failed for data set 'dsWorkItemTypeParam'. (rsErrorExecutingCommand)
    Query (1, 452) The dimension '[System_WorkItemType]' was not found in the cube when the string, [System_WorkItemType].[System_WorkItemType].[System_WorkItemType], was parsed.

     

    Resolution: Rebuild the OLAP schema and cube

    1) Back up all data

    2) Delete the TFSWarehouse database in *Analysis Services* (I don't believe this is absolutely necessary but doesn't hurt if you back it up). 
    3) From  a command prompt on the TFS Application Tier machine, use the following command (substituting the correct values for your installation in place of the following values: ServerName, Domain\TFSService, Domain\TFSReports):

    %ProgramFiles%\Microsoft Visual Studio 2005 Team Foundation Server\Tools\setupwarehouse.exe -o -s ServerName -d TfsWarehouse -c Warehouseschema.xml -a Domain\TFSService -ra Domain\TFSReports

     

    Note: -o is the important param here which causes the OLAP cube to get rebuilt.


    Feedback welcome. I will post a tool that will automate all of this by RTM.

    Thanks
    Bryan MacFarlane

    Friday, December 2, 2005 3:37 AM
    Moderator

All replies

  • Thank You! You're the man.
    Friday, December 2, 2005 12:38 PM
  • Thanks a lot for this.

    I went through these steps for this issue (http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=145329&SiteID=1) and it looks like the data warehouse is doing everything it is supposed to do and without error.

    This leads me to suspect that our reports for the scenario details and related work items are corrupt. We selected the CMMI process on beta 3 refresh.

    What would be the correct set of steps to obtain clean versions of these reports and upload these to our report server so that we can take advantage of these helpful reports?

    Thanks,

    Barton

    Saturday, December 3, 2005 8:03 AM
  • I have answered the post above.  As you noted, the data is in the warehouse - the problem was the processing instruction we give when processing the cube so the data didn't get into the cube.  The dev has asserted that it is fixed in RTM and we have not seen it for awhile on the RTM build.

    The workaround is to fully process the cube:
    http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=139746&SiteID=1 

    I am updating the troubleshooting steps with this as a known issue.

    thx
    Sunday, December 4, 2005 6:59 AM
    Moderator
  • (beta3 refresh)
    I went through the entire troubleshooting guide, but some reports still have this error:

    • An error has occurred during report processing. (rsProcessingAborted)
      • Query execution failed for data set 'dsWorkItemTypeParam'. (rsErrorExecutingCommand)
        • Query (1, 452) The dimension '[System_WorkItemType]' was not found in the cube when the string, [System_WorkItemType].[System_WorkItemType].[System_WorkItemType], was parsed.

    I haven't found a clue yet. I also have this possibly unrelated (but who knows) error in windows system log:

    The application-specific permission settings do not grant Local Activation permission for the COM Server application with CLSID {BA126AD1-2166-11D1-B1D0-00805FC1270E} to the user NT AUTHORITY\NETWORK SERVICE SID (S-1-5-20). This security permission can be modified using the Component Services administrative tool.

    Checking with regedit, this belongs to "Network Connection Manager Class", but I can't find/fix it in Component Services.

    Anybody knows how to fix this?

    Regards,

    Eric






Monday, December 5, 2005 3:12 PM
  • Hello Bryan,

    Regarding item 2 of the beta 3 issue list: "Are iterations or area nodes not showing up in reports".

    Can you please provide a quick fix for this?

    As noted, I deleted a previous Team Project and then recreated it again using the same name.
    Monday, December 5, 2005 4:51 PM
  • We have a fix ready - one of the devs should be posting tonight and I'll update the guide.

    thx
    Thursday, December 8, 2005 5:50 AM
    Moderator
  • Hi Gary,

    When there's a project name collision, the reports will not be able to find the proper project by name.  One symptom is the iteration and area nodes for that project don't show up in the reports.

    The following query will help you identify if you have a deleted project in the warehouse:


    SELECT
    TP.*

    FROM [TFSWarehouse].[dbo].[Team Project] as TP

            JOIN

        (SELECT

            'vstfs:///Classification/TeamProject/' + CAST(project_id AS VARCHAR(36)) AS Uri

        FROM [TFSIntegration].[dbo].[tbl_projects_tombstone]) as dTP

            ON (TP.[Project Uri] = dTP.[Uri])

    Renaming the deleted project will fix your problem.  You can run the following to patch your warehouse:

    UPDATE [TFSWarehouse].[dbo].[Team Project]

    SET [Team Project] = ('Deleted Project - ' + TP.[Team Project] + ' (' + CAST(dTP.id AS VARCHAR(36)) + ')')

    FROM [TFSWarehouse].[dbo].[Team Project] as TP

            JOIN

        (SELECT

            'vstfs:///Classification/TeamProject/' + CAST(project_id AS VARCHAR(36)) AS Uri,

            project_id as Id

        FROM [TFSIntegration].[dbo].[tbl_projects_tombstone]) as dTP

            ON (TP.[Project Uri] = dTP.[Uri])


    -Jim

    Friday, December 9, 2005 8:21 AM
  • Hello Jimmy,

    This is great. The fix is working well.
    Thanks for the quick response.

    Regards,
    Gary Russo
    Friday, December 9, 2005 3:48 PM
  • Great.  Thanks for the confirmation.  The troubleshooting guide above has been updated.
    Friday, December 9, 2005 5:26 PM
    Moderator
  • I too have one of these reporting errors that aren't fixed after fixing the Locale issue as well as making sure the OLAP cube got processed. It occurs on the 'Project Velocity' report and simply says "An error has occurred during report processing. (rsProcessingAborted)"

    The Report Server log gives more detail. Any help would be appreciated:

    w3wp!library!1!14/12/2005-18:37:57:: i INFO: Call to RenderFirst( '/TAPSDEV/Project Velocity' )
    w3wp!processing!9!12/14/2005-18:37:58:: e ERROR: Throwing Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: Exception of type 'Microsoft.ReportingServices.ReportProcessing.ReportProcessingException' was thrown., ; Info: Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: Exception of type 'Microsoft.ReportingServices.ReportProcessing.ReportProcessingException' was thrown.

    w3wp!processing!9!12/14/2005-18:37:58:: e ERROR: An exception has occurred in data source 'dsWorkItemHistoryChart'. Details: Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: The Group expression for the grouping 'Date' contains an error: Argument 'DateValue' cannot be converted to type 'Date'. at Microsoft.ReportingServices.ReportProcessing.ReportRuntime.RegisterRuntimeErrorInExpression(VariantResult& result, Exception e, IErrorContext iErrorContext, Boolean isError)

    Wednesday, December 14, 2005 7:05 PM
  • Hi,

    Great post!  Thank you.

    I do the following:

    1.  Add a new bug to work item s using Team Explorer

    2.  Rerun your query (SQL)

    I find that the changes appear after about a minute.  Why is this?

    Is it perhaps becuase TE saves the data to another table/database and it is copied to TFSWarehouse at alater time? 

    I have looked at a query on [Current Work Item] - same result!

    How can I access the data immediately?

    Kind Regards

    Jean-Pierre

     

     

    Thursday, December 15, 2005 7:26 AM
  • I notice that the _Warehouse Config table contains some fields which can be set - notably the following:

    - FirstStartDelaySeconds

    - RunIntervalSeconds

    How do these settings work? 

    Do they operate on TFSWarehouse (OLTP) or on the cube?

    Kind Regards

    Jean-Pierre

     

     

     

    Thursday, December 15, 2005 8:12 AM
  • The RunInterval seconds is the latency of the warehouse.  In my write up above I refered to the TfsServerScheduler NT service - that's the interval it kicks the warehouse to process.

    Processing includes pulling data from many data sources into warehouse and processing the cube.

    Note, that the process of pulling data and processing the cube isn't always cheap which is why we ship with an hour latency.  The warehouse should be a source of aggregated latent data for historical purposes (trending over time, stats etc...) and the granularity is typically a day on trending but you want the latest date to be relatively recent.

    Friday, December 16, 2005 8:31 AM
    Moderator
  • Bryan,

    Thanks for your response.

    - So the _WarehouseConfig values are the same as those manipulated by the warehousecontroller.asmx? 

    - The TFSWarehouse (*relational*) database is always going to be slightly behind the actual/raw data used to track work items. 

    - I suppose there are some stored procedures -  or perhaps other logic - to draw the raw data into TFSWarehouse.  Is there a way to force updates of just the relational data warehouse (TFSWarehouse)?

     

    Jean-Pierre

     

    Friday, December 16, 2005 8:45 AM
  • I have tried:

    - sql analysis services - right-click on TFSWarehouse cube and click Process

    Is this functionally equivalent to invoking the Run method on warehousecontroller.asmx?

    Jean-Pierre

     

     

     

    Saturday, December 17, 2005 9:04 AM
  • See step 3 of the troubleshooting steps to Run manually.

    Warehousecontroller.asmx::Run() does two things:
    1.  Load each adapter (look in the warehouse\bin\plugins folder) - each system (wit, sc, build etc...) provides an adapter which is a plugin which reads data from it's operational system and uses a provide API to write data into the warehouse.   Note that each adapter can also use common APIs to dynamically extend the logical schema of the warehouse (relational and cube).  In RTM each adapter will makeschemachanges and then each adapter will pull data in.   They run in parallel in RTM - in beta3, they block if making schema changes.  Note that the warehouse is extensbile - any system can write an adapter.   At this point, the data is in the relational warehouse.
    2.  Process the cube - it then processes the cube so the data is reflected in the cube.  This is done transactional so the cube is available while it's being processed.

    Also - yes, the warehouse controller settings method writes the values that are in the config table.

    Hope that helps.

    Sunday, December 18, 2005 3:12 AM
    Moderator
  • Brilliant!

      Thanks.

    So, in sum (please check me if I'm wrong):

    Processing the cube is not sufficient to update the TFSWarehouse data store.  You should invoke Run on warehousecontroller.asmx

     

    Sunday, December 18, 2005 2:47 PM
  • Dear Bryan,

     

    Herewith, the output.

    You will find Troubleshooting interesting (see below for app event log error).

    Also note the discrepancy between the scheduler .xml and the report headers.

    STEP 1  (Discrepancy between this and step 2)
    ======

      <?xml version="1.0" encoding="us-ascii" ?>
    - <tools>
      <tool url="http://lab1:8080/Warehouse/v1.0/WarehouseController.asmx" lastrun="2005:12:19:11:31:7" />
      </tools>


    STEP 2
    =======

    Last Cube Processing
    ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    2005/12/19 01:57:56 PM

    (1 row(s) affected)

    LastVCWHWrite
    -----------------------
    2005-12-05 12:24:55.513

    (1 row(s) affected)

    LastWITWHWrite
    -----------------------
    2005-12-18 18:23:45.703

    (1 row(s) affected)

    WH Revisions
    ------------
    1488

    (1 row(s) affected)

    WIT Revisions
    -------------
    1034

    (1 row(s) affected)

    WH Changeset
    ------------
    225

    (1 row(s) affected)

    VC Changeset
    ------------
    295

    (1 row(s) affected)

    Warehouse Identity Id
    ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    387

    (1 row(s) affected)

    Integration Identity Id
    -----------------------
    387

    (1 row(s) affected)

    Warehouse Structure Id
    ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    365

    (1 row(s) affected)

    Integration Structure Id
    ------------------------
    365

    (1 row(s) affected)

     

     


    STEP 3
    ======
    Done.  See results, above


    STEP 4
    ======

    - Report header:

    Scenario Details
     
     
     Report Generated: 12/19/2005 2:03:46 PM by BBDNET\bbdnet0456; Last Warehouse Update: 2005/12/19 01:57:56 
     
    - Caching is disabled on Report Manager


    3.  Scenario Details and Related Workitem Reports show no data:  The data should be in the warehouse if you run the queries above but there was an issue in the Beta 3 code which caused the data from time to time not to get into the cube during incremental processing.  The workaround is to process the cube fully:
    http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=139746&SiteID=1

    **Done this** (added the line in web.config)

    note the following :

    a.  Line added to show processingtype
    b.  the tracing options are at variance with what you suggest.  General = 0, and this is not under the tracing section.

    you suggest :

    Change: <add name="General" value="1" /> to <add name="General" value="4" />

    whereas the xml here shows:

    ================================

            <switches>
                <add name="API" value="0" />
                <add name="Authentication" value="0" />
                <add name="Authorization" value="0" />
                <add name="Database" value="0" />
                <add name="General" value="0" />
                <!-- Temp: Team Build trace switches -->
                <add name="EqtTraceLevel" value="0" />
                <!-- Temp: Shared TeamBuild/WorkItem trace switches -->
                <add name="traceLevel" value="1" />
                <!-- Temp: Shared BIS/Warehouse trace switches -->
               <add name="eSwitch" value="1" />
            </switches>       

     


    ===================================================================================


        <appSettings>
     <add key="ProcessingType" value="Full" />
            <!-- Timeout for all SQL commands (in seconds) -->
            <add key="sqlTimeout" value="3600" />

            <!--  WEB METHOD LOGGING
                Specify web method logging behavior. The default value is 'None'. Valid
                values and their meaning are:
                  None        (Never log web methods)
                  OnError     (Include web methods that encounter errors)
                  ReadWrite   (Include web methods that change database(s))
                  Normal      (Above, plus web methods that don't change database(s))
                  LightWeight (Above, plus web methods that have minimal database access)
                  All         (Above, plus always include web method request details when avail.)
            -->
            <add key="commandLogging" value="None"/>

            <!-- BisDomainUrl, this allows to find the BIS web server address -->
            <add key="BisDomainUrl" value="http://LAB1:8080" />
            <add key="TFS Name" value=""/>

            <!--
                This switch determines whether tracing is captured to a file
                (true) or not (false). Pay attention to the trace switches in
                the system.diagnostics section; those switches govern the types
                and amount of information collected.
            -->
            <add key="traceWriter" value="false"/>

            <!--
                This switch determines the directory where trace output is placed.
                Trace files are named as Month-Day-Year.Hour-Minute based on the
                time that they're created. If a file exists with the same name, it
                is appended to; otherwise, a new file with that name will be created.
                Note that Month, Day, Hour, and Minute are represented as two digit quantities (and
                hours are expressed in 24-hour format. Year is expressed as a four
                digit number. Environment variables will be expanded before the
                file is created or opened (if it already exists).
                Here are a few examples:
                          Date/Time             File name
                     3/14/2005 12:48:05 PM    03142005.1248
                     10/7/2005 02:48:05 AM    10072005.0248
                      4/7/2005 10:37:05 PM    04072005.2237

                Note that %TEMP% is usually %WINDIR%\Temp.
            -->
            <add key="traceDirectoryName" value="%TEMP%\\TFLogFiles"/>
        </appSettings>

        <system.diagnostics>
            <assert assertuienabled="false" />
            <trace autoflush="false" indentsize="4" />
            <!--  Trace Switches
                Each of the trace switches should be set to a value between 0 and 4, inclusive.
                  0: No trace output
                  1-4: Increasing levels of trace output; see Systems.Diagnostics.TraceLevel
            -->
            <switches>
                <add name="API" value="0" />
                <add name="Authentication" value="0" />
                <add name="Authorization" value="0" />
                <add name="Database" value="0" />
                <add name="General" value="0" />
                <!-- Temp: Team Build trace switches -->
                <add name="EqtTraceLevel" value="0" />
                <!-- Temp: Shared TeamBuild/WorkItem trace switches -->
                <add name="traceLevel" value="1" />
                <!-- Temp: Shared BIS/Warehouse trace switches -->
               <add name="eSwitch" value="1" />
            </switches>       
        </system.diagnostics>

     

    ===================================================================================

     

     

     

     

     

     

     

     

     


    ***Troubleshooting***
    ===============

    App Event Log (Error in TFS Warehouse)


    An unexpected condition has occurred in a Team Foundation component. The information contained here should be made available to your site administrative staff.
    Technical Information (for the administrative staff):
    Date (UTC): 2005/12/19 11:57:34 AM
    Machine: LAB1
    Application Domain: /LM/W3SVC/3/Root/Warehouse-3-127794670044100775
    Assembly: Microsoft.TeamFoundation.Warehouse, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a; v2.0.50727
    Process Details:
      Process Name: w3wp
      Process Id: 2364
      Thread Id: 3416
      Account name: BBDNET\TFSSETUP

    Detailed Message: Microsoft.TeamFoundation.VersionControl.Adapter: An error occurred while processing changeset 226, unexpected exception:
    Exception Message: Object reference not set to an instance of an object. (type NullReferenceException)

    Exception Stack Trace:    at Microsoft.TeamFoundation.VersionControl.Adapter.VCAdapter.persistChangesetInformation(FilePair fp, Boolean doOnlyRenameDebit, String& affectedProjectUri)
       at Microsoft.TeamFoundation.VersionControl.Adapter.VCAdapter.processChangesets()


    For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

     

     


     

    Monday, December 19, 2005 12:25 PM
  •  

    This is related to the following post:

    http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=154637&SiteID=1

    I'm getting the same error.  I have also done a full process on the analysis services (successfully) and followed the troubleshooting guide (successfully).  But still getting the same error.  Any assistance would be appreciated.

    Thanks

    EJ

    Monday, December 19, 2005 6:31 PM
  • Likewise for me.  I have the exact same results as the below post form Lakusha_

    Any guidance would be greatly appreciated.

    In reference to

    ------------------------------------------------

    (beta3 refresh)
    I went through the entire troubleshooting guide, but some reports still have this error:

    • An error has occurred during report processing. (rsProcessingAborted)
      • Query execution failed for data set 'dsWorkItemTypeParam'. (rsErrorExecutingCommand)
        • Query (1, 452) The dimension '[System_WorkItemType]' was not found in the cube when the string, [System_WorkItemType].[System_WorkItemType].[System_WorkItemType], was parsed.

    I haven't found a clue yet. I also have this possibly unrelated (but who knows) error in windows system log:

    The application-specific permission settings do not grant Local Activation permission for the COM Server application with CLSID {BA126AD1-2166-11D1-B1D0-00805FC1270E} to the user NT AUTHORITY\NETWORK SERVICE SID (S-1-5-20). This security permission can be modified using the Component Services administrative tool.

    Checking with regedit, this belongs to "Network Connection Manager Class", but I can't find/fix it in Component Services.

    Anybody knows how to fix this?

    Regards,

    Eric


    --------------------------------

  • Tuesday, December 20, 2005 12:38 PM
  • I too have this same problem as Lakusha and Michael.

    (beta3 refresh)
    I went through the entire troubleshooting guide, but some reports still have this error:

    • An error has occurred during report processing. (rsProcessingAborted)
      • Query execution failed for data set 'dsWorkItemTypeParam'. (rsErrorExecutingCommand)
        • Query (1, 452) The dimension '[System_WorkItemType]' was not found in the cube when the string, [System_WorkItemType].[System_WorkItemType].[System_WorkItemType], was parsed.
  • Any troubleshooting for this problem would be helpful.  BTW, My reports work fine through the visual studio IDE.

    Thanks

    EJ

     

    Wednesday, December 21, 2005 2:10 PM
  • I continue to have the error below on my reports:

    • An error has occurred during report processing. (rsProcessingAborted)
      • Query execution failed for data set 'dsWorkItemTypeParam'. (rsErrorExecutingCommand)
        • Query (1, 452) The dimension '[System_WorkItemType]' was not found in the cube when the string, [System_WorkItemType].[System_WorkItemType].[System_WorkItemType], was parsed.

     

    The trace below shows the results from a warehousecountroller.aspx/run. I have followed all the guidance I can find with the same results.

    Trace:

    00000001 4:46:42 PM [3888] [DW] [Info, PID 3888, TID 4028, 21:46:42.655] ProcessWarehouse entered.  
    00000002 4:46:42 PM [3888] [DW] [Info, PID 3888, TID 5232, 21:46:42.686] Starting run on Adapter Microsoft.TeamFoundation.Warehouse.CommonStructureAdapter.  
    00000003 4:46:42 PM [3888] [DW] [Info, PID 3888, TID 5244, 21:46:42.686] Starting run on Adapter Microsoft.TeamFoundation.Build.Adapter.TeamBuildAdapter.  
    00000004 4:46:42 PM [3888] [DW] [Info, PID 3888, TID 5244, 21:46:42.686] Microsoft.TeamFoundation.Build.Adapter: Entering the Processing state.  
    00000005 4:46:42 PM [3888] [DW] [Info, PID 3888, TID 4072, 21:46:42.686] Starting run on Adapter Microsoft.TeamFoundation.VersionControl.Adapter.VCAdapter.  
    00000006 4:46:42 PM [3888] [DW] [Info, PID 3888, TID 4072, 21:46:42.686] Microsoft.TeamFoundation.VersionControl.Adapter: Entering the Processing state.  
    00000007 4:46:42 PM [3888] [DW] [Verbose, PID 3888, TID 3740, 21:46:42.686] Microsoft.TeamFoundation.WorkItemTracking.Adapter: InitRun called  
    00000008 4:46:42 PM [3888] [DW] [Info, PID 3888, TID 5248, 21:46:42.686] Starting run on Adapter Microsoft.VisualStudio.TestTools.WarehouseAdapter.Adapter.  
    00000009 4:46:42 PM [3888] [CS] [Info, PID 3888, TID 3804, 21:46:42.686] CommonStructureService.ListProjects entered.  
    00000010 4:46:42 PM [3888] [DW] [Verbose, PID 3888, TID 3740, 21:46:42.702] Microsoft.TeamFoundation.WorkItemTracking.Adapter: InitRun returnsNormal  
    00000011 4:46:42 PM [3888] [DW] [Info, PID 3888, TID 3740, 21:46:42.702] Starting run on Adapter Microsoft.TeamFoundation.WorkItemTracking.Adapter.Adapter.  
    00000012 4:46:42 PM [3888] [DW] [Verbose, PID 3888, TID 3740, 21:46:42.702] Microsoft.TeamFoundation.WorkItemTracking.Adapter: Run called  
    00000013 4:46:42 PM [3888] [DW] [Info, PID 3888, TID 3740, 21:46:42.702] Microsoft.TeamFoundation.WorkItemTracking.Adapter: Executing query to retrieve updated Work Items.  
    00000014 4:46:42 PM [3888] [DW] [Info, PID 3888, TID 5244, 21:46:42.702] The job running on adapter Microsoft.TeamFoundation.Build.Adapter.TeamBuildAdapter run finished.  
    00000015 4:46:42 PM [3888] [DW] [Info, PID 3888, TID 5244, 21:46:42.702] Adapter Microsoft.TeamFoundation.Build.Adapter.TeamBuildAdapter exited.  
    00000016 4:46:42 PM [3888] [DW] [Info, PID 3888, TID 5248, 21:46:42.702] The job running on adapter Microsoft.VisualStudio.TestTools.WarehouseAdapter.Adapter run finished.  
    00000017 4:46:42 PM [3888] [DW] [Info, PID 3888, TID 5248, 21:46:42.702] Adapter Microsoft.VisualStudio.TestTools.WarehouseAdapter.Adapter exited.  
    00000018 4:46:42 PM [3888] [VC] [Verbose, PID 3888, TID 3804, 21:46:42.702] Checking whether request: ~/v1.0/repository.asmx requires init to be complete  
    00000019 4:46:42 PM [3888] [DW] [Info, PID 3888, TID 3740, 21:46:42.718] Microsoft.TeamFoundation.WorkItemTracking.Adapter: Retrieved 0 Work Items to process.  
    00000020 4:46:42 PM [3888] [DW] [Info, PID 3888, TID 3740, 21:46:42.718] The job running on adapter Microsoft.TeamFoundation.WorkItemTracking.Adapter.Adapter run finished.  
    00000021 4:46:42 PM [3888] [DW] [Info, PID 3888, TID 3740, 21:46:42.718] Adapter Microsoft.TeamFoundation.WorkItemTracking.Adapter.Adapter exited.  
    00000022 4:46:42 PM [3888] [VC] [Verbose, PID 3888, TID 3512, 21:46:42.718] Checking whether request: ~/v1.0/repository.asmx requires init to be complete  
    00000023 4:46:42 PM [3888] [DW] [Info, PID 3888, TID 5232, 21:46:42.718] The job running on adapter Microsoft.TeamFoundation.Warehouse.CommonStructureAdapter run finished.  
    00000024 4:46:42 PM [3888] [DW] [Info, PID 3888, TID 5232, 21:46:42.718] Adapter Microsoft.TeamFoundation.Warehouse.CommonStructureAdapter exited.  
    00000025 4:46:42 PM [3888] [DW] [Info, PID 3888, TID 4072, 21:46:42.983] Microsoft.TeamFoundation.VersionControl.Adapter: All active changesets have been processed; last changeset processed was 75.  
    00000026 4:46:42 PM [3888] [DW] [Info, PID 3888, TID 4072, 21:46:42.983] The job running on adapter Microsoft.TeamFoundation.VersionControl.Adapter.VCAdapter run finished.  
    00000027 4:46:42 PM [3888] [DW] [Info, PID 3888, TID 4072, 21:46:42.983] Adapter Microsoft.TeamFoundation.VersionControl.Adapter.VCAdapter exited.  
    00000028 4:46:43 PM [3888] [DW] [Info, PID 3888, TID 4028, 21:46:43.171] Starting OLAP processing.  
    00000029 4:46:43 PM [3888] [DW] [Info, PID 3888, TID 4028, 21:46:43.171] Cube processing starting.  
    00000030 4:46:43 PM [3888] [DW] [Info, PID 3888, TID 4028, 21:46:43.358] Found OLAP db: TFSWarehouse  
    00000031 4:46:45 PM [3888] [DW] [Info, PID 3888, TID 4028, 21:46:45.186] <Batch xmlns="http://schemas.microsoft.com/analysisservices/2003/engine">  
    00000032 4:46:45 PM [3888] <Bindings>  
    00000033 4:46:45 PM [3888] </Bindings>  
    00000034 4:46:45 PM [3888] <Parallel MaxParallel="3">  
    00000035 4:46:45 PM [3888] </Parallel>  
    00000036 4:46:45 PM [3888] </Batch>  
    00000037 4:46:45 PM [3888] [DW] [Info, PID 3888, TID 4028, 21:46:45.186] Writing processing results.  
    00000038 4:46:45 PM [3888] [DW] [Info, PID 3888, TID 4028, 21:46:45.186] End of processing results.  
    00000039 4:46:45 PM [3888] [DW] [Info, PID 3888, TID 4028, 21:46:45.217] Cube processing completed.  
    00000040 4:46:45 PM [3888] [DW] [Info, PID 3888, TID 4028, 21:46:45.217] Finished OLAP processing.  
    00000041 4:46:45 PM [3888] [DW] [Info, PID 3888, TID 4028, 21:46:45.217] The cube processing run finished.  
    00000042 4:46:45 PM [3888] [DW] [Info, PID 3888, TID 4028, 21:46:45.217] ProcessWarehouse exited.   

     

    Wednesday, December 21, 2005 2:59 PM
  • Michael,

    I'm having the same problem. I followed the troubleshooting guide and tried a number of other things without success. I tried re-creating the project twice (time consuming process for everybody). We even reinstalled without success.

    How come we do miracles at once but take days/weeks to solve simple problems?

    If you find the solution and post it, I'll light a candle for you under the tree

    Eric

     

    Wednesday, December 21, 2005 3:17 PM
  • Bryan,

     

    I'd like to recommend including the following threads answer as the last step when all else fails from the troubleshooting guide. I finally ran across this in the setup and admin forum and after trying everything else it worked.

     

    Lakusha_, give it a whirl.

     

    http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=171424&SiteID=1

     

     

    Monday, December 26, 2005 2:55 PM
  • Thanks Michael,

    That thread does not solve it but it does point in the right direction: my TFSWarehouse db is missing at least 1 table (dbo.WorkItem). Since the installation process did not report any error, I suppose something is not trap properly. I replied to that thread asking for directions to fix it.

    Thanks again for replying, it is appreciated.

    Eric

     

    Tuesday, December 27, 2005 4:02 AM
  • We have fixed this issue in RTM by restructuring the way we process the warehouse.  We schema changes occured (project creation does this) we used to make relational schema changes, pull the data (most time - prone to shutdowns/app cycles) and then processed the olap schema and the dirty bit was in memory and not persisted.  In RTM it's re-ordered to make all schema changes up front to make the warehouse consistent quicker and we also persist the schema dirty bit.

    By rebuilding the olap cube, (setupwarehouse -o) it forces the schema to get rebuilt via the definition.

    I am updating the troubleshooting guid now.

    1) Back up all data

    2) Delete the TFSWarehouse database in *Analysis Services* (I don't believe this is absolutely necessary but doesn't hurt if you back it up)

    3) From  a command prompt on the TFS Application Tier machine, use the following command (substituting the correct values for your installation in place of the following values: ServerName, Domain\TFSService, Domain\TFSReports):

    • %ProgramFiles%\Microsoft Visual Studio 2005 Team Foundation Server\Tools\setupwarehouse.exe -o -s ServerName -d TfsWarehouse -c Warehouseschema.xml -a Domain\TFSService -ra Domain\TFSReports

     

    Thursday, December 29, 2005 8:38 PM
    Moderator
  • Pinning this useful warehouse thread
    Tuesday, January 3, 2006 6:29 PM
  • Great Post!

    Than You, it was very much needed!

    Thursday, January 5, 2006 8:17 PM
  • I too have one of these reporting errors that aren't fixed after fixing the Locale issue as well as making sure the OLAP cube got processed. It occurs on the 'Project Velocity' report and simply says "An error has occurred during report processing. (rsProcessingAborted)"

    The Report Server log gives more detail. Any help would be appreciated:

    w3wp!library!1!14/12/2005-18:37:57:: i INFO: Call to RenderFirst( '/MYPROJ/Project Velocity' )
    w3wp!processing!9!12/14/2005-18:37:58:: e ERROR: Throwing Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: Exception of type 'Microsoft.ReportingServices.ReportProcessing.ReportProcessingException' was thrown., ; Info: Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: Exception of type 'Microsoft.ReportingServices.ReportProcessing.ReportProcessingException' was thrown.

    w3wp!processing!9!12/14/2005-18:37:58:: e ERROR: An exception has occurred in data source 'dsWorkItemHistoryChart'. Details: Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: The Group expression for the grouping 'Date' contains an error: Argument 'DateValue' cannot be converted to type 'Date'. at Microsoft.ReportingServices.ReportProcessing.ReportRuntime.RegisterRuntimeErrorInExpression(VariantResult& result, Exception e, IErrorContext iErrorContext, Boolean isError)

    Thursday, January 12, 2006 5:23 PM
  • Jim,

    I have the same problems with areas not showing up, but I haven't deleted any projects.

    I do however wrongly install TFS as localcomputer/administrator, and then change the serviceaccount to domain/tfsservice with the TFSAdminUtil. I suspect this has caused the error. Can you check to see if this is a bug, and perhaps suggest a workaround?

    Thanks a lot.

     

    Friday, February 10, 2006 8:49 AM
  • I guys i've everything in the troubleshooting guide but anything has worked for me. I'm using the beta 3 refresh version and although my reports are working they are not current i seen to have stopped in time. Here is the error that i get in the event viewer:

     

    An unexpected condition has occurred in a Team Foundation component. The information contained here should be made available to your site administrative staff.

    Technical Information (for the administrative staff):

    Date (UTC): 2/18/2006 3:44:18 AM

    Machine: WIN2003-DEVSERV

    Application Domain: /LM/W3SVC/3/Root/Warehouse-1-127847078448449208

    Assembly: Microsoft.TeamFoundation.Warehouse, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a; v2.0.50727

    Process Details:

    Process Name: w3wp

    Process Id: 1412

    Thread Id: 3728

    Account name: ENVISION\TFSSERVICE

    Detailed Message: TF51209: A run-time error System.Web.Services.Protocols.SoapException: Invalid project name.

    at Microsoft.TeamFoundation.Proxy.CommonStructureService.GetProjectFromName(String projectName)

    at Microsoft.TeamFoundation.WorkItemTracking.Adapter.Adapter.ProcessProject(FactEntry fact, String factName, String projectName)

    at Microsoft.TeamFoundation.WorkItemTracking.Adapter.Adapter.AddWorkItemFactEntry(PayloadRow dr, String previousState, DateTime dtChangedDate, Boolean isCompensating, Boolean isUpdate, String logicalTrackingId)

    at Microsoft.TeamFoundation.WorkItemTracking.Adapter.Adapter.PopulateFacts(PayloadRow dr, String logicalTrackingId)

    at Microsoft.TeamFoundation.WorkItemTracking.Adapter.Adapter.Run()

    at Microsoft.TeamFoundation.Warehouse.AdapterWrapper.RunTimerAdapter() occurred on adapter Microsoft.TeamFoundation.WorkItemTracking.Adapter.Adapter.

    For more information, see Help and Support Center at

    Saturday, February 18, 2006 3:57 AM
  • That bug should be fixed between B3 and RC.  Can you upgrade?

    It has to do with a project that is mentioned in a workitem but it has been deleted (likely with the /force option which was very bad).  The exception for project not exists was changed from ArgumentException but the proper change wasn't made in the warehouse.

    We could get you past this manually but RC will be a better solution.

    Thanks

    Wednesday, March 1, 2006 4:44 AM
    Moderator
  • Hi Bryan,

    Thanks for your answer. It would be much better if you could send me a TSQL to fix this problem... i'm in a hurry here and i'll not have the time to stop and install de RC version for the next 4 weeks...

    Is it hard for you to create this SQL for me? if it will take much of your time, nevermind, i can live without the reports until i install the RC version...

     

    []'s

    Ciro Fonseca

    Wednesday, March 1, 2006 5:59 PM
  • Hi Bryan,

    I am getting an error on line of discussions in this thread. We are using the new TFS RC. The problem is that while viewing project portals, anyone who is not a member of the default BUILTIN\Administrators group is not able to view the reports in the project portal. Here is the error:

    ==================================================================

    An error has occurred during report processing. (rsProcessingAborted) Get Online Help

    Query execution failed for data set 'IterationParam'. (rsErrorExecutingCommand) Get Online Help

    For more information about this error navigate to the report server on the local server machine, or enable remote errors

    ==================================================================

    I have been tinkering with assigning users to the appropriate roles but it just does not seem to work. I tried giving RSExecRole and SQLAgent* mappings onto master, msdb and Report DBs but nothing seems to solve the problem. I also tried disabling caching of queries / rebuilding the Cube etc as described in this form but nothing helped.

    I would greatly appreciate any guidance.

    Thanks and regards,

    Rahul

    Monday, March 13, 2006 5:17 PM
  • Have you tried adding them to the Reporting Services Browser Role? Browse to http://yourat/reports and click on the properties tag and add them to the Browsers role.  If you're having any latency issues, try stopping and start the Reporting Services windows NT service on the Application tier.

    Hope that helps.

    Tuesday, March 14, 2006 7:16 AM
    Moderator
  • Hi Bryan,

    Thanks for the reply. I have tried all that but to no avail. The only solution seems to be adding users to the BUILTIN\Administrators group. I am trying to fiddle with connections to the DS. Could you please list down the properties for the default TfsOlapReportDS and TfsReportDS connections pls. This is for the MSF Agile Template (TFS RC).

    Also, what DB access rights should I assign to the "Reports Group" I have on my machine? As per this forum, they have access to ReportServer, ReportServerTempDB, Master, and msdb.

    I truly appreciate all your time and effort.

    Thanks and regards,

    Rahul

    Tuesday, March 14, 2006 3:25 PM
  • In practice DataWarehouses are not typically updated real time.  That is the function of operational databases.  Usually big companies would suck data into their database at the end of the business day and process their cubes & reports for the following day.

    The Team Foundation Warehouse is set to update every hour.  You can force an update to happen by calling the WarehouseController.Run web service. 

    In the future we are thinking of making the relational updates more frequent and decoupeling the cube processing.

    Thanks,
    dan

     

    Wednesday, March 15, 2006 10:35 PM
  • Sire,

    This is a known bug in the TFSAdminUtil tool.  You can connect to the TFSWarehouse database and manually update the DataReaderAccount setting in the [dbo].[_WarehouseConfig] table to donmain/tfsservice.

    -Jim

    Wednesday, March 22, 2006 9:02 PM