Data missing when changing query used by TFS Integration Platform

Question

Hi,

I recently used the TFS integration platform to migrate all work items contained in the TFS 2010 team project A to the TFS 2010 team project B (both team projects are in the same project collection).  In order to validate my configuration of the TFS integration platform, the first thing I tried was to migrate all work items of type "Task" using the query "[System.TeamProject] = @project and [System.WorkItemType] = 'Task'".  Then, once I saw that all the tasks were migrated properly, I was convinced that the tool worked correctly and I was ready to do it with everything else.  I therefore changed the query of my current configuration (I didn't create a new configuration) to migrate everything: "[System.TeamProject] = @project".  

At first, it looked like everything was migrated.  After a few days, I realized that many non-task work items were not migrated.  This is because the TFS integration platform keeps track the biggest work item ChangedDate everytime a migration is done.  Then, when you start a new migration, it will only take the work items that have been modified since that ChangedDate. In my case, the biggest 'Task' work items ChangedDate migrated so far was 2011-06-09.  So, when starting the migration (on 2011-08-09), I expected to get all the remaining work item types (bugs, product backlog items, etc), it took only the ones that had been changed AFTER 2011-06-09.  I therefore was missing a lot of work items.

Note that I understook that logic by looking at the log files.  Here is the 'grep' output that made me understand this:

 

 

8/9/2011 1:47:41 PM] TfsMigrationShell.exe Information: 0 : WorkItemTracking: TFS Query: SELECT [System.Id], [System.Rev] FROM WorkItems WHERE ([System.TeamProject]=@project AND ([System.Id] = 0) AND [System.ChangedDate] > '2011-08-09 17:42:06Z') ORDER BY [System.Id] 
/9/2011 1:56:29 PM] TfsMigrationShell.exe Information: 0 : WorkItemTracking: TFS Query: SELECT [System.Id], [System.Rev] FROM WorkItems WHERE ([System.TeamProject]=@project AND ([System.TeamProject] = @project and [System.WorkItemType] = 'Task') AND [System.ChangedDate] > '2010-09-15 20:54:41Z') ORDER BY [System.Id] 
8/9/2011 1:57:37 PM] TfsMigrationShell.exe Information: 0 : WorkItemTracking: TFS Query: SELECT [System.Id], [System.Rev] FROM WorkItems WHERE ([System.TeamProject]=@project AND ([System.TeamProject] = @project and [System.WorkItemType] = 'Task') AND [System.ChangedDate] > '2010-09-15 20:54:41Z') ORDER BY [System.Id] 
8/9/2011 1:57:42 PM] TfsMigrationShell.exe Information: 0 : WorkItemTracking: TFS Query: SELECT [System.Id], [System.Rev] FROM WorkItems WHERE ([System.TeamProject]=@project AND ([System.Id] = 0) AND [System.ChangedDate] > '2011-08-09 17:46:41Z') ORDER BY [System.Id] 
/9/2011 2:12:51 PM] TfsMigrationShell.exe Information: 0 : WorkItemTracking: TFS Query: SELECT [System.Id], [System.Rev] FROM WorkItems WHERE ([System.TeamProject]=@project AND ([System.TeamProject] = @project) AND [System.ChangedDate] > '2011-06-10 17:37:01Z') ORDER BY [System.Id] 
[8/9/2011 2:19:58 PM] TfsMigrationShell.exe Information: 0 : WorkItemTracking: TFS Query: SELECT [System.Id], [System.Rev] FROM WorkItems WHERE ([System.TeamProject]=@project AND ([System.TeamProject] = @project) AND [System.ChangedDate] > '2011-06-10 17:37:01Z') ORDER BY [System.Id] 

 

So my point is the following: I would have never expected that behavior.  So would it be possible to enhance the UI to flag to the users that changing the query after a first migration has been done is dangerous?  It would have saved me a lot of time...

Thanks a lot for understanding,

Maxime

Answer 1

Hi Maxime,

You're correct in your analysis of the behavior.  Whenever the filter is changed, only work items that have been modified since the last sync and are in the new filter will get migrated.  Another caveat is for work items that are no longer in the filter, the work items will become "orphaned" and will no longer get synced.

Your feedback for a more visible warning of these scenarios is great.  Ideally, I'd also like to see a feature where the user can choose to have all work items re-analyzed if they'd like (but not have it required in the case where there are many many work items).

Thanks,
Erin

Answer 2

So, what's the solution? Can I reset the "last sync" time?

---

John Saunders
WCF is Web Services. They are not two separate things.
Use WCF for All New Web Service Development, instead of legacy ASMX or obsolete WSE
Use File->New Project to create Web Service Projects

Answer 3

You create a new configuration if you want the tool to forget what it has done.  Consider the scenario where the change to the query in this case was to add a clause instead of to remove one.  Most users wouldn't expect the tool to delete work items in the target system.  It may sometimes be possible to go and weave together a consistent historical record of work items under purely additive work item query transformations, but those are kind of tough to cleanly isolate.  The tool is biased to support an iterate early with new configurations and then migrate when you are happy model more than the incremental development model.

Bill