TestRunChangedNotification not raised for TestRunState.Completed

Question

I have written a server-side handler for several of the the test management events, and I'm finding more and more issues. First and foremost, I can listen for the Microsoft.TeamFoundation.TestManagement.Server.TestRunChangedNotification event to follow changes to a queued test run, but it appears that the Completed state does not actually raise the event. Arguably the two most important states that one would track are run started and run completed, but Completed does not even seem to raise the event! Is there any way to figure this out?!

More generally, I'm finding the design of most of the test management related events fairly useless: there are a number of different events which can be raised, but little to no context is included as to _what_actually_occurred_ to raise the event in the first place. For example:

The TestSuiteChangedNotification event is (presumably) raised to indicate a change in a test suite, but the event args only contains the id of the project & suite which changed, NOT what actually did change. With this information one has to go retrieve the suite and discern manually what actually changed -- which seems near impossible unless your application is constantly keeping track of test suites itself (effectively having to do dual-ledger for everything TFS should already be tracking). Even knowing only what properties of the test suite changed to cause the event would be better than the current implementation. This is like being told: 1) Something happened to Joe. 2) I'm not going to tell you WHAT happened to Joe, and 3) You can't ask Joe what happened. Pretty useless.

Answer 1

Followup note: a snippet of sample code handling the TestRunChangedNotification event (and not being raised currently):

if (notificationType == NotificationType.Notification
    && requestContext.ServiceHost is CollectionServiceHost)
{
 ...
 if (notificationEventArgs is Microsoft.TeamFoundation.TestManagement.Server.TestRunChangedNotification)
 {
     Microsoft.TeamFoundation.TestManagement.Server.TestRunChangedNotification runChangeEvent =
         notificationEventArgs as Microsoft.TeamFoundation.TestManagement.Server.TestRunChangedNotification;
     TeamFoundationTrace.Info("Project {0} Run {1} changed.", runChangeEvent.ProjectName, runChangeEvent.TestRunId);

     ITestManagementService tms = coll.GetService<ITestManagementService>();
     IEnumerable<ITestRun> runs = tms.QueryTestRuns(
         string.Format("SELECT * FROM TestRun WHERE TestRunId={0}", runChangeEvent.TestRunId));
     foreach (ITestRun run in runs)
     {
         TeamFoundationTrace.Info("Run: {0} [{1}]", run.Title, run.State);
     }
 }
...
}

Answer 2

Hi AlanDye,

Thank you for your question.

I am trying to involve someone familiar with this topic to further look at this issue. There might be some time delay. Appreciate your patience.

Thank you for your understanding and support.

Best regards,

---

Lily Wu [MSFT]
MSDN Community Support | Feedback to us

Answer 3

Hi Alan,

It appears to me that you misunderstood API TestRunChangedNotification, this is class, not an event. By the way, there's no documentation public for it, and I believe there's no API to detect test run complete event.

best regards,

---

Forrest Guo | MSDN Community Support | Feedback to manager

Answer 4

Hi Alan,

Do you have follow up question?

Hope above reply get you on right direction.

regards,

---

Forrest Guo | MSDN Community Support | Feedback to manager

Answer 5

Yes, actually I do have some followup thoughts:

YES -- I know that TestRunChangedNotification is a class. It is the class the contains the particulars of the event which was raise (similar to the standard EventArgs pattern). When registering to handle an event raised to a listener on the TFS server (this is a SERVER-SIDE plugin, see any of the number of articles about server-side event handling, such as http://www.codeproject.com/Articles/110292/Team-Foundation-Server-2010-Event-Handling-with-Su or the MSDN documentation, http://msdn.microsoft.com/en-us/library/microsoft.teamfoundation.framework.server.isubscriber.aspx for more info)

When implementing ISubscriber, there is a SubscribedTypes method which returns an array of System.Type containing the types you want to listen for. This is my implementation:

        public System.Type[] SubscribedTypes()
        {
            System.Type[] result = new System.Type[]
            {
                typeof(Microsoft.TeamFoundation.TestManagement.Server.TestRunChangedNotification)
            };
            return result;
        }

When the handler is then called, it is PASSED an instance of TestRunChangedNotification which contains details of the current event being raised (the EventArgs pattern mentioned previously). In this case, the ID of the test run causing the event is a property of TestRunChangedNotification. From this, I then retrieve the ITestRun implementation instance itself which has more details of the run, such as STATE (See the sample code I provided previously)

The problem is, the event handler appears to never be called for the "Completed" state -- it IS called for the "NotStarted" and "InProgress" values of the enumeration -- but never for "Completed". This is the reason of my original question, and has yet to be answered.

Thanks for your thoughts.

Answer 6

OK. I get other people help on this question. Please wait for some time.

best regards,

---

Forrest Guo | MSDN Community Support | Feedback to manager

Answer 7

Hello AlanDye,

You may try to capture the Completed event in TestCaseResultChangedNotification, and the code can be changed to:

TestCaseResultChangedNotification runChangeEvent = notificationEventArgs as TestCaseResultChangedNotification;

ITestManagementService tms = coll.GetService<ITestManagementService>();

IEnumerable<ITestRun> runs = tms.QueryTestRuns(string.Format("SELECT * FROM TestRun WHERE TestRunId={0}", runChangeEvent.TestCaseResultIdentifier.TestRunId));

Thanks & Regards,

Jian-Wei Yu [MSFT]

Microsoft Online Community Support

--------------------------------------------------------------------------------

Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

Answer 8

Thanks Jian-Wei. Unfortunately, this doesn't really solve the problem. I can catch the TestCaseResultChanged notification, but this occurs for EVERY test case in a test plan. This means that not only can I potentially get thousands of these events (if I have a plan which has a large number of test cases), but I STILL don't know if the entire test PLAN is complete, only that some of the tests in the plan completed. I need to know if the containing test plan is finished (to send mail, check status, whatever).

Does this make sense?

Answer 9

Hi AlanDye,

Apologies for delay in response.

In your original question, you had raised 2 questions. First was about the event on run completion and second was about granularity of TestSuiteChangedNotification.

1. On run completion, we raise TestRunCompletedNotification instead of TestRunChangedNotification. So please listen on that event if you are interested in run completion.

2. I agree with you completly that currently our TestSuiteChangedNotification is too high level and we should include some information on what changed. I will file a bug in our system to address this. Thanks for raising this issue.

Regards

Aseem Bansal