locked
Occasionally Failing UnitTest when Tasks are involved RRS feed

  • Question

  • Hi,

    I currently have 22/650 unit-test which occasionally fail. When a test fails and I start it again, it mostly passes (>99%).
    The failing rate is very low. Mostly (70-80%) "Run All" passes without failure. Still I can't tollerate ocasionally failing tests.

    The failing tests cover a bunch of viewmodels, which use tasks to handle "incoming messages".
    The tests fail because they don't process a incoming message within a given time (Timeout).

    Investigating this issue I realized, that (mostly if not always) the task in question takes to long to begin with the execution of its action.
    In the case of failure I measured delays of well over one second from the moment when calling:
    "Task.Factory.StartNew(this.Execute, TaskCreationOptions.LongRunning);" to the moment "this.Execute" is entered.

    Is there something I can do to keep my unit-tests from failing ?

    Regards
    Rainer

    Friday, March 21, 2014 9:39 AM

Answers

  • Hi,

    Could you provide us  code in your unit test method so that we can further look at this issue?

    If possible, could you provide us a sample to repro this issue?

    Thanks,


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.


    • Edited by Amanda Zhu Monday, March 24, 2014 9:56 AM edit
    • Marked as answer by Rainer Queck Monday, March 24, 2014 12:35 PM
    Monday, March 24, 2014 9:56 AM
  • Hello Crystal Zhu,

    It looks like I found "MY BUG":

    Actually I recognized it as I prepared the code snippets for you. The fault (my fault) is cause because the thread is not to slow but to fast ;-)

    Here is what I am doing wrong:

    1. Start Task
    2. Set IsWorkerTaskRunning to false
    3. Expect the task to set IsWorkerTaskRunning to true...

    If now the task starts very fast, it has set IsWorkerTaskRunning to true, before I set it to false in step 2

    I have modified my code to first set IsWorkerTaskRunning to false and then start the Task an since then I did about 40 "RunAll Tests" without a single one failing :-)

    Thank you for your help! It was the fact, that I had to prepare the snippets which then brought the problem points into my vision field.

    Regards
    Rauber

    • Marked as answer by Rainer Queck Monday, March 24, 2014 12:35 PM
    Monday, March 24, 2014 12:34 PM

All replies

  • Hi,

    The tests fail because they don't process a incoming message within a given time (Timeout).

    As far as I know, we can increase the Timeout for a test run in .testsettings file. Please see: http://msdn.microsoft.com/en-us/library/ee256991.aspx#VSTestSettingsTestTimeouts

    It seems that that Timeout issue occurred at the code under test, if yes, I think that the development forum should be a better forum. Please consult the Timeout issue on proper development forum based on the type of your project under test.

    Best regards,


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Monday, March 24, 2014 7:38 AM
  • Hi,

    thank you for your response. May be you are right about, moving to a other forum, but this issue seems to exist only in the test environment. When running the application, I don't experience this issue.

    Could it be, that MSTest is causing this issue because of running a bunch of test methods in one test class which all test the same viewmodel more or less at the same time?

    Regards
    Rainer

    Monday, March 24, 2014 8:27 AM
  • Hi,

    Could it be, that MSTest is causing this issue because of running a bunch of test methods in one test class which all test the same viewmodel more or less at the same time?

    In order to check whether this issue is related to unit test, could you describe your project under test(is it the viewmodel?) and your test scenario in details? All unit test methods are in one test class? And they tested the same viewmodel? Did you run all of them at the same time  through ‘Run All’? The failed test can succeed when you run it alone again?

    For those failed tests, whether there are any error messages? If yes, please post here.

    Thanks,


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Monday, March 24, 2014 8:58 AM
  • Hi,

    there currently are 24 occasionally failing unit tests spread over 3 test project with a total of 5 test classes.
    There is one test class per viewmodel. Since I am using Prism I have 3 Prism-Modules and there is one test project per module.

    When the tests fail, there are no other error messages. They simply fail because it takes too long to start the Task. If I increase the timeout to two seconds they fail less often, but this still happens.
    You might say, to just extend the timeout, but this seems not to be a solution to me. I would rather like to understand on why it takes that long, because from my point of view it should only take a few milliseconds from "Task.Factory.StartNew" to "entering the task action method.

    Regards
    Rainer


    Monday, March 24, 2014 9:28 AM
  • Hi,

    Could you provide us  code in your unit test method so that we can further look at this issue?

    If possible, could you provide us a sample to repro this issue?

    Thanks,


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.


    • Edited by Amanda Zhu Monday, March 24, 2014 9:56 AM edit
    • Marked as answer by Rainer Queck Monday, March 24, 2014 12:35 PM
    Monday, March 24, 2014 9:56 AM
  • Hi,

    sorry, I can not provide you with a sample to reproduce the issue.

    Below you will find the code snippets to show you what we are doing. All tests in question are setup in the same way.

    Please be aware: Although I added the test method below, The tests fail in the TestInitialization phase when "PrepareViewModel" is called -> "Timeout on waiting for Task to be started"!

    Regards
    Rainer

    ---------------------------- unit test ------------------------

    Test Initializing an Cleanup:

    #region Test Intializing and Cleanup
    [ClassCleanup]
    public static void MyClassCleanup()
    {
        // empty
    }
    
    [ClassInitialize]
    public static void MyClassInitialize(TestContext testContext)
    {
        // empty            
    }
    
    [TestCleanup]
    public void MyTestCleanup()
    {
        this.viewModel.StopTask();
    }
    
    [TestInitialize]
    public void MyTestInitialize()
    {
        this.PrepareViewModel();
    }
    #endregion
    
    ....... 
    private void PrepareViewModel()
    {
        this.PrepareUnityContainer();
        this.viewModel = new ContentGenerateJobViewModel(this.unityContainer);
    
        var timeStamp = DateTime.Now;
        while (!this.viewModel.IsWorkerTaskRunning)
        {
            var dt = DateTime.Now - timeStamp;
            Assert.IsTrue(dt.TotalSeconds < 1, "Timeout on waiting for Task to be started");
        }
    }

    A unit test:

    [TestMethod]
    [TestCategory(TestConstants.OccasionallyFailingTest)]
    public void When_PrintJobGenerated_event_fires_then_properties_are_set()
    {
        // Arrange
        var PrintJobGeneratedMessage = new EbcMessage(null, EbcMessageType.PrintJobGenerated);
    
        // Act
        this.viewModel.Input(PrintJobGeneratedMessage);
    
        // Assert
        var timeStamp = DateTime.Now;
        while (this.viewModel.IsReadyLabelVisible != true)
        {
            var dt = DateTime.Now - timeStamp;
    
            // Assert
            Assert.IsFalse(dt.TotalSeconds > 1, "Timeout on waiting for PrintJobGeneratedMessage sent to be true");
        }
    
        // Assert
        Assert.AreEqual(this.viewModel.IsReadyLabelVisible, true);
    }

    ---------------------------- view model --------------------

    public ContentGenerateJobViewModel(IUnityContainer unityContainer)
    {
    ...
        this.PrepareWorkerTask();
    ...
    }
    
    private void PrepareWorkerTask()
    {
        this.guiDispatcher = Dispatcher.CurrentDispatcher;
        this.workerTaskEvent = new AutoResetEvent(false);
        Task.Factory.StartNew(this.Execute, TaskCreationOptions.LongRunning);
        this.IsWorkerTaskRunning = false;
    }
    
    private void Execute()
    {
        try
        {
            this.taskThreadReference = Thread.CurrentThread;
            this.IsWorkerTaskRunning = true;
            this.DoContextStartup();
    
            while (!this.terminated)
            {
                try
                {
                    if (this.workerTaskEvent.WaitOne(TaskWaitMilliSec, false))
                    {
                        this.ProcessMessages();
                    }
    
                    this.DoIdleWork();
                }
                catch (Exception exception)
                {
                    Logger.Error(LogMessages.Exception000, exception.FullMessage());
                }
            }
    
            this.DoContextCleanup();
            this.IsWorkerTaskRunning = false;
        }
        catch (Exception e)
        {
            Logger.Fatal(e.FullMessage);
        }
    }

    • Marked as answer by Rainer Queck Monday, March 24, 2014 12:35 PM
    • Unmarked as answer by Rainer Queck Monday, March 24, 2014 12:35 PM
    Monday, March 24, 2014 11:00 AM
  • Hello Crystal Zhu,

    It looks like I found "MY BUG":

    Actually I recognized it as I prepared the code snippets for you. The fault (my fault) is cause because the thread is not to slow but to fast ;-)

    Here is what I am doing wrong:

    1. Start Task
    2. Set IsWorkerTaskRunning to false
    3. Expect the task to set IsWorkerTaskRunning to true...

    If now the task starts very fast, it has set IsWorkerTaskRunning to true, before I set it to false in step 2

    I have modified my code to first set IsWorkerTaskRunning to false and then start the Task an since then I did about 40 "RunAll Tests" without a single one failing :-)

    Thank you for your help! It was the fact, that I had to prepare the snippets which then brought the problem points into my vision field.

    Regards
    Rauber

    • Marked as answer by Rainer Queck Monday, March 24, 2014 12:35 PM
    Monday, March 24, 2014 12:34 PM