locked
Load testing: Virtual User Activity Chart disappears for distributed tests RRS feed

  • Question

  • We have a problem that the virtual user activity chart is empty when we run distributed tests. When we run tests locally, we see the chart OK, but if we change the test settings to run on a test rig instead of running locally (but make no other changes), the chart stays empty after the test has run and when we try to display it, we see a dialog that says 'no detail exists at the selected point'.

    Does anyone know what's going on or what we can do to avoid losing the chart?

    Thanks!

    Thursday, May 31, 2012 2:03 PM

Answers

  • Hi Jack, thanks for checking in.

    I have progress on this issue. A strong indication that the problem is that certain specific agents are not sending full data back to the controller.

    I was using a setup of:

    Machine 1: VS + agent.

    Machine 2: Controller

    Machines 3 + 4: Agents

    In other words, 3 agents.  (I did try your suggestions of trying out a different load test, and resetting VS settings - these had no affect. I had also had previously seen the same problem with VS on machine 2, with the controller, everything else the same).

    The apparent breakthrough came when a new virtual machine (call it M5) became available to use as an agent, so I tried substituting it for M1, giving M1: VS, M2: Controller, M3,4,5: Agents.

    To my surprise, the virtual user chart immediately started appearing.  This made think of two possibilities: Perhaps the agent service on M1 was faulty and breaking everything, or perhaps having VS share a machine with an agent or controller was breaking it. So to investigate further, I re-added M1 as an agent - so now using 4 agents. To my surprise the VU  (virtual user) chart was there - which seemed to kill both those theories. But then I noticed an oddity: My load test had 10 VU's but the VU chart was only showing 2.  So I tried a couple of experiments:

    • Put all agents offline except the new M5. Result: VU chart shows 10 VUs - working normally.
    • Change to just M5 and M3 online - so the new agent plus one old agent. Result: VU chart shows 5 VUs.

    This all seems very consistent with a theory that the VU chart is only showing data from the new machine M5.

    So - to track this down more: Two questions:

    1. Is there a way to tell which VU is running on which agent (so I can confirm the VUs being displayed are the ones from M5)?
    2. Is there anything you know of that would stop an agent sending detailed VU info back? (But still so all the other results tabs show results)?

    (Or do you have any other ideas?)

    • Edited by GreatElectron Thursday, June 7, 2012 4:51 PM Updating with more recent tests
    • Marked as answer by GreatElectron Friday, June 8, 2012 11:00 AM
    Thursday, June 7, 2012 2:24 PM
  • Hi GreatElectron,

    Glad to receive your reply.

    Is there a way to tell which VU is running on which agent (so I can confirm the VUs being displayed are the ones from M5)?

    As my understanding, VS would not run a test/ many tests with one controller and many agents simultaneously, as you said that if just one specific agent generated this issue, you would try to run your load test just with the specific agent, as you said that if it has the same issue when you run all load test with this agent, my suggest is that you could try to reinstall and check it.

    Of course, when there are multiple agents associated with the controller, we can filter the agents on which the test has to be run based on properties, for detailed information, see http://blogs.msdn.com/b/mathew_aniyan/archive/2009/03/03/distributed-coded-ui-tests.aspx.

    Is there anything you know of that would stop an agent sending detailed VU info back?

    It really hard to make sure what generated this issue, if reinstall the agent still no help, I really suggest you submit this feedback to Microsoft Connect feedback portal: http://connect.microsoft.com, Microsoft engineers will evaluate them seriously. I will vote it. Thanks for your understanding.

    Have a nice weekend,


    Jack Zhai [MSFT]
    MSDN Community Support | Feedback to us

    • Marked as answer by GreatElectron Friday, June 8, 2012 11:00 AM
    Friday, June 8, 2012 3:17 AM
    Moderator

All replies

  • Hi GreatElectron,

    Thank you for posting in the MSDN forum.

    Would you mind letting us know more information? What is your load test result in the Test Results window? Is it completed?

    If we want to run a load test remotely, we need to install the Controller and Agent, did you install them?

    See “How to: Run a Load Test Using Test Controllers and Test Agents Specified in a Test Setting

    Best Regards,


    Jack Zhai [MSFT]
    MSDN Community Support | Feedback to us

    Friday, June 1, 2012 8:56 AM
    Moderator
  • Hi Jack, yes the load test is completed when we see the results. ('Test completed' is displayed at the top of the Summary tab on the results page). 

    We have installed the controller and agents. (and if you haven't already guessed, this is on VS 2010 Ultimate)

    What we are seeing is that if I run a load test locally, it works fine - no errors - and all the results appear in all tabs in the results window (summary, graphs, tables, detail).

    I then change the active .testsettings file in the solution to have it run tests remotely. I don't make any changes to the test itself, but simply re-run it. The test again completes without any errors, but for this run, if I look at the various tabs in the results window, I see that all the tabs *except* the Detail tab are present and full of the expected data (as a double check, the 'Controller and Agents' data now includes graph and tabular data from all the various machines). The problem is with the detail tab which, for the Virtual User Activity Chart, now displays nothing except for popping up the 'there is no data' dialog - that's the problem.  This behaviour seems to be reliably reproducible on our system whenever we run a remote test, but we never see it when we run a local test. I wondered if there might be some issue that could prevent the agents sending back details data to the controller (but the controller is able to read the summary data required for the other tabs)?

    To be more specific about the change I'm making when I say I'm modifying the .testsettings file: I change the Test execution method drop-down from 'local execution' to 'remote execution' and select the name of the machine that the controller sits on from the 2nd drop-down.  


    • Edited by GreatElectron Friday, June 1, 2012 11:05 AM Extra clarification
    Friday, June 1, 2012 11:02 AM
  • Hi GreatElectron,

    Glad to receive your reply.

    1. Open Test->Manage Test Controller, then check whether you select the controller and Load test results store.
    2. In addition, could you share us more information about your settings?

    For example,

    Machine1:  installed the VS and the Agent.

    Machine2:  installed the controller.

    I try to run a load test with “Local execution with remote collection”, I could get the Virtual User Activity Chart.

    Or

    Machine1: VS and the controller

    Machine2:  the Agent

    Run a load test with “remote execution”, I could get the Virtual User Activity Chart.

      3.  Since it works well in my machine, if possible, I suggest you try to run other load tests to check whether it has the same issue, if all load tests have the same issue, to make sure that it is not related to your VS settings, try to reset your VS settings to check it.

      4.  If still not help, you could try to run it with other Controller/Agent to check it.

    Best Regards,


    Jack Zhai [MSFT]
    MSDN Community Support | Feedback to us

    Monday, June 4, 2012 5:15 AM
    Moderator
  • Hi GreatElectron,

    I am writing to check the status of the issue on your side. 

    What about this problem now?

    Would you mind letting us know the result of the suggestion?

    Best Regards,


    Jack Zhai [MSFT]
    MSDN Community Support | Feedback to us

    Thursday, June 7, 2012 5:10 AM
    Moderator
  • Hi Jack, thanks for checking in.

    I have progress on this issue. A strong indication that the problem is that certain specific agents are not sending full data back to the controller.

    I was using a setup of:

    Machine 1: VS + agent.

    Machine 2: Controller

    Machines 3 + 4: Agents

    In other words, 3 agents.  (I did try your suggestions of trying out a different load test, and resetting VS settings - these had no affect. I had also had previously seen the same problem with VS on machine 2, with the controller, everything else the same).

    The apparent breakthrough came when a new virtual machine (call it M5) became available to use as an agent, so I tried substituting it for M1, giving M1: VS, M2: Controller, M3,4,5: Agents.

    To my surprise, the virtual user chart immediately started appearing.  This made think of two possibilities: Perhaps the agent service on M1 was faulty and breaking everything, or perhaps having VS share a machine with an agent or controller was breaking it. So to investigate further, I re-added M1 as an agent - so now using 4 agents. To my surprise the VU  (virtual user) chart was there - which seemed to kill both those theories. But then I noticed an oddity: My load test had 10 VU's but the VU chart was only showing 2.  So I tried a couple of experiments:

    • Put all agents offline except the new M5. Result: VU chart shows 10 VUs - working normally.
    • Change to just M5 and M3 online - so the new agent plus one old agent. Result: VU chart shows 5 VUs.

    This all seems very consistent with a theory that the VU chart is only showing data from the new machine M5.

    So - to track this down more: Two questions:

    1. Is there a way to tell which VU is running on which agent (so I can confirm the VUs being displayed are the ones from M5)?
    2. Is there anything you know of that would stop an agent sending detailed VU info back? (But still so all the other results tabs show results)?

    (Or do you have any other ideas?)

    • Edited by GreatElectron Thursday, June 7, 2012 4:51 PM Updating with more recent tests
    • Marked as answer by GreatElectron Friday, June 8, 2012 11:00 AM
    Thursday, June 7, 2012 2:24 PM
  • Hi GreatElectron,

    Glad to receive your reply.

    Is there a way to tell which VU is running on which agent (so I can confirm the VUs being displayed are the ones from M5)?

    As my understanding, VS would not run a test/ many tests with one controller and many agents simultaneously, as you said that if just one specific agent generated this issue, you would try to run your load test just with the specific agent, as you said that if it has the same issue when you run all load test with this agent, my suggest is that you could try to reinstall and check it.

    Of course, when there are multiple agents associated with the controller, we can filter the agents on which the test has to be run based on properties, for detailed information, see http://blogs.msdn.com/b/mathew_aniyan/archive/2009/03/03/distributed-coded-ui-tests.aspx.

    Is there anything you know of that would stop an agent sending detailed VU info back?

    It really hard to make sure what generated this issue, if reinstall the agent still no help, I really suggest you submit this feedback to Microsoft Connect feedback portal: http://connect.microsoft.com, Microsoft engineers will evaluate them seriously. I will vote it. Thanks for your understanding.

    Have a nice weekend,


    Jack Zhai [MSFT]
    MSDN Community Support | Feedback to us

    • Marked as answer by GreatElectron Friday, June 8, 2012 11:00 AM
    Friday, June 8, 2012 3:17 AM
    Moderator
  • Is there a way to tell which VU is running on which agent (so I can confirm the VUs being displayed are the ones from M5)?

    Actually turns out this is really simple. Just hover the mouse over each bar in the VU activity chart and the tooltip tells you which agent ran that VU!  After some more testing using that feature, I'd now be confident enough to say I *know* that our problem is that data from only some of our agents is being reported in the VU chart.  This means that we have an obvious fix - either, as you say, reinstall on the problematic agents, or simply replace those agents altogether.  So from that point of view I imagine my original problem is fixed.

    I'll keep in mind your other suggestion of submitting feedback. Unfortunately, as yet I have no idea how to reproduce the issue or what causes it. Our agents that work are so far as I can see no different in their setup from the ones whose data isn't making it to the VU chart. That means that the situation I'm in amounts to: 'We have a bug that's on our system, I now have a workaround for it, but I can't tell anyone how to reproduce it' - I imagine I need more than that before I can sensibly submit to connect.microsoft.com ;)   I'll let you know if I get any more info that makes submitting it worthwhile.

    In the meantime I'll mark this as answered - thanks for all your advice, Jack. Much appreciated.

    Friday, June 8, 2012 11:00 AM
  • You're welcome!

    If you need further assistance, please feel free to let us know.

    Have a nice day,


    Jack Zhai [MSFT]
    MSDN Community Support | Feedback to us

    Monday, June 11, 2012 1:38 AM
    Moderator