locked
App certification kit fail for "hello World" app

    Question

  • Hi, I am developing a Metro app and I wanted to test the cert process.  It failed on "Launch Time Performance" and "Suspend Performance".  I removed all logic and most UI elements.  It is now simply a button (with no code behind) and a textbox.  There is no start-up logic (all commented out) and no additional references beyond those that are there by default.  When run manually the app does what you would expect - opens and closes fine.  However, the app cert still fails it on these two items.  I have run  the test many times and cannot understand the cause.

    Here is the raw XML failure element:   

        <REQUIREMENT NUMBER="17" TITLE="Launch time performance" RATIONALE="The application should have a fast and responsive launch experience while consuming a reasonable amount of system resources (CPU, File IO, Memory etc.) to enable fast switching and multitasking between previously unopened applications.">
          <TEST INDEX="43" NAME="Performance Launch" DESCRIPTION="Application should have a responsive launch time with reasonable CPU, File IO and Memory usage" EXECUTIONTIME="00h:07m:06s.80ms">
            <RESULT><![CDATA[FAIL]]></RESULT>
            <MESSAGES>
              <MESSAGE TEXT="Infrastructure Error: Failure occurred while processing application App.  Please consider re-running the test and avoid interacting with the application while tests are running." />
              <MESSAGE TEXT="Exception:  Infrastructure Error: Timeout occurred during testing while running C:\Program Files\Windows Kits\8.0\App Certification Kit\..\Windows Performance Toolkit\xperf.exe.  Please consider re-running the test and  avoid interacting with the application while tests are running." />
            </MESSAGES>
            <Observed_Values />
          </TEST>
        </REQUIREMENT>
        <REQUIREMENT NUMBER="18" TITLE="Suspend performance" RATIONALE="The application should have a fast and responsive suspend experience while consuming a reasonable amount of system resources (CPU, File IO, Memory etc.) and making sure that resources are released in a performant manner.">
          <TEST INDEX="44" NAME="Performance Suspend" DESCRIPTION="Application should have a responsive suspend with reasonable CPU and File IO usage" EXECUTIONTIME="00h:15m:02s.38ms">
            <RESULT><![CDATA[FAIL]]></RESULT>
            <MESSAGES>
              <MESSAGE TEXT="Infrastructure Error: Failure occurred while processing application App.  Please consider re-running the test and avoid interacting with the application while tests are running." />
              <MESSAGE TEXT="Exception:  Infrastructure Error: Timeout occurred during testing while running C:\Program Files\Windows Kits\8.0\App Certification Kit\..\Windows Performance Toolkit\xperf.exe.  Please consider re-running the test and  avoid interacting with the application while tests are running." />
            </MESSAGES>
            <Observed_Values />
          </TEST>
        </REQUIREMENT>

    Sunday, March 25, 2012 3:12 AM

All replies

  • Hi Shane,

    I need to check if you are running into an issue that is similar to a bug that will be fixed in a future release of xperf.exe.  Do you have a file in %userprofile%\AppData\Local\Microsoft\AppCertKit\LaunchData.etl?  If so, can you double click on it to open it up in Windows Performance Analyzer.  Does it open fine or are there any warnings?  Also how much RAM does the machine have where you are running the appcertkit?

    much appreciated,

    mike

    • Proposed as answer by vaughanknight Thursday, May 10, 2012 11:46 AM
    Monday, March 26, 2012 6:06 PM
    Moderator
  • Thanks for the quick response. That file doesn't exist, however I have a user.etl in that folder.  It opens fine.  The machine is a Toshiba Laptop.  Intel Core i5 with 6Gb RAM running with a dual screen setup.

    I have just run the test again and now it seems ok.  I don't know why as nothing has changed.  The Appstart time is now giving a warning with a start time of 1.02 seconds.  It is also giving a warning on suspend with a suspend time of 0.004sec.  I think the warning is related to CPU utilisation of 7% and 11% in app start and app suspend.  This is odd as the app does nothing at the moment and has only a couple of buttons and text blocks.

    One thing I noticed, is that during the run, a number of etl files are created including kernel.etl.  Yesterday I had a disk space issue which I found was caused by a file kernel.etl in the root of the c: drive being 50Gbyte in size.  I don't think this is a coincidence.

    The warnings about CPU util concern me for an app that does nothing!

    After another run, I now have launchdata.etl and suspenddata.etl.  They both open without errors.

    After examining LaunchData.etl, I think the utilisation is based on overall CPU not the process being tested.


    Shane Michelon

    Tuesday, March 27, 2012 5:33 AM
  • Hi Mike

    I have the same issue as Shane, however I have a generated LaunchData.etl. When I try to open it, the following warning occurs.

    Is this related to the known bug, that you've mentioned? My notebook has 8Gb RAM and 34Gb of free disk space. Is there a workaround for this issue?

    Many thanks,

    Stefan

    Wednesday, March 28, 2012 8:35 AM
  • same problem

    • Infrastructure Error: Failure occurred while processing application App.  Please consider re-running the test and avoid interacting with the application while tests are running.
    • Exception:  Infrastructure Error: Failure occurred during testing, Error code -2147023504 occurred while running E:\Program Files\Windows Kits\8.0\App Certification Kit\..\Windows Performance Toolkit\xperf.exe.  Please consider re-running the test and  avoid interacting with the application while tests are running.

    INFO:[ProcessAction::Run] Starting Run of: E:\Program Files\Windows Kits\8.0\App Certification Kit\..\Windows Performance Toolkit\xperf.exe with arguments -i "E:\Users\Raptor\AppData\Local\Microsoft\AppCertKit\SuspendData.etl" -o "E:\Users\Raptor\AppData\Local\Microsoft\AppCertKit\xperfTemp.txt" -a dumper -provider {315a8872-923e-4ea2-9889-33cd4754bf64}
    INFO:[ProcessAction::WaitForExit] Waiting for exit with timeout of 300000
    INFO:[ProcessAction::myProcess_ErrorDataReceived] error data:   108 Events were lost in this trace.  Data may be unreliable.
    INFO:[ProcessAction::myProcess_ErrorDataReceived] error data:   This is usually caused by insufficient disk bandwidth for ETW logging.
    INFO:[ProcessAction::myProcess_ErrorDataReceived] error data:   Please try increasing the minimum and maximum number of buffers and/or
    INFO:[ProcessAction::myProcess_ErrorDataReceived] error data:   the buffer size.  Doubling these values would be a good first attempt.
    INFO:[ProcessAction::myProcess_ErrorDataReceived] error data:   Please note, though, that this action increases the amount of memory
    INFO:[ProcessAction::myProcess_ErrorDataReceived] error data:   reserved for ETW buffers, increasing memory pressure on your scenario.
    INFO:[ProcessAction::myProcess_ErrorDataReceived] error data:   See "xperf -help start" for the associated command line options.
    INFO:[ProcessAction::WaitForExit] Exitcode: -2147023504


    i7 2600K, 16GB RAM, 10G free on E:


    • Edited by Raptor K Monday, April 02, 2012 2:27 AM
    Monday, April 02, 2012 2:27 AM
  • The same problem, I'm experiencing.

    Before I had installed "April 2012 Update for Visual Studio 11 Beta", this problem did not occur. The following result was obtained.

          <TEST INDEX="43" NAME="起動パフォーマンス" DESCRIPTION="Application should have a responsive launch time with reasonable CPU, File IO and Memory usage" EXECUTIONTIME="00h:00m:58s.99ms">
            <RESULT><![CDATA[WARNING]]></RESULT>
            <MESSAGES>
              <MESSAGE TEXT="Application App has a launch time of 1.257 secs." />
              <MESSAGE TEXT="Application App has CPU utilization of 12.06 percent.  CPU utilization of 50 percent or higher can significantly degrade performance for Application Launch/Suspend." />
              <MESSAGE TEXT="Application App allocates process private memory of 0 MB.  Process private memory usage of greater than 25 MB can significantly degrade performance for Application Launch/Suspend." />
              <MESSAGE TEXT="Application App issues File I/O of  0.084 MB.  Issuing File I/O of 50 MB or higher can significantly degrade performance for Application Launch/Suspend." />
            </MESSAGES>

    However, the FAILED has come after I installed the update today. In addition, the time spent on certification is increasing very much (about 30 minutes).

          <TEST INDEX="43" NAME="起動パフォーマンス" DESCRIPTION="Application should have a responsive launch time with reasonable CPU, File IO and Memory usage" EXECUTIONTIME="00h:08m:02s.50ms">
            <RESULT><![CDATA[FAIL]]></RESULT>
            <MESSAGES>
              <MESSAGE TEXT="Infrastructure Error: Failure occurred while processing application App.  Please consider re-running the test and avoid interacting with the application while tests are running." />
              <MESSAGE TEXT="Exception:  Infrastructure Error: Timeout occurred during testing while running C:\Program Files\Windows Kits\8.0\App Certification Kit\..\Windows Performance Toolkit\xperf.exe.  Please consider re-running the test and  avoid interacting with the application while tests are running." />
            </MESSAGES>

    Then, I uninstalled the update, however the problem continues...

    Postscript: On the next day, I tried again without the update, then, it succeeded this time. The time concerning certification was also several minutes.


    biac [ http://bluewatersoft.cocolog-nifty.com/ ]


    • Edited by biacMVP Wednesday, April 11, 2012 5:51 AM postscript
    Tuesday, April 10, 2012 10:18 AM
  • Hi Shane, Stefan, Biac, Raptor,

    sorry for the late reply!  This is a known issue that will be resolved in a future build of the WACK.  Please feel free to contact me directly at mikewong at microsoft.com if you are not able to wait for another build.

    Specifically, the telltale signs of the bug are if you open the launchdata.etl file and get the dialog that Stefan has posted in the message above.  Another sign that we are running into the bug is

    much appreciated,

    mike

    Thursday, April 12, 2012 9:16 PM
    Moderator
  • mikewong I have same issue as Stefan, need to send my app to store next week, kindly send me fix at sanjeev.agarwals@gmail.com
    Friday, April 27, 2012 5:56 PM
  • I everyone, I'm trying to test my metro app with WACT. I made test three times and the results were different.
    First time failed, the second one "Warning" due to 0,954 second to lunch the application and now is "failed" again with this error:

    Performance Launch

    • Error: Following results were encountered while running the Performance Launch test.
      • Infrastructure Error: Failure occurred while processing application App.  Please consider re-running the test and avoid interacting with the application while tests are running.
      • Exception:  Infrastructure Error: Timeout occurred during testing while running C:\Program Files\Windows Kits\8.0\App Certification Kit\..\Windows Performance Toolkit\xperf.exe.  Please consider re-running the test and  avoid interacting with the application while tests are running

    Performance Suspend

    • Error: Following results were encountered while running the Performance Suspend test.
      • Infrastructure Error: Failure occurred while processing application App.  Please consider re-running the test and avoid interacting with the application while tests are running.
      • Exception:  Infrastructure Error: Timeout occurred during testing while running C:\Program Files\Windows Kits\8.0\App Certification Kit\..\Windows Performance Toolkit\xperf.exe.  Please consider re-running the test and  avoid interacting with the application while tests are running

    The file LaunchData.etl is not present, I found user.etl and kernel.etl in the root , opened fine with the analyzer.
    During test I was not using the pc and test activity takes 25 minutes. Is it normal?

    I've also tried  to test a simple application composed by a page containing a  texbox and a button, the result is the same.

    I've installed all the updates.

    Thanks in advance,

    Sara

    Thursday, May 03, 2012 2:28 PM
  • This may be a known issue/bug - but here is the workaround that worked for me:

    Delete everything under \Users\[YOUR_USERNAME]\AppData\Local\Microsoft\AppCertKit and re-run the WACK tool again.


    Krishna

    Tuesday, May 08, 2012 11:16 AM
  • I do have the same problem: It failed on "Launch Time Performance" and "Suspend Performance". I will have my Microsoft evaluation Lab this Monday and I have to bring my certification done correctly. I tested an empty app with the split templates, and I get the same failure as my app does.

    Is there a way you may help me?


    Lina

    Friday, May 11, 2012 4:17 PM
  • Hi Mike,

    I am also seeing this issue. In a blank application, the WACK returns a failed result. I do have the LaunchData.etl file that seems to open properly, however the test fails "Performance Suspend", "Debug App Check" and produces a warning for "Performance Launch". I have included a screen shot for the opened LaunchData.etl file as I am unsure if I am experiencing the correct behavior (see below). I am running Win8 on a virtual machine with 4GB of memory. Is there any news of when the new build of the WACK will be released or is there a known work around for this issue?

    Thanks,

    Kelly

    Tuesday, May 15, 2012 8:59 PM