locked
Why do the testers give no detailed feedback when the app has failed to meet requirement 1.2 RRS feed

  • Question

  • This situation is very annoying. A short sentence what happened with the app (crash, missing functionality) might help to actually solve the problem before resubmitting the app. For both sides that would be less stressful.

    First of all, point 1.2 is too broad: "Anything that might cause our testers to think that your app is not completely finished will cause your app to fail certification."
    What does that even mean? If a picture viewer does only support jpeg2000 images, is it incomplete in the eyes of the testers, or is it the publishers intention to only support jpeg2000 images in his app?
    Why can the testers decide when an app is completely finished? And how come certain apps in the store don't look completly finished to me, yet had no problem passing the testers? And why do they not at least describe why they think the app is not complete?

    I hope MSFT fixes this until GA.

    Wednesday, June 20, 2012 8:38 AM

Answers

  • Now I will defend the team a bit here through my current certification nightmare, things have improved slightly over the past few weeks.

    There is a link on the cert failure report next to each failure pointing to the full policy descriptions (although the policies still need more detail and help/guidance plus the link should jump to the specific cert)

    They do now include a pdf report in a sip, depending on the cert failure you may get multiple reports, 1 per section ( although there is limited detail and a hand written paragraph would go a long ways better than an autogenerated report)

    Granted there is still a long way to go to improve the entire process and they could learn some tricks from the phone team (escape the silo guys)

    But! They are working hard with incremental improvements coming which are very appreciated, with the looming launch of the store and the more apps racing to the finish this does need more sooner


    Darkside Admin and blogger on #XNAUK - http://xna-uk.net Master Chief of @Zenith_Moon studios

    • Proposed as answer by mark_1h Monday, October 1, 2012 10:04 PM
    • Marked as answer by Roberts_EModerator Tuesday, April 30, 2013 9:21 PM
    Monday, October 1, 2012 4:55 PM

All replies

  • ehem........ some of the apps are created by Microsoft Employees. Hope that explains the "rare events". *rolls eyes*.

    But jokes aside, i believe point 1.2 is to prevent junk apps from going to the store, you know...... junk apps, apps that are created for testing purposes, or lack content. Maybe apps that only have one level, or apps whose content is only a picture with no interactivity (it's still a non crashing app per se, but it's a junk app). Hope that helps.

    Wednesday, June 20, 2012 12:07 PM
  • Sure, but what if the app is not a junk app and actually does something useful? Anyway, without specific feedback from the testers how do we know whats wrong with the app in the eyes of the testers? Do we need MS Mindread(tm) for that?
    Wednesday, June 20, 2012 12:09 PM
  • now that is a pretty darn good one. The employees are the one that can answer that question.
    Wednesday, June 20, 2012 12:13 PM
  • so uhm, when they reject your app, they don't send you an email or notifications on why it's rejected ? they just say it did not fulfill point 1.2
    Wednesday, June 20, 2012 12:31 PM
  • Yes, that is so. No further information.
    Wednesday, June 20, 2012 12:35 PM
  • Yes, that is so. No further information.

    I know this is an old thread, but it would appear that nothing has changed, I've been given next to nothing in the way of information about a fail.  One of the fail reasons is not having a privacy statement, which I do!

    Why not give a little to [url=http://www.justgiving.com/teams/BBQGames]charity[/url]?

    Tuesday, September 18, 2012 1:00 PM
  • Yes, that is so. No further information.


    I know this is an old thread, but it would appear that nothing has changed, I've been given next to nothing in the way of information about a fail.  One of the fail reasons is not having a privacy statement, which I do!

    Why not give a little to [url=http://www.justgiving.com/teams/BBQGames]charity[/url]?


    Just got off a chat session with Microsoft support and this is something that they are working on.  They (a chap called John) were very helpful in giving me further information on my failure issues.

    Why not give a little to [url=http://www.justgiving.com/teams/BBQGames]charity[/url]?

    Tuesday, September 18, 2012 1:53 PM
  • How did you contact them?
    Tuesday, September 18, 2012 2:31 PM
  • MarkMD76,

    There are two places where the privacy info is needed.  My question to you would be:

    Do you have a privacy URL listed in the description data (in the onboarding workflow)?

    Do you have a privacy statement in your settings charm (in the app itself)?

    Thanks,

    Ben

    Wednesday, September 19, 2012 12:02 AM
  • From my view this is getting ridiculous now, i'm not a basher of MS support because I think they do a real good job in general but this REALLY needs sorting.

    I have an app that is ALREADY published in the store, but subsequent updates have gone like this:

    1st attempt - Fail Cert 1.2 - no details as to why because content hadn't changed from the published version

    2nd attempt - removed any reference to "coming soon" or descriptions of coming features, result Failed for Cert 3.1, granted I had been working on the app and hadn't disabled / removed some beta code so my fault !  NOTE no reference to a 1.2 fail, it passed that

    3rd attempt and where I'm really annoyed now, it failed again for 1.2.  It is the stripped down app and all I have added is AdDuplex.

     

    Sorry but without any guidance, notes or even a 1 line comment how are we supposed to know why it is actually failing, given the 3-5 day turn around this is antagonising to say the least.

     

    If the publishing process cannot support feedback at present, at least give a unique or general contact point where we can query for more info FROM THE CERTIFICATION FAILURE REPORT


    Darkside Admin and blogger on #XNAUK - http://xna-uk.net Master Chief of @Zenith_Moon studios


    Wednesday, September 19, 2012 7:35 AM
  • Its sad that this process has still not improved. And after seeing a lot of apps carrying the default SDK sample icon in the store lately, one really has to wonder how easy it seems for some people to get medicore (at best) apps through certification.

    We as developers have a lot of requirements to met so there should at least be one requirement for the store cert people too: give a one sentence explanation why you think the cert failed.

    Wednesday, September 19, 2012 12:56 PM
  • MarkMD76,

    There are two places where the privacy info is needed.  My question to you would be:

    Do you have a privacy URL listed in the description data (in the onboarding workflow)?

    Do you have a privacy statement in your settings charm (in the app itself)?

    Thanks,

    Ben

    Thanks Ben, but I have managed to resolve this issue and it was human error on the part of the reviewer.  After talking to one of your colleagues, he suggested that I include notes to the tester pointing out where the policy was located ie Settings-->Privacy Policy and the actual link and that did the trick.

    Regards,

    Mark


    Why not give a little to charity?

    Friday, September 21, 2012 8:28 AM
  • I have now been bounced four times for collecting private data without a privacy policy.

    Except I don't collect any private data, and I do have a privacy policy.

    They're doing this because of weak supervision and a system that manufactures immunity.  John arranged a requirement that they respond with PDFs explaining the rejection; they're just ignoring it.

    Saturday, September 29, 2012 12:48 AM
  • John, do you have a privacy policy in both the metadata _and_ in the Settings charm in your app?

    The following may help: http://msdn.microsoft.com/en-us/library/windows/apps/Hh921583.aspx#acr_4_1

    Saturday, September 29, 2012 4:16 AM
  • i'd also add to that, if you follow the policy to the letter (and you can bet your bottom dollar they will), you also have to have a link to your privacy policy in any about page.

    what is unclear is if it HAS to be a web link or whether thats just an advisory as all the places you could add it (provided by the store) are http links only.


    Darkside Admin and blogger on #XNAUK - http://xna-uk.net Master Chief of @Zenith_Moon studios

    Sunday, September 30, 2012 12:08 AM
  • John, do you have a privacy policy in both the metadata _and_ in the Settings charm in your app?

    The following may help: http://msdn.microsoft.com/en-us/library/windows/apps/Hh921583.aspx#acr_4_1

    No.  I didn't have it in either place, because neither the requirements nor the checklist mention that. 

    Also, when your staff have been rejecting me, they have failed to tell me why.

    This is the first I've heard of it.  Your staff's unwillingness to be complete, and your documentation's incompleteness, has cost me more than a month, with less than three weeks to go.

    Which is why I've been trying, for more than a month, to get someone at Microsoft to address that the review team is the only chance that developers have to find out *why* the bounce is happening.

    This thing where you guys go "did you try X" and then don't fix the system?  That saves one person, but leaves EVERYONE ELSE STRANDED.

    FIX THE SYSTEM.

    All you have to do to genuinely fix the problem is convince reviewers to give a one-sentence explanation of what's actually wrong, then make your documentation on completion actually mention the complete list of steps for completion.

    Nothing about charms or metadata: http://msdn.microsoft.com/en-us/library/windows/apps/hh694062.aspx

    Seriously, this problem is happening because every time we go "here's where you were supposed to tell us that," you try to fix it with one-time answers for one person in a forum.

    Start addressing this correctly, by fixing the broken documentation and teaching the reviewers that they need to do their entire job.

    Three weeks to go, and you haven't even tried to fix the real problem.  What a train wreck.

    Sunday, September 30, 2012 1:05 PM
  • I have run into same issue - failed for 1.2 without any explanation.

    My app does not require a user account. There are no in-app purchases. There are no non-functional sections. It does work on all architectures. The screenshots are from the app. There is no play/pause buttons. The description matches exactly what the app does.

    So now instead of working on my other apps, I am trying to figure out whats in the tester's mind when they failed the app. Not the best use of my time :-(

    Is there a way to communicate with the test team? Why is there no pdf file with this report - as with any other reports.

    very frustrated.....

     
    • Proposed as answer by Steve Dao Thursday, April 4, 2013 2:19 PM
    • Unproposed as answer by Steve Dao Thursday, April 4, 2013 2:19 PM
    Sunday, September 30, 2012 1:24 PM
  • i'd also add to that, if you follow the policy to the letter (and you can bet your bottom dollar they will), you also have to have a link to your privacy policy in any about page.

    what is unclear is if it HAS to be a web link or whether thats just an advisory as all the places you could add it (provided by the store) are http links only.


    Darkside Admin and blogger on #XNAUK - http://xna-uk.net Master Chief of @Zenith_Moon studios

    Where is this policy please?  I'd love to know where I could have read that rule.

    Edit: Voted helpful

    Sunday, September 30, 2012 2:08 PM
  • John, do you have a privacy policy in both the metadata _and_ in the Settings charm in your app?

    The following may help: http://msdn.microsoft.com/en-us/library/windows/apps/Hh921583.aspx#acr_4_1

    That link is *extremely* helpful.

    Can it be placed in the dashboard next to a rejection?  I've never seen that before, and it answers a ton of questions.  It needs to be more visible.

    Thank you, sir.

    Sunday, September 30, 2012 3:10 PM
  • That link is helpful, yes. But nevertheless the testers MUST specify why the app failed the cert process.

    I mean there are currently apps in the store that adhere to no UX guidelines and even have the standard SDK star app icon. How could that happen?

    We try everything to adhere to the MS guidelines and MS is doing nothing to help us through the cert process.

    This needs to change quickly. This is not the first App Store ever developed so the process should be pretty straight forward.

    Sunday, September 30, 2012 8:41 PM
  • That link is helpful, yes. But nevertheless the testers MUST specify why the app failed the cert process.

    I mean there are currently apps in the store that adhere to no UX guidelines and even have the standard SDK star app icon. How could that happen?

    We try everything to adhere to the MS guidelines and MS is doing nothing to help us through the cert process.

    This needs to change quickly. This is not the first App Store ever developed so the process should be pretty straight forward.


    Yeah, they're actually required to provide an explanatory PDF; they just don't do it, and there's no way to flag them for failing at their job, so nothing's improving.
    Monday, October 1, 2012 2:24 PM
  • Additionally it should be required for them so "sign" their report so that a tester that constantly lets through incomplete apps (with the default SDK icon) has to resign from his position. I mean MSFT demands highest quality from us and we try to comply (in our own interest in caring for the user), yet MSFT fails to send back quality reports to us.
    Monday, October 1, 2012 2:48 PM
  • Now I will defend the team a bit here through my current certification nightmare, things have improved slightly over the past few weeks.

    There is a link on the cert failure report next to each failure pointing to the full policy descriptions (although the policies still need more detail and help/guidance plus the link should jump to the specific cert)

    They do now include a pdf report in a sip, depending on the cert failure you may get multiple reports, 1 per section ( although there is limited detail and a hand written paragraph would go a long ways better than an autogenerated report)

    Granted there is still a long way to go to improve the entire process and they could learn some tricks from the phone team (escape the silo guys)

    But! They are working hard with incremental improvements coming which are very appreciated, with the looming launch of the store and the more apps racing to the finish this does need more sooner


    Darkside Admin and blogger on #XNAUK - http://xna-uk.net Master Chief of @Zenith_Moon studios

    • Proposed as answer by mark_1h Monday, October 1, 2012 10:04 PM
    • Marked as answer by Roberts_EModerator Tuesday, April 30, 2013 9:21 PM
    Monday, October 1, 2012 4:55 PM
  • I just got my PDF report.  It has one line of text:

    Failed one or more Automated tests and/or Capabilities requirements.

    What's worse is that the app passes WACK locally and it passed the WACK on my first submission.  This was my third.  Only change was that I added a privacy policy.  Now I look like an idiot to non-technical people on the project because they assume i'm doing something wrong to cause the app to fail.

    Simply asking for a PDF or one sentence of explanation is not enough because you get a bad explanation like that.

    Wednesday, October 10, 2012 4:42 PM
  • I mean there are currently apps in the store that adhere to no UX guidelines and even have the standard SDK star app icon. How could that happen?

    You won't believe how much that has been irking me.

    Matter of fact, lets just rant a bit more: I've submitted a tool that records from your mike, line-in or whatever you have set up and renders either a FFT or does a sound pressure level calculation. It got failed because the utility was unclear (which is also point 1.2). You'd think that a software tester would be knowledgeable enough to figure what such a tool would be good for. But no, it got rejected. I resubmitted it and wrote half a novella in the tester notes to list use cases. Can't await the certification results, the 7 days are due this evening.

    But meanwhile, there's fart apps.


    • Edited by Tom Servo Thursday, October 11, 2012 1:32 PM
    Thursday, October 11, 2012 1:25 PM
  • This whole "control over what is published" is also driving me insane. You should be able to run everything on your computer, whether that are fart apps or the newest Playboy app. This whole lock-in approach is the wrong direction.

    Side-by-side installations will be the norm for b2b apps anyway and soon "private" customers will be able to download properly signed apps for side-by-side install from regular websites. Just because some tester in Bangalore (or wherever they sit) does not understand a certain program is reason enough to reject it? He has to hand it to people who actually understand it or use google. Beside that, for the same reason they could say: "a stop watch app?" "We already have one. Try again". Its just not for the testers to decide what might be useful for others. If for that, who needs a BMW promotion app, whose only purpose is to, well, promote BMW news and videos. Who needs that, except BMW salesman? One could say... but of course I say let the people install crap apps. Its their decision, not MSFT. We do not need a nanny on our PCs.

    The whole store concept is one giant step away from free customers, and at the same time free developers. But most people seem to accept that sad fact. 

    That aside: I never understood the whole notion of "clean" app stores. There nudity is forbidden but games where you can hack off limbs are allowed. Go figure!

    Thursday, October 11, 2012 2:18 PM
  • I also have the same issue, failed for point 1.2, they said I didn't provide a valid user account for testing, But I DID provide a testing account for them and clearly mentioned the details in the test instruction, they just ignore it and fail my app. I resubmit with more clear instruction and they ignore it again, that said my app is failed again.

    What's wrong with these testers?

    Saturday, October 13, 2012 1:58 AM
  • Having worked on many commercial projects I read these comments in disbelief that tester would report the equivalent of 'it does not work'. I would hope Microsoft is hiring testers with commercial experience otherwise they will annoy developers very quickly, especially since it takes a week to attempt certification again.

    In regard to 4.1 they really need to lead by example as the Microsoft Photo app states in Permissions that it will use your internet connect but provides no privacy policy in plain view it is hidden in About and other default apps will wait for ever before giving user feedback.

    Windows 8 needs positive feedback and good quality apps to gain popularity and if us developers who are used to working with testers get these nonsense responses we would reconsider it as a serious platform and go back to desktop.


    Brett Styles

    Saturday, October 13, 2012 9:34 AM
  • I continue to get the "Requirement 1.2: Your app must be fully functional when the customer gets it from the Windows Store".

    The tester provides one line of text: "App did not complete the expected action."

    All the tester has to do is paste a URL into a textbox and click a button. They continually show me a TextBox with the text pasted in it with the above fail reason. Is it seriously possible they need me to spell out that they need to click the button?!

    UPDATE: Even explicitly telling them to press the button doesn't work.

    I reached out to support on this and Alex, from Ireland sideloaded my app and confirmed it worked as described and that he would escalate the issue to the content compliance supervisor. However, one week has passed and the app has failed 3 times already! This is the 18th submission in 3 months. I just don't know what else to do....but cry.

    Wednesday, May 8, 2013 4:31 AM