locked
Changes to the search results RRS feed

  • General discussion

  • We have been rolling out upgrades to the Windows Phone Store infrastructure over the last several months, including changes to our search engine and our list algorithms. 

    As part of this deployment we changed over to the Bing search engine and have modified our criteria to be more flexible, to return apps that take into account user mis-spellings and to display apps that are similar or within the same genre.  The objective in making these changes is to aid users in discovering their favorite apps in the growing catalog. 

    This new criteria not only utilizes the app name like our previous search engine, but now also takes into consideration app metadata, downloads, and ratings with the intent to return a list of high quality apps that are relevant to the query.

    We have begun working to fine-tune the algorithm now that the new service has rolled out to most users across the globe.  One area we are examining for relevancy is app keywords.  We are finding that some developers are using phrases or popular search terms as keywords.  We believe there is room for improvement in the way our algorithms use this app metadata to determine relevance of the search results.  We are testing a potential fix for this issue now that will address some of these shortcomings and look to have it in production in a few days’ time. And, we will continue to make improvements to ensure we promote high quality apps.

    Please continue to provide feedback, we’re listening.

    Tuesday, October 23, 2012 5:13 AM

All replies

  • 1. Substring matching (or even singular form matching) for app title is no longer happening with the new algorithm.

    For e.g. earlier if a user search for "emojicon" then it used to return my app "emojicons" in the result. This is no longer happening.

    http://www.windowsphone.com/en-us/store/search?q=emojicon

    http://www.windowsphone.com/en-us/store/search?q=emojicons

    2. Split word search is not matching with title.

    For e.g. earlier if user searched for "my text twister" then it used to return "mytexttwister" in search results. This is also not happening anymore.

    http://www.windowsphone.com/en-us/store/search?q=my+text+twister

    http://www.windowsphone.com/en-us/store/search?q=mytexttwister

    3. Substring matching (or even singular form matching) is no longer happening for keywords.

    For e.g. Searching for "emoticon" used to result in my app "emojicons" as I have "emoticons" in its keywords. This is no longer happening.

    http://www.windowsphone.com/en-us/store/search?q=emoticon

    http://www.windowsphone.com/en-us/store/search?q=emoticons

    4. Low rated apps are being giving too much priority just because of exact app title match due to which searching for apps with keywords is not giving high quality results. One has to scroll down a lot to find good apps.

    For e.g "Emojicons" (a top 100 app with 4.5+ rating) is listed way down when searching for "emoticons"

    http://www.windowsphone.com/en-us/store/search?q=emoticons

    IMO, the earlier algorithm was much better in finding good apps as it used to give priority to rating/downloads etc.

    Also, I am pretty disappointed with the new search algorithm because of the points I mentioned above. It is pretty normal for user to search for "emojicon" instead of "emojicons" or "my text twister" instead of "mytexttwister", but they are no longer finding my apps. My app downloads have gone significantly down after this new search algo has come into place. I don't see any reason why the new search algo doesnot want to match singular word with their plural forms while it want to correct mis-spelled words.

    Kindly consider the issues I have mentioned above.


    [url]http://www.kunruchcreations.com[/url]


    Tuesday, October 23, 2012 6:12 AM
  • Totally agree with KanishkKunal
    Tuesday, October 23, 2012 6:59 AM
  • I have updated the keywords in my apps to include the singular form and the split form of words too. But I really hope that the search algorithm could have handled it also I hope that the search algo was fine tuned more towards finding good quality apps, so that developers can spend time in writing good apps instead of spending time to reverse engineering the search algo to make their app appear higher in search result.


    [url]http://www.kunruchcreations.com[/url]

    Tuesday, October 23, 2012 11:20 AM
  • Hi Tamara,

    Thank you for your thoughtful response to this issue. I look forward to the fix you mentioned and will continue to provide feedback when this is rolled out. I do have some further thoughts on this issue that I will address below.

    Firstly, I just want to get this out of the way before concentrating on more important matters. I remember reading the blog post you mentioned above about improvements to the algorithm, but this was very brief and quite a few weeks before the algorithm was rolled out. Since the new algorithm presented a significant change to developers I believe another blog post devoted to this would have been (and still would be) worthwhile. More communication can only be a good thing! Also, the quality of the algorithm rolled out at the start of October was very poor and I was actually quite surprised by this. As I've stated this significantly effects developers, some of whom are trying to build a livelihood out of their app business, and I believe more internal testing should have been implemented before this roll out.

    Back to the algorithm and the other changes to the searches. I believe that, aside from the issue of favouring the app name, these changes are good improvements to the Store. The top free and top paid lists in the apps section no longer returns games, the new best-rated list is a great addition and returns only those apps with a statistically relevant number of ratings (the Windows 8 Store doesn't do this yet, although that's probably because it is at such an early stage), the revised new+rising list is a major improvement. The ability of the new algorithm to cope with spelling mistakes is a good idea and well implemented. If the issue of favouring the app name could be fixed then all these changes will result in a fantastic Store.

    Interestingly, in the original blog post Mazhar gave an example of returning relevant results: 'type “guitar” and you’ll see relevant apps even if they don’t have “guitar” in the title'. However, a search for "guitar" on the phone returns 50 apps that all have "guitar" in the title! This is another example of the issue we've been describing.

    In your post above you state how some developers are using popular search terms as keywords. Surely the best way to tackle this is to be more strict with keywords during the certification process, and to review all apps already in the Store and email developers that are abusing the system? Employing machine learning so that the algorithm itself can determine the relevant apps is both fascinating and welcome, but this is a long term measure and must have a sound basis to build on. (I'm sure the Windows Phone team are banging their heads as they read this saying that they already know and do this, but I just wanted to re-emphasise the point).

    Finally, I just wanted to add that more detailed instructions on what sort of keywords developers should use would be a very beneficial addition to Dev Center. Kanish has described substring matching for keywords, which used to work but now doesn't. A guide with examples would help developers tremendously, and as Kanish states would mean that they don't have to spend time reverse engineering the search algorithm to optimise their apps' visibility. On this point another good idea could be to provide detailed statistics in Dev Center on the downloads/views of apps resulting from different searches.

    Thanks again for your reply and I hope this feedback is useful. Please continue to be involved in this thread over the coming weeks as we further evaluate the new search.

    Kind regards,

    Richard Walters.

    Tuesday, October 23, 2012 1:22 PM
  • Last night, I noticed that a couple of my titles weren't visible at all, (http://social.msdn.microsoft.com/Forums/en-US/wpdevelop/thread/1a758a16-b169-4dbf-ba10-2725993603d8) searching either for their name or for our publisher name Wieser, 8 of the apps showed up but the two best sellers didn't.

    I'm also a little concerned that our app Skied Demon doesn't show up when searching for ski, yet it's keywords are these:

    Ski Logger, Ski Tracks, Ski Speedometer, Skiing,
    Snowboarding


    Anthony Wieser | Wieser Software Ltd | www.wieser-software.com

    Wednesday, October 24, 2012 8:24 PM
  • Yeah Anthony they've been pretty bad with keeping search results stable lately.

    The official Twitter app also disappeared from search results for a day.

    They'll probably pop right back up within a day or two. It's hilarious how little Microsoft has tested their software.

    Wednesday, October 24, 2012 8:26 PM
  • Hello Tamara,

    The marketplace search is very frustrating on the phone while the marketplace search on the Windows Phone web site is great (http://www.windowsphone.com/en-us/store/featured-apps).  

    Why can't the both work as well?

    I just released a coin collection management app call "Coinia" on 10/18/2012. It is almost impossible to find it via the marketplace search on the phone. My keywords are:

    Coin,Collecting,Collection,Gold,Silver

    Searching on "Coin" does not find my app, but instead finds a ton of free apps that just flip an image of a coin.  I wish you would bring back the "Paid" apps category so that the user would not have to wade through all the free crap apps.  I use to browse the "paid" category because I knew, for the most part, if people were charging for an app it most likely had some good value/functionality. Please add "paid" to the current list of "featured", "top", "free", and "new".

    I just tried a search today, 10/25/2012, with the other key words and could not find the "Coinia" listing.  Two days ago when I searched all the keywords my app was listed for the keyword "Silver" only.  Now today, "Silver" does not work.

    On the Windows Phone web site the search always works.  That's great, but most people will be using the phone for their searches.

    I can search and find my app on the phone using "Coinia", the complete title.  That's great but only if you know the exact name.  Search is more useful when you don't know the name of the app with the features you want.

    With more testing I discovered that my keywords will work if I enter "Coin collecting".  Is the phone search only exposing a limited set of apps?  If so, and you can't deliver all the entries to to storage or performance limitations, can you at least add a "More" button so the user can continue to receive more matches?

    I will continue to test the search for the next week.
    • Edited by Takimchu Thursday, October 25, 2012 12:31 PM
    Thursday, October 25, 2012 12:30 PM
  • Last night, I noticed that a couple of my titles weren't visible at all, (http://social.msdn.microsoft.com/Forums/en-US/wpdevelop/thread/1a758a16-b169-4dbf-ba10-2725993603d8) searching either for their name or for our publisher name Wieser, 8 of the apps showed up but the two best sellers didn't.

    I'm also a little concerned that our app Skied Demon doesn't show up when searching for ski, yet it's keywords are these:

    Ski Logger, Ski Tracks, Ski Speedometer, Skiing,
    Snowboarding


    Anthony Wieser | Wieser Software Ltd | www.wieser-software.com

    This doesn't appear to have changed.  Substring matching appears not to work.  As of yesterday, our app did not appear for the search "Ski".

    So, it's a new set of keywords now.  Pity it takes so long to do the update, and that we're not being discovered due to this.


    Anthony Wieser | Wieser Software Ltd | www.wieser-software.com

    Sunday, November 11, 2012 7:30 AM
  • Such a shame that we get to this scenario where the searching changes after 1 year of my app being published.  Now one of my apps is no longer returned when searching for "notebox" even though it is called "noteboxpro".

    Not only this, but my apps dropped in the charts overnight from within the top 10 to around 50/60 place, and MS seem to think that is normal.  Now a few weeks later my apps are climbing back up the charts, but that just adds (in my opinion) to the evidence that there was some sort of re-index during the search "enhancement"!

    I may sound annoyed, and I am, mostly because I have spent so much time and effort building the apps and advertising (sometimes at cost via BSA), only for the apps to drop down the charts or nolonger return in a search.

    There is now little point in naming your FREE and PAID versions of your apps similarly because they will not appear in the same search!


    Paul Marques - Lead developer @ Bytemarq - http://www.bytemarq.com


    • Edited by bytemarq Monday, November 26, 2012 10:12 AM
    Monday, November 26, 2012 10:06 AM
  • Yep. They haven't fixed a thing.

    SEARCH IS STILL AWFUL.

    Looking for "Twitter" still places a number of crappy non-popular results before Rowi.

    I had to change the name of my homework planner to Power Planner (Homework Planner) with a bunch of spaces between the parenthesis such that they don't show up in the app list, but search now places my app first for "homework" as it used to, just cause I now have the word "homework" in it.

    It's obvious the people who wrote the search algorithm don't even use Windows Phone :/

    Monday, November 26, 2012 4:05 PM
  • Producing search results based on downloads or rating alone is a bad system. It is not clear if this has changed much but it was previously communicated that results are ordered by number of downloads.

    Now, consider new apps in the marketplace (and everybody's are new at some point). Their downloads are of course 0 or low. They therefore do not appear in results at all on the phone because they are throttled. And on the web they are also not likely to either, in a saturated area such as news or weather and so on.

    So they are not discovered by users looking for new apps in an area - instead the user sees the same old apps at the top of the list time after time. How do the new apps get discovered? Only by brute force category browsing, which are not the primary means of finding apps of course. But a user wanting to see a variety would have to resort to that kind of exhaustive enumeration. It does not matter if the new app is good or bad - the user will not find it!

    This type of ordering is beneficial to neither the developer nor user.

    It is more fair to the developer, and gives a more varied picture to the end user, if app searches could have some element of randomness (other than the search keyword itself) so that new apps with low downloads have a chance to be seen and tried out, allowing their downloads to actually increase because of discoverability afforded by the search result.

     

    Friday, November 30, 2012 8:18 PM
  • I've been following this thread and keeping an eye on searches in the Store, and thought I would reply again with my comments.

    Firstly, the searches have improved greatly since the changes were first rolled out, so my thanks to the Windows Phone team for the improvements and for listening to the feedback. My app now appears within the top 8 app under searches for each of its keywords in the US Store, showing that the issue with favouring the app name is much less severe than a few weeks ago. I will be changing the app name back to Calculator² from Calculator in the coming days and will report back here with the results.

    Unfortunately there is still some weight associated to the app name for ranking the search results, and I still believe the Store searches would be best if the app name and the keywords had equal weight. As Andrew pointed out with the twitter example, quality apps such as Rowi still appear below other apps with twitter in the name even though Rowi has a significantly better rating and number of reviews, and presumably a significantly greater number of recent downloads. My app in the UK Store appears much further down the search lists than in the US because it has fewer downloads in the UK (in a relative comparison), and therefore the weight associated with number of recent downloads, ratings, etc. has not been enough to negate the weight of the app name for other apps of lower quality.

    I would like to see further improvements and would be grateful for more communication from Microsoft on this issue. Details of future upgrades to the algorithms on this thread would be greatly appreciated. However, at this point I'm actually more concerned with the search algorithms in the Windows Store, which are more like those of the Windows Phone Store immediately after the switch to being powered by Bing. I will be posting in the Windows Dev Center forums on this issue, but I thought I'd mention this here in case the Windows Phone team can work with the Windows Store team on this.

    Monday, December 3, 2012 10:56 PM
  • @deanmccrae

    Dean, I agree that ranking by total downloads is not the right approach for the reasons you suggest. The new algorithms rank by recent downloads, recent ratings, etc. so that apps that have been in the Store for a while don't have an unfair advantage and new apps will have a chance to rank highly. The new apps section and a small amount of marketing by the developer should allow enough discoverability to give new apps this chance. Quality apps will always have good recent downloads and should climb/remain high on the lists.

    I don't think that the rankings should be random at all. Users want to have some idea of the order of quality in the search results so that they don't have to use brute force to find the best apps. This is the problem with app name being favoured -- I'm sure users are having to look at far more Store pages than necessary to find the quality apps because low quality apps with matching names appear too high in the results.

    Monday, December 3, 2012 11:19 PM
  • I second what Richard said. We really need a response from Microsoft about these issues.

    It looks like Microsoft hasn't really changed much in terms of their search algorithm.

    It's simply ridiculous that Rowi, with 1,824 ratings and a 4.5 star average is placed below something called "Twitter Stalker", which only has 11 ratings and a 3.5 star average. There is no way that Rowi is downloaded less than that other app.

    Twitter search results:

    • 9th and 10th place - Rowi
    • 13th place - Seesmic
    • 17th place - glEEk

    All of those should be placed 2nd, 3rd, and 4th behind the official Twitter app, but they're not, simply because they don't have Twitter in their name. It's a bad algorithm, and is definitely giving people the impression that apps on Windows Phone are non-existent and crappy, since they can't find the good apps.

    Are we all supposed to rename our apps Rowi (Twitter)? That's what I had to do with my homework planner. It's now titled Power Planner (Homework Planner), and it now rightly shows up first when searching for "homework", just because I changed the title.

    The weight that you put on the app's name is ridiculous. It's a super easy change, why can't this be fixed?

    Monday, December 3, 2012 11:32 PM
  • I've posted in the Windows Dev Center forums about the same issue in the Windows Store:

    http://social.msdn.microsoft.com/Forums/en-US/windowsstore/thread/6b745c02-d5ea-4761-8ec9-c2b22d888422

    Tuesday, December 4, 2012 1:08 AM
  • Ok, so last week I updated my app and changed the name back to Calculator² from Calculator. The results of this change on the US search rankings were as follows (listed is each search term, the old ranking, the new ranking):

    Calculator: 1st to 58th

    Scientific Calculator: 5th to 9th

    Currency Converter: 8th to 8th (unchanged)

    Unit Converter: 4th to 4th (unchanged)

    There are two features to extract from the above. Firstly, the app is ranked highly under Scientific Calculator, Currency Converter and Unit Converter. This shows the improvements the Windows Phone team have made to the algorithm since October. My app is quite popular and this has improved its search ranking even if the search term doesn't match the app name.

    Secondly, the search rankings changed for search terms which previously matched part of the app name, i.e. Calculator and Scientific Calculator. This shows that the search rankings are still weighted by the app name. A change to the app name should not have any effect on the search rankings at all if the keywords include these search terms. Obviously when my app was called just Calculator it took advantage of this flaw and was ranked 1st, perhaps higher than it should have been. However, 1st to 58th is a significant drop and now it is the other way around -- my app is suffering because of this flaw. Also, the drop has been more significant in other regions where the app isn't quite as successful as it has been in the US.

    I'm still more concerned about this issue in the Windows Store, but no-one has replied to my thread on the Windows Dev Center forums. On Windows 8, Calculator² is the 28th top free app in the entire UK Store (it's 11th in the US Store), yet it ranks 29th under a search for Unit Converter. All of the top 25 apps under this search have either Unit or Converter in the title, yet most have no ratings and likely not many downloads. The flaw is so obvious for this search, and a host of others, it is quite striking. The issue is only going to make quality apps harder to find as the Windows 8 Store grows.

    Please can this be fixed asap? I hope that we have provided enough feedback on here that to show that weighting the search results by app name is a flawed approach. I love developing for Windows and Windows Phone, but at the moment I am frustrated that I am not receiving the rewards I should be because the Stores are not optimised for users to find the quality apps.

    Tuesday, December 18, 2012 8:19 PM
  • Richard.Walters

    Same happened to me.  My #Notebox went from top 10 in UK and US (Business FREE category) to around 60th!

    Thankfully it is now creeping back up, but WTF.

    There is also NO consistency in the searching.  For example, previously a search for "notebox" would return my free version #Notebox as well as the NoteboxPro version.  Post the MS algo "improvement" the search only returns the FREE version.  Not only that, but do the very same search on your phone and you get both apps!

    Honestly, where do MS get their testers from?


    Paul Marques - Lead developer @ Bytemarq - http://www.bytemarq.com

    Tuesday, December 18, 2012 8:33 PM
  • Hi Tamara. Has any progress been made in improving the search algorithms in the Windows and Windows Phone Stores? As far as I can tell there have been no changes since the improvements last November, and with the Stores growing rapidly this issue is becoming more and more severe.

    I recently ran a free promotional period for the paid version of my calculator app on Windows Phone, and WPCentral were kind enough to post an article about this. The end result is that the app is currently the Top Paid app in the US Windows Phone Store. Yet, if I search for 'calculator', which is one on the app's keywords, the app appears 62nd in the results. This is very contradictory and exemplifies the flaw in the search algorithm. The situation in the Windows Store is the same; my app is in the top 100 free apps in the entire Store, above all other calculators, yet it appears outside the top 30 results under searches for 'unit converter', 'currency converter', etc.

    The fix seems straightforward in principle: to make the ranking of the search results independent of matching the app name. I believe this would greatly improve the percieved quality of the Stores. Whether this fix is difficult to implement technically I couldn't say, but a solution need to be found asap.

    Sunday, February 24, 2013 7:05 AM
  • Bump
    Saturday, March 2, 2013 8:28 AM
  • I posted about the issue with our app Skied Demon not appearing in the results searching for ski back in November.

    Since then we've done an update, and the keywords now contain the word ski, as does the text description.

    And yet we're not visible.

    So, we've pretty much missed the ski season in the northern hemisphere again.  Why should a developer support this platform?  Must be from the great support we get from MS.


    Anthony Wieser | Wieser Software Ltd | www.wieser-software.com

    Sunday, March 10, 2013 4:59 PM