locked
Changes to the search results favour the app name too highly RRS feed

  • Question

  • I've noticed yesterday that the search results returned using the phone's marketplace app have changed. I urgently want to bring this to Microsoft's attention, since I believe the new algorithm for searches is flawed and bad for Windows Phone at a very crucial time. Please note that I have only checked this in the UK, and that the web marketplace is unchanged. I'm also assuming this is a planned change and not a bug.

    Search results now return all the apps with the same name as the search term first, and then other apps that use this keyword next. For example, a search for 'twitter' returns 15 apps called 'twitter', and then other apps such as rowi and gleek next. My concern is with searches for 'calculator', 'scientific calculator', 'currency converter', 'unit converter', since my app, Calculator², is multi-purpose and features all of these. Previously, my app appeared within the top 10 for any searches using these search terms. With the new algorithm, because there are so many apps called 'calculator', 'scientific calculator', etc. it doesn't appear under any of these searches. A search for 'calculator' returns more than 50 apps called 'calculator', most of which have no ratings and likely not very many downloads. The quality calculator apps such as mine, Calc Pro and MathCalculator cannot be found using the phone marketplace. I expect that downloads for my app, which currently stand at 350-400 per day (it has an average 4.5 rating from 150+ reviews), are about to drop to zero. Obviously, this is very disappointing given the hard work I have put into buidling and promoting the app (also now on Windows 8), and I expect many developers will be equally disappointed and angry. In addition, the best rated and most downloaded apps are much harder to find, and consequently the quality of the app selection for consumers is drastically diminshed. Currently, the only solution I see is to rename my as as just 'Calculator' so that it appears high on a search using this term (I actually believe it would appear at the top). However, this wouldn't help with the other keywords that the app users.

    I think it is good that Microsoft are looking into ways for improving the results for marketplace searches, especially with the large number of apps now available. However, please can the algorithm be switched back to old one until a more suitable solution is found? I would be happy to provide further feedback and suggestions using my experience as a developer to help with this.

    To summarise, the new approach for returning search results is flawed and places too much emphasis on the app name, rendering the keywords almost useless for apps that require common keywords (such as 'calculator'). Ironically, this approach encourages developers to rename their apps as their most used keyword, resulting in lots of apps with the same name, a situation Microsoft has prevented with Windows 8. Moreover, the current approach is unfair on developers and doesn't provide the best selection for consumers.

    Kind regards,

    Richard Walters.

    Tuesday, October 9, 2012 5:32 PM

Answers

  • We have been rolling out upgrades to the Windows Phone Store infrastructure over the last several months, including changes to oursearch engine and our list algorithms. 

    As part of this deployment we changed over to the Bing search engine and have modified our criteria to be more flexible, to return apps that take into account user mis-spellings and to display apps that are similar or within the same genre.  The objective in making these changes is to aid users in discovering their favorite apps in the growing catalog.

    This new criteria not only utilizes the app name like our previous search engine, but now also takes into consideration app metadata, downloads, and ratings with the intent to return a list of high quality apps that are relevant to the query.

    We have begun working to fine-tune the algorithm now that the new service has rolled out to most users across the globe. One area we are examining for relevancy is app keywords.  We are finding that some developers are using phrases or popular search terms as keywords. We believe there is room for improvement in the way our algorithms use this app metadata to determine relevance of the search results. We are testing a potential fix for this issue now that will address some of these shortcomings and look to have it in production in a few days’ time. And, we will continue to make improvements to ensure we promote high quality apps.

    Please continue to provide feedback, we’re listening.

    Tuesday, October 23, 2012 5:49 AM

All replies

  • I've noticed the use of a new search algorithm in the US marketplace as well. I agree with Richard in that the old method was preferred, at least in the particular situation of my app/game too. :)

    I've made a game that emulates a popular 3d puzzle cube simulation. Only the official app should use the name "Rubik", and so I allege I wisely used "rubik" simply as a keyword for my game having an original name "Cuber". This allowed users to discover my game without knowing the name.

    My app was also previously highly ranked based on user reviews. However, now no longer appears within the top 5 as it used to. It is now ranked 14th, well below many other apps having a lesser rating simply because they rather inappropriately use a trademarked name in their app title. So that's kind of a bummer.
     
    I suppose technically the search is more accurate when only considering the app title. My game will no longer be so easily discovered and prominently featured when searching for the appropriately related keywords. (So the official game should now be much better off, because my game was a lot cooler. :)

    Thanks for listening!

    -Taber
    Wednesday, October 10, 2012 6:18 AM
  • Please can someone from the Microsoft Support team reply on this thread about the above issue? I believe the search results look very amateurish and a rethink is needed here. For example, consider a search for 'currency converter', http://www.windowsphone.com/en-gb/store/search?q=currency+converter (the web Windows Phone Store is also now using the new algorithm). Do I really want to find all the apps called 'Currency Converter' first, or do I want the best apps that are currency converters? My app is in the top 10 most downloaded for 'currency converter', but now because of the name it sits at 66 and therefore isn't even visible on the phone. Ditto other searches, such as 'scientific calculator' and 'unit converter'. Unfortunately I'm about to submit an update to change the name to just 'Calculator' so that the app at least appears under this search. Is this really what Microsoft wants developers to be doing?

    I just cannot believe this got through testing, there are so many better ways of presenting search results to the user. Further, as far as I'm aware this change, which will have a significant impact on the downloads for most apps, has not been announced at all by Microsoft, therefore we as developers cannot understand the reasons for using the new algorithm. I think this is very poor on Microsoft's part, particularly at a time when Microsoft are trying to encourage developers to contribute to their platforms.

    Richard Walters.

    Tuesday, October 16, 2012 9:27 AM
  • Truly agree. New search algorithm is a bullshit. Total number of downloads of my apps decreased upto 10 times because of new algorithm.
    Tuesday, October 16, 2012 12:31 PM
  • Agreed. Their new algorithm strongly discourages the use of unique names. More samples include:

    Baconit: Used to be #1 spot when searching for "Reddit" (Baconit is a Reddit viewer). Now, it's 4th, and ahead of it are apps that have zero reviews or 3 terrible reviews. Baconit has 4,900 reviews and a 4.5 star average, and is the 400th most popular app on WP. It definitely deserves to be at spot one (like it used to be).

    Power Planner: Used to be #1 spot for "homework", now it's 6th (my free version somehow stayed at spot #3).

    Rowi: Doesn't even show up when searching for "Twitter".

    Weather: Why the heck is "Weather+" the first result??? Over "Weather" by The Weather Channel??

    Apparently their search algorithm always places anything with a "+" first. LOL. Such a fail.

    Tuesday, October 16, 2012 9:37 PM
  • Totally BS! So this is the reason why my app downloads were crashing for the past few days despite heavy advertising on Adduplex network.

    Oh please revert to the old search algorithm.


    • Edited by LastBattle Wednesday, October 17, 2012 1:40 AM
    Wednesday, October 17, 2012 1:30 AM
  • To show how horrible this is, I did a simple search on the query 'Twitter'.

    As you can see, the first page is filled with junk app except for the official version.

    Good Twitter client like Rowi, Carbon, Gleek are clearly missing unless you scroll all the way down. This is discouraging..

    Wednesday, October 17, 2012 10:36 AM
  • Something changed recently, because "reddit" now brings up Baconit first again... "weather" seems to be fixed too.

    But searching for "homework" still brings up the crappy results. Same as "twitter", that's still crappy.

    I can't believe Microsoft hasn't replied yet! This is a joke. How can developers care about writing for WP when Microsoft doesn't even care about them!!

    Friday, October 19, 2012 4:15 PM
  • Update: my app is now called Calculator instead of Calculator² (and correspondingly in the other supported languages). The app is once again visible near the top of searches for 'calculator' and 'scientific calculator' (interestingly, top in some regions but not in others, I'm guessing number of downloads still has some relevance), however, it still cannot be found on the phone under searches for 'currency converter' and 'unit converter'. I really hope this name change will be temporary as I feel the app has lost its unique identity, and moreover is now different to the Windows 8 version, which remains as Calculator².

    Please can we have a statement from Microsoft as to why these changes have been brought about? Is this in preparation for Windows Phone 8 and will there be further changes, e.g. new filters for search results? Previously I could work out how the searches returned results and knew what I had to do to make my app a success. Now I'm frustrated by the fact that other apps, which have fewer downloads and ratings and in general are lower in quality, are easier to find than mine under the same relevant searches, and there is nothing more I can do about this.

    Sunday, October 21, 2012 11:17 AM
  • Ha, it didn't take long for a review like this to appear. Fortunately they still like the app and gave it a 4/5 rating.

    "Good software, but the name change is confusing, as it has the same name as the system's app.... Maybe change it to something else?"

    Monday, October 22, 2012 11:07 AM
  • Yep I'm going to have to change my app's name just because of this ridiculous algorithm.

    I contacted MS support, and they said the typical "Search results are based on downloads" crap. That's absolutely a lie, since their new search results do NOT reflect downloads in any which way!

    Monday, October 22, 2012 5:20 PM
  • Generally speaking, all 'results', whether you are viewing a single category, all categories, paid apps, free apps, search results, etc. are always ordered by current download frequency ranking. You can love it or hate it, but that's how the ranking system (and search relevance system) works.

    So when you perform a search based on keywords, you are simply viewing all apps in the same order as they would be anywhere else, filtered to show only those that match the keyword. MS does not rearrange these arbitrarily -- the results adhere to the same natural ranking system as when viewing them anyplace else.

    Monday, October 22, 2012 6:04 PM
  • Nate, that used to be true, but the new algorithm is not quite as simple as you describe.

    Previously, a search would return any apps that have either a matching name or matching keyword. The way the results were ordered had no dependence on the name or the keyword, but instead on the total number of downloads (I'm 99% sure of this based on following the progress of my app and others in various regions over the last six months). I believe the ordering of the top apps lists was/is based on the number of recent downloads, say over the past 30 days (I'm not sure of the exact length).

    The new algorithm, as you say, also returns any apps that have either a matching name or matching keyword. The crucial difference is that the ordering of results now depends on whether the search term matches the app's name or the app's keywords, with precedence given to the app name. The results are then ordered by other parameters, such as number of downloads. When I first noticed the new algorithm, all apps with a matching name were presented before those with a matching keyword. For example, a search for 'twitter' returned 14 apps called 'twitter' and then other apps such as Rowi that have the keyword 'twitter'. When I said my app cannot be found under certain searches, it's because it has been pushed so far down by this effect that it doesn't actually appear in the search list, which on the phone is limited to 50 apps.

    As Andrew pointed out, the algorithm changed again on 19 October. I'm guessing that the ordering is now achieved by some sort of points system, where now apps with a lot of downloads, e.g Baconit, receive enough points to move them ahead of other apps with a matching name. Exactly how the points system works is unclear. However, the app name still has too much priority over keywords. I'll explain below what I think Microsoft were trying to achieve, discuss the advantages and disadvantages of this new approach, and discuss why I believe this approach is not right one.

    Firstly, returning search results ordered by total number of downloads is clearly not going to work in the long run. New apps will never receive enough downloads to jump into the top 50 under a given search without a lot of marketing via other means. Therefore, a new way of ordering the apps was required.

    Secondly, when someone is searching for a specific app, say twitter, they want to be presented with that app in the results. Using the old approach, other apps that use 'twitter' as a keyword might appear above the twitter app because they have more downloads. These apps might not be twitter clients at all, but use this keyword because they have something to do with twitter. Thus the search results are cluttered with apps that make finding the actual (official) twitter app much more difficult. One way of alleviating this issue is to give priority to the app name, but this is flawed for the following reasons:

    1. Who is to say that the user was actually searching for the twitter app, but instead apps that have something to do with twitter. In this case returning apps with this keyword before the actual twitter app is not a problem.

    2. Many apps are multifunctional, and I'll use mine as an example. My app is a calculator, a scientific calculator, a programmer calculator, a currency converter, and a unit converter. One of the reasons it is so popular is because it provides all these features in a single app. Previously it appeared in the top 10 in searches for each of these terms. Now because it is not called 'currency converter', and because there are so many apps with 'currency' and/or 'converter' in the title, my 'currency converter' keyword is pretty much irrelevant. The market had already decided that they preferred my app over most of the 'currency converters', but now these apps are presented before mine. Multifunctional apps really suffer with this new algorithm.

    3. The new algorithm is intended to help users find official apps such as twitter more easily. However, these apps are generally so popular that they would be easily visible using any suitable ranking system. Were they hard to find with the old algorithm? Not in my view.

    4. The new algorithm simply encourages developers to rename their apps as common search terms, resulting in lots of apps with the same name. A search for 'twitter' already returns lots of apps called twitter, which makes the official twitter app harder to find. 50 more apps also called twitter isn't going to help.

    What better approaches are there? I believe any/all of the following would improve the Store:

    1. Return to having no dependence on the app name and keywords for the ordering of search results. The current algorithm doesn't solve the problems and only introduces more issues.

    2. Order searches based only on the number of recent downloads, the number of recent reviews, and the rating of these recent reviews. These factors determine the overall quality of the app in an up-to-date fashion. This also gives new apps a chance to succeed if the market likes them (I actually believe the new algorithm does this to some extent, but is hindered by the issue of favouring the app name).

    3. Allow users to filter apps according to number of downloads, rating, etc. This allows users to find apps according to what they believe represents 'quality'. Also, if ordering by rating only include apps that have a certain minimum number of ratings, say ten. This prevents apps with one 5 star rating appearing before those with an average of 4.9 from 100+ reviews.

    4. Encourage developers to use unique names for their apps. The more unique the name, the fewer results are returned when searching for a specific app.

    5. Allow apps with trademarked names, e.g. facebook, to have total precedence under a search for this term. For example, the developer could tick a checkbox in dev center indicating that the name is trademarked (they would then have to provide relevant evidence), and any search term matching the name of such an app would return that app first.

    6. In addition to the above, prevent other apps from using trademarked names in their title. There should not be more than one app called twitter, or facebook, etc.

    7. Also be quite strict with the keywords used by developers. In general I believe the Windows Phone team were doing a good job with this.

    To the Windows Phone team, please, please take this feedback into consideration! And please join this discussion! We developers would much prefer an open and honest debate regarding ways in which Windows Phone can be improved. We all want the same thing!

    Monday, October 22, 2012 9:50 PM
  • Really great response Richard! I would have said all of that if I had the time.

    I really wish someone at Microsoft who actually has control over this would reply. We don't need the generic "Searches are calculated based on an algorithm" reply. We need something to actually be done.

    Tuesday, October 23, 2012 12:16 AM
  • Oh and by the way, the biggest joke of them all is that if you want to be at the top of the search results, just add a + sign after the keyword in your title and BAM you're at the top of everything.

    See "weather" search results for this. His app "Weather+" is now third on the list, despite being far less popular than others.

    Anyone saying that this search system does not need immediate modifications is frankly oblivious.

    Tuesday, October 23, 2012 12:18 AM
  • Numerous developers, including I, are complaining about Microsoft's new search algorithm on the Windows Phone marketplace. I'll summarize various posts by members to document the issue.

    We would like to hear from someone who actually has control over the Marketplace search features on Windows Phone, because we believe this issue needs immediate attention.

    I've noticed yesterday that the search results returned using the phone's marketplace app have changed. I urgently want to bring this to Microsoft's attention, since I believe the new algorithm for searches is flawed and bad for Windows Phone at a very crucial time. Please note that I have only checked this in the UK, and that the web marketplace is unchanged. I'm also assuming this is a planned change and not a bug.

    Search results now return all the apps with the same name as the search term first, and then other apps that use this keyword next. For example, a search for 'twitter' returns 15 apps called 'twitter', and then other apps such as rowi and gleek next. (Richard.Walters)

    (Image Credit: Andrew Bares)

    A list of some apps affected include:

    • Power Planner (A homework planner) - The paid version is now displayed 6th when searching for "homework" whereas it used to be 1st.
    • Rowi (A twitter client) - Displayed 33rd when searching for "Twitter", behind SO MANY Twitter "apps" that statistically no one uses.
    • Calculator² (A calculator app) - He had to change his app's name to "Calculator" so that it would be visible near the top for searches "calculator", etc.
    • AmazingWeather (A weather app) - Somehow "Weather+" appears third on the list... simply because it has a "+" after the search term "weather".
    • Cuber (A rubik game) - Now shows up 14th instead of 5th when searching for "rubik".

    Because of this issue, here's what some developers have experienced:

    • I've made a game that emulates a popular 3d puzzle cube simulation. Only the official app should use the name "Rubik", and so I allege I wisely used "rubik" simply as a keyword for my game having an original name "Cuber". This allowed users to discover my game without knowing the name.

      My app was also previously highly ranked based on user reviews. However, now no longer appears within the top 5 as it used to. It is now ranked 14th, well below many other apps having a lesser rating simply because they rather inappropriately use a trademarked name in their app title. So that's kind of a bummer. (Taberh)
    • Truly agree. New search algorithm is a bul****. Total number of downloads of my apps decreased upto 10 times because of new algorithm. (asdfasdfasfdfsdfas)
    • Totally BS! So this is the reason why my app downloads were crashing for the past few days despite heavy advertising on Adduplex network.

      Oh please revert to the old search algorithm. (LastBattle)

    Links to topics on this:

    Somebody from Microsoft who has authority on this issue... please respond! This issue is affecting consumers and developers. When consumers search the Marketplace, they'll think that there aren't any good apps available. And independent developers are significantly being hurt by their random and irrational demotion in the search results!

    Tuesday, October 23, 2012 12:41 AM
  • We have been rolling out upgrades to the Windows Phone Store infrastructure over the last several months, including changes to oursearch engine and our list algorithms. 

    As part of this deployment we changed over to the Bing search engine and have modified our criteria to be more flexible, to return apps that take into account user mis-spellings and to display apps that are similar or within the same genre.  The objective in making these changes is to aid users in discovering their favorite apps in the growing catalog.

    This new criteria not only utilizes the app name like our previous search engine, but now also takes into consideration app metadata, downloads, and ratings with the intent to return a list of high quality apps that are relevant to the query.

    We have begun working to fine-tune the algorithm now that the new service has rolled out to most users across the globe. One area we are examining for relevancy is app keywords.  We are finding that some developers are using phrases or popular search terms as keywords. We believe there is room for improvement in the way our algorithms use this app metadata to determine relevance of the search results. We are testing a potential fix for this issue now that will address some of these shortcomings and look to have it in production in a few days’ time. And, we will continue to make improvements to ensure we promote high quality apps.

    Please continue to provide feedback, we’re listening.

    Tuesday, October 23, 2012 5:49 AM
  •  

    1. Substring matching (or even singular form matching) for app title is no longer happening with the new algorithm.

    For e.g. earlier if a user search for "emojicon" then it used to return my app "emojicons" in the result. This is no longer happening.

    http://www.windowsphone.com/en-us/store/search?q=emojicon

    http://www.windowsphone.com/en-us/store/search?q=emojicons

    2. Split word search is not matching with title.

    For e.g. earlier if user searched for "my text twister" then it used to return "mytexttwister" in search results. This is also not happening anymore.

    http://www.windowsphone.com/en-us/store/search?q=my+text+twister

    http://www.windowsphone.com/en-us/store/search?q=mytexttwister

    3. Substring matching (or even singular form matching) is no longer happening for keywords.

    For e.g. Searching for "emoticon" used to result in my app "emojicons" as I have "emoticons" in its keywords. This is no longer happening.

    http://www.windowsphone.com/en-us/store/search?q=emoticon

    http://www.windowsphone.com/en-us/store/search?q=emoticons

    4. Low rated apps are being giving too much priority just because of exact app title match due to which searching for apps with keywords is not giving high quality results. One has to scroll down a lot to find good apps.

    For e.g "Emojicons" (a top 100 app with 4.5+ rating) is listed way down when searching for "emoticons"

    http://www.windowsphone.com/en-us/store/search?q=emoticons

    IMO, the earlier algorithm was much better in finding good apps as it used to give priority to rating/downloads etc.

    Also, I am pretty disappointed with the new search algorithm because of the points I mentioned above. It is pretty normal for user to search for "emojicon" instead of "emojicons" or "my text twister" instead of "mytexttwister", but they are no longer finding my apps. My app downloads have gone significantly down after this new search algo has come into place. I don't see any reason why the new search algo doesnot want to match singular word with their plural forms while it want to correct mis-spelled words.

    Kindly consider the issues I have mentioned above.


    [url]http://www.kunruchcreations.com[/url]

    Tuesday, October 23, 2012 6:23 AM
  • We have been rolling out upgrades to the Windows Phone Store infrastructure over the last several months, including changes to oursearch engine and our list algorithms. 

    As part of this deployment we changed over to the Bing search engine and have modified our criteria to be more flexible, to return apps that take into account user mis-spellings and to display apps that are similar or within the same genre.  The objective in making these changes is to aid users in discovering their favorite apps in the growing catalog.

    This new criteria not only utilizes the app name like our previous search engine, but now also takes into consideration app metadata, downloads, and ratings with the intent to return a list of high quality apps that are relevant to the query.

    We have begun working to fine-tune the algorithm now that the new service has rolled out to most users across the globe. One area we are examining for relevancy is app keywords.  We are finding that some developers are using phrases or popular search terms as keywords. We believe there is room for improvement in the way our algorithms use this app metadata to determine relevance of the search results. We are testing a potential fix for this issue now that will address some of these shortcomings and look to have it in production in a few days’ time. And, we will continue to make improvements to ensure we promote high quality apps.

    Please continue to provide feedback, we’re listening.

    Three years later, and its still not fixed? give me a break:

    https://windows.uservoice.com/forums/265757-windows-feature-suggestions/suggestions/10287726-fair-keyword-search-and-app-discoverability

    and microsoft still dont care:

    https://social.msdn.microsoft.com/Forums/windowsapps/en-US/a3ce97f1-69ce-4070-b6d9-f1e2c900419b/dev-centerlistings-whats-the-point-in-keywords?forum=wpsubmit

    "we are listening". yep. nice.

    Wednesday, October 21, 2015 5:19 AM
  • Unfortunately, this issue seems to have become worse rather than improved with the Windows 10 Store. The image below compares the current ranking of my app for searches using its keywords in the Windows Phone and Windows 10 Mobile Stores.

    Keywords search ranking: WP8.1 vs W10M

    I've added a user voice suggestion here: https://wpdev.uservoice.com/forums/110705-universal-windows-platform/suggestions/11072484-improve-the-store-search-results-by-placing-more-e, with more detailed info here: https://social.msdn.microsoft.com/Forums/windowsapps/en-US/955ce7d3-3828-408a-91ec-352708f26f6d/improve-the-store-search-results-by-placing-more-emphasis-on-keywords?forum=wpsubmit.

    Sunday, December 13, 2015 3:29 PM