locked
Yahoo crawler doesn't recognize the correct language RRS feed

  • Question

  • User44709200 posted

    Hello,

    I have a multilanguage website where www.mysite.com is recognized as a french website for all french browsers, and in english for any other foreign browser.

    By default Google and Bing crawlers seem to come without any language, so in that case we explain them the www. is in french.

    So thousands of pages from www. are indexed in french by Google and Bing, which is perfect (and a en.mysite.com subdomain is dedicated to english).

    The problem comes from Yahoo: it is the only one to see the www. in english.

    So my conclusion was that the Yahoo crawler was detected as english. In order to solve this issue, I have implemented a crawler detection: if a crawler comes with a language, then it sees the site in french.

    Here it is:

    If HttpContext.Current.Request.UserLanguages Is Nothing Then
                                ' robots see www.mysite.com as a french site
                                ' ok for google and bing since don't have any language, but wrong for yahoo which seems to arrive in english
                                sLanguage = "fr"
    Else
                                ' case robot has a language, especially for yahoo        
                                Dim myBrowserCaps As System.Web.HttpBrowserCapabilities = HttpContext.Current.Request.Browser
                                If (CType(myBrowserCaps, System.Web.Configuration.HttpCapabilitiesBase)).Crawler Then
                                    sLanguage = "fr"
                                End If

    End If

    But it didn't change anything after 3 weeks (I don't know how to see the date of the pages in cache under Yahoo so it's hard to debug).

    Do you have any idea ?  Anybody noticed a different behaviour from Yahoo ?

    Thanks for your help.

     

     

     

     

    Sunday, December 27, 2009 11:16 AM

Answers

  • User44709200 posted

    Hi,

    Thanks for your answer.

    It looks like the code from my first doesn't work with Yahoo.

    But I got an answer from their tech support: they recommand to detect "slurp".

    I did it, and it's working perfectly now!

    Hope it helps...

    • Marked as answer by Anonymous Thursday, October 7, 2021 12:00 AM
    Friday, January 15, 2010 5:13 AM

All replies

  • User78336405 posted

    Hi,

    If you want to know more about ASP.NET Globalization and Localization, please refer:http://msdn.microsoft.com/en-us/library/c6zyy3s9.aspx .

    I am sorry that I also not encounter this question before , and you could contact some Yahoo Tech Support Engineer.

    Tuesday, December 29, 2009 3:05 AM
  • User44709200 posted

    Hi,

    Thanks for your answer.

    It looks like the code from my first doesn't work with Yahoo.

    But I got an answer from their tech support: they recommand to detect "slurp".

    I did it, and it's working perfectly now!

    Hope it helps...

    • Marked as answer by Anonymous Thursday, October 7, 2021 12:00 AM
    Friday, January 15, 2010 5:13 AM