Yahoo crawler doesn't recognize the correct language

Question

User44709200 posted

Hello,

I have a multilanguage website where www.mysite.com is recognized as a french website for all french browsers, and in english for any other foreign browser.

By default Google and Bing crawlers seem to come without any language, so in that case we explain them the www. is in french.

So thousands of pages from www. are indexed in french by Google and Bing, which is perfect (and a en.mysite.com subdomain is dedicated to english).

The problem comes from Yahoo: it is the only one to see the www. in english.

So my conclusion was that the Yahoo crawler was detected as english. In order to solve this issue, I have implemented a crawler detection: if a crawler comes with a language, then it sees the site in french.

Here it is:

If HttpContext.Current.Request.UserLanguages Is Nothing Then
                            ' robots see www.mysite.com as a french site
                            ' ok for google and bing since don't have any language, but wrong for yahoo which seems to arrive in english
                            sLanguage = "fr"
Else
                            ' case robot has a language, especially for yahoo        
                            Dim myBrowserCaps As System.Web.HttpBrowserCapabilities = HttpContext.Current.Request.Browser
                            If (CType(myBrowserCaps, System.Web.Configuration.HttpCapabilitiesBase)).Crawler Then
                                sLanguage = "fr"
                            End If

End If

But it didn't change anything after 3 weeks (I don't know how to see the date of the pages in cache under Yahoo so it's hard to debug).

Do you have any idea ?  Anybody noticed a different behaviour from Yahoo ?

Thanks for your help.

 

 

 

 

Answer 1

User78336405 posted

Hi,

If you want to know more about ASP.NET Globalization and Localization, please refer:http://msdn.microsoft.com/en-us/library/c6zyy3s9.aspx .

I am sorry that I also not encounter this question before , and you could contact some Yahoo Tech Support Engineer.

Answer 2

User44709200 posted

Hi,

Thanks for your answer.

It looks like the code from my first doesn't work with Yahoo.

But I got an answer from their tech support: they recommand to detect "slurp".

I did it, and it's working perfectly now!

Hope it helps...