Can the user-agent of the SEO crawler be changed? RRS feed

  • Question

  • User-1787108167 posted

    The site I own that wish to check is hosted for me, and I do not control the robots.txt page it serves up. My problem is that it only allows the common web search crawlers and disallows everything else: http://photos.hemsfamily.com/robots.txt

    Can I override the user-agent that the SEO crawler behaves as in order for it to index my stuff?

    Monday, July 11, 2011 10:21 AM

All replies

  • User-47214744 posted

    Unfortunately no, we decided not to allow it since it could be used to disguise the crawler and potentially try to bypass logic that servers would not be able to identify. So even though you could do that with others, we decided to "be good citizens" and not enable hiding it.

    That means that unfortunately in valid cases like yours it won't be possible.

    Monday, July 18, 2011 12:33 AM
  • User-1493879410 posted

    Hi seoblogging, do we really need to edit the pages html for that matter? I'm using wordpress with SEO plugin that allows me to select which page to be index and which not. Is this the same as compare to modifying the html code?

    Thursday, October 27, 2011 3:51 AM
  • User2044130619 posted

     I agree with Calos that, cannot control the robots.txt then it is not possible to change the behavior of crawlers

    Tuesday, November 8, 2011 2:25 AM
  • User-1787108167 posted

    Thanks everyone - I do use Webmaster Tools on both Bing and Google to see what is being indexed but I have so many pages (30k+) which are created automagically by my hosting provider (hence I don't know what they all are), that I was hoping to use the SEO toolkit to do do a thorough analysis of what my site actually looks like to a crawler like Bing and Google so that I can exlude the duplicate pages via the Webmaster tools. Alas, I cannot do so as my hosting provider has one robots.txt for every single site they host so I just have to do it manually :-P

    - Adam Hems

    Monday, November 21, 2011 11:04 AM
  • User-1059655912 posted

    There is an add-on that can be installed, after installing that, go to Firefox option and you will come across the add-on user agent switcher added to the list of the existing add-ons. This add-on comprises of a Google bot user agent that you can select.Sometimes the existing user agents get outdated when Google changes the user agent; therefore in case you dont want the existing agent, it is recommended that you edit and add your own user agent.

    Saturday, January 14, 2012 2:01 AM
  • User-1059655912 posted

     You can install an add on if you want that provides with the present Google bot user agent. You can edit or add your own user agents there. This is the user agent switcher provided by Firefox.

    Saturday, January 14, 2012 2:04 AM
  • User-2110569270 posted

     yes you can change it using webmaster tool

    Friday, January 20, 2012 6:18 AM
  • User-810625232 posted

    I would like to use IIS SEO Toolkit to crawl our site with a mobile user agent. Our site displays differnet pages to desktop and mobile users. Ideally, I'd like to be able to set the useragent to emulate Google's smartphone bot:

    Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

    Wednesday, June 12, 2013 8:13 PM