locked
Robot.txt file help RRS feed

  • Question

  • User-1668014665 posted

    I wish to disallow all Bing robots.txt only crawling my site, allow all others.

    Is it like this

    #Stop MScroft bots
    User-agent: bingbot
    User-agent: msnbot
    User-agent: msnbot-media
    User-agent: adi dxbot
    Disallow: /
    
    #Allow all other bots
    User-agent: *
    Disallow: /folder1/
    Disallow: /folder2/
    Disallow: /folder3/
    Disallow: /folder4/
    Disallow: /myfile.aspx


    Thursday, June 29, 2017 5:40 AM

All replies

  • User-1315512054 posted

    Hello,

    It seems Ok.

    Regards

    Thursday, June 29, 2017 6:16 AM
  • User-1838255255 posted

    Hi icm63,

    According to your description, as far as I know, you want to disallow the bingbot to crawl your site.

    You could try write like it:

    #robots.txt 
    User-agent: *
    Disallow:
    Disallow: *.axd
    Disallow: /cgi-bin/
    Disallow: /member
    User-agent: bingbot
    Disallow: /
    User-agent: ia_archiver
    Disallow: /

    For more details, please refer to the following tutorial:

    Block bingbot from crawling my site:

    https://stackoverflow.com/questions/27188531/block-bingbot-from-crawling-my-site 

    Best Regards,

    Eric Du

    Friday, June 30, 2017 5:50 AM
  • User-1668014665 posted
    
    User-agent: *
    Disallow: /folder1/
    Disallow: /folder2/
    Disallow: /folder3/
    Disallow: /folder4/
    Disallow: /myfile.aspx
    User-agent: bingbot
    Disallow: /
    User-agent: msnbot
    Disallow: /
    User-agent: msnbot-media
    Disallow: /
    User-agent: adi dxbot
    Disallow: /
    

    Are you the above format, order matters then, and a Disallow: / for each User-agent ??

    Friday, June 30, 2017 6:09 PM