Finding wrong robots.txt RRS feed

  • Question

  • User1509032000 posted
    On our server I've been trying to run a Site Analysis report but it doesn't get beyond the site root due to a Robots.txt rule not allowing it, however there is no Robots.txt.

    If I go to 'http://.../robots.txt' it does find a file, and it's as below,

    User-agent: *
    Disallow: /

    But again, this file does not exist anywhere on the server, I've done a full search across all drives, all folders to be sure and there isn't a single Robots.txt on there, creating one with the correct content in it and placing that one in site root doesn't change anything and it continues to show the above.

    I just can't figure out whats going on. Any ideas would be really appreciated.

    Wednesday, August 1, 2012 5:37 PM

All replies

  • User-736483138 posted
    Have you verified that you are looking at the right server? What does http://network-tools.com show as the A record for that domain?
    Wednesday, August 1, 2012 10:54 PM
  • User128512046 posted
    First thing that your web site has robot.txt not Robot.txt (case sensitive issue). Try to generate URL list using free sitemap tool generate or else use Xenu links analyzer to find the right path of robots.txt. However check files status might be any hidden file over there on web server or else might your checking wrong server. Get full status of listed issue...! Wish you good luck!
    Monday, August 13, 2012 5:26 AM