Event ID 2436: The start address cannot be crawled

Question

I'm getting the following error in the event log:

 

The start address <http://www.polymedia.co.nz> cannot be crawled.

Context: Application 'SharedServices', Catalog 'Portal_Content'

Details:

The specified address was excluded from the index. The crawl rules may have to be modified to include this address. (0x80040d07)

 

1.       I can’t see why this URL is excluded, any ideas? (This is our public web site)

2.       I added an explicit craw rule in include this URL, stopped and started the osearch services, did an iisreset and started a full crawl and the same error occurred.

 

Any ideas out there?

 

Rgds

Chris

Answer 1

OK, fixed my own problem ;-)

 

For some reason that no one can tell me we had a robots.txt file the disallowed all indexing to our public site. The contents of robots.txt was:

 

User-agent: *

Disallow: /

 

After removing the file, issuing an iisreset and initiating a full crawl, the site is now indexed. Good to see the Microsoft indexing service observing these directives.

 

Rgds

Chris

Answer 2

Hi,

I am facing problem. There is no robots.txt file in the root folder of my application. Still getting same error. I have included site url in the Crawl Rule but not working.

Any other solution?


Regards,
Dhaval

Answer 3

Dhaval on the same boat like you and still no show....
Have u fixed it?