Crawling complex url's - cannot add a new content source - not possible?

Question

Hi

I have Sharepoint site. Within this site I have a aspx page that takes an id and displays different info based on that; aspx?id=9394 for instance. I am unable to crawl these for the reasons I discuss below.

This site is crawled as a Sharepoint content source type with all the starting addresses we want. From what I've read, this sort of crawling will not include any complex url's ( http://objectmix.com/sharepoint/304300-moss-search-crawl-rules-crawler-not-crawling-complex-urls.html#post1089016 ) no matter what rules one set up.

So, I'm thinking a new content source should do the trick with Web Pages as content source type, but I am not allowed to add a content source that will specify the same start address anywhere within the Sharepoint content source, meaning I'm pretty much stuck (this confirms the same thing: http://sharepoint.microsoft.com/blogs/LKuhn/Lists/Posts/Post.aspx?ID=89 )

Anyone been in the same situation?

Regards,
Mats

Answer 1

Same problem solved.
1. Changed the default Sharepoint content source to the path of an other sharepoint site (ex. intranet site), with no crawling schedule.
2. Added a rule that excluded all items in this path, to prevent crawling this site by accident.
3. Made a new Web site content source, now it was possible to specify the start addresses that earlier were not allowed to add.
4. Made rules for the pages containing the complex url's, and checked for 'Crawl complex URLs'.

Now the crawl log showed that these pages were accessed for crawling, but they were excluded from indexing.

5. Finally looked in, in Site administration > Search visibility. 'Allow this web to appear in search results' were checked.
    Changed from 'Do not index ASPX pages if this site contains fine-grained permissions' to 'Always index all ASPX pages on this site'
    (in this instance there were no security reason for not exposing the information, see http://technet.microsoft.com/en-us/library/cc262033.aspx).

In the next crawl the complex url's were indexed and visible in the search results.

Regards,
Tore