SharePoint Dev Center >
> 
> 
How much crawling is to be expected?

Traitée How much crawling is to be expected?

  • Friday, April 13, 2012 11:28 AM
     
     
    Sign In to Vote
    0

    I recently analyzed the access logs for one of our busier sites and was surprised to find that 66% of the accesses came from our index server.  This seems strange to me.  We do a full crawl once/week and incremental crawls each evening.  Does this seem high to you?

    Randy

     

All Replies

  • Friday, April 13, 2012 12:02 PM
     
     Answered
    Sign In to Vote
    0

    The recurrence doesn't seem all that high. But, really it depends an the requirements/SLA for the site and users being able to search for new content within a certain time frame.

    I personally think that full crawls aren't needed as often as weekly, an incremental does a lot more than it typically gets credit for. You could look at moving the full crawl to once a month, but again I would look at the requirements, since this should drive your search topology and crawl schedule.


     
  • Tuesday, April 17, 2012 4:00 AM
    Moderator
     
     Answered
    Sign In to Vote
    0

    Hi Randy,

    Extract the items you can consider on your situation and requirement.

    "When you plan crawl schedules, consider the following best practices:

    • Group start addresses in content sources based on similar availability and with acceptable overall resource usage for the servers that host the content.
    • Schedule incremental crawls for each content source during times when the servers that host the content are available and when there is low demand on the resources of the server.
    • Stagger crawl schedules so that the load on the servers in the farm is distributed over time.
    • Schedule full crawls only when you have to for the reasons listed in the next section. We recommend that you run full crawls less frequently than incremental crawls.
    • Schedule administration changes that require a full crawl to occur shortly before the planned schedule for full crawls. For example, we recommend that you schedule creating the crawl rule before the next scheduled full crawl so that an additional full crawl is not necessary.
    • Base concurrent crawls on the capacity available. For best performance, we recommend that you stagger the crawling schedules of content sources. You can optimize crawl schedules over time as you become familiar with the typical crawl durations for each content source.   "

    For more information you can check the article below,

    http://technet.microsoft.com/en-us/library/cc262926.aspx

    Thanks

    Daniel Yang

    TechNet Community Support