SharePoint Server 2007 search crawler and Large Lists

Question

Hi,

We have many large lists in our site and I like to make them efficient for search crawler.

Is there any “Crawling large list” best practices?

Can I have crawler only index the default view of the list? So I can limit the number of list items being indexed.

Thank you,

Answer 1

Hi,

Based on my research, firstly, if the indexer is timing out when crawling large lists, you can increase the time-out value. Secondly, I learned that modifying these registry values can help you crawl large lists in SharePoint 2007:

Key	Default Value	New Value
HKLM\SOFTWARE\Microsoft\Office Server\12\Search\Global\GatheringManager\DedicatedFilterProcessMemoryQuota	104,857,600	209,715,200
HKLM\SOFTWARE\Microsoft\Office Server\12\Search\Global\GatheringManager\FilterProcessMemoryQuota	104,857,600	209,715,200
HKLM\SOFTWARE\Microsoft\Office Server\12\Search\Global\GatheringManager\FolderHighPriority	50	500

For the detailed information about crawling large lists in SharePoint 2007, check out the following article:

http://blogs.msdn.com/b/joshuag/archive/2009/10/05/crawling-large-lists-in-sharepoint-2007.aspx

Thanks,

Rock Wang

Forum Support

Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Subscriber Support, contact [e-mail].

---

Rock Wang TechNet Community Support

Answer 2

Hi Rock Wang,

Thank you for your post, but this is not exactly refers to my question.

I am looking for best practices, how to optimize the large lists to be crawled better.

Are large lists that are broken into folders indexed better?

How Can I create a filtered view that represent limited number of large list items, and have crawler only index the filtered view?

Thank you,

Answer 3

Hi,

You also should use folders in order not to degrade indexing (i.e. crawling) of large web-structures with several large lists. Views won't help here.

As a general guideline, any list should not have more than 2,000 items per list container. By container, I mean the root of the list, or any folder inside it. So for example, if you have a list with 1,900 list items in the list root, plus 100 folders each containing 2,000 items, then you are fine. Just make sure that your views do not attempt to show or edit more than 2,000 items at a time.  In total, counting all items recursively in folders, SharePoint supports up to 5 million items in each list.

For the detailed information about managing large SharePoint lists for better performance, please see the following articles:

http://sharepoint.microsoft.com/blogs/GetThePoint/Lists/Posts/Post.aspx?ID=162

http://technet.microsoft.com/library/cc262813(office.12).aspx

Thanks,

Rock Wang

Forum Support

Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Subscriber Support, contact [e-mail].

---

Rock Wang TechNet Community Support