locked
Experiences with storing 250.000 items in a Sharepoint 2010 Document Library without folders ? RRS feed

  • Question

  • Hello everybody,
     
    I would like to store about 250.000 PDFs (average size of one PDF:  about 1,4 MB) in a SharePoint 2010 Document Library.  Does somebody have experience with this amout of items without folders in a single Document Library  ?
     
    My questions are especially:
    1.       Is storing this amout of items flat (without folders) officially supported by Microsoft ? (the only data i found is that 30 million items per document library  is supported, but it doesn’t explicitly say if this amount is only possible/supported with folders [source: http://technet.microsoft.com/en-us/library/cc262787.aspx]) 
    2.       One of our solution architects expressed concerns that some items might get corrupted when storing this amount( items flat, do you have experience with that ?
    3.       Are there any other reasons you would't recommend storing this amount of items flat ?
    Peformance is not a primary conern, as the items would be filtered with views
    Best regards
    Wednesday, March 23, 2011 5:50 PM

Answers

  • In a word: No - what you are suggesting is typically not supported due to the total size of the site collection that contains the document library.

    Generally speaking a site collection should be a maximum of 100GB.

    However, there are a couple of exceptional scenarios where you can exceed this limit:

    • It is the only site collection in the content database (in which case the maximum supported limit is 200GB).
    • It is a non-collaborative, single site repository (e.g. a records centre), in which case the limit is up to 1 terabyte.

    Given that you are storing PDFs, you may fall into the second category assuming this will be the only Web in your site collection.

    Assuming your SQL server is tuned and maintained correctly you should have no issues around corruption.

    See "SharePoint Server 2010 capacity management: Software boundaries and limits" for more info.

     


    Benjamin Athawes - MCTS & MCITP SharePoint 2010
    Twitter
    SharePoint Blog

    • Marked as answer by KeFang Chen Thursday, March 31, 2011 1:31 AM
    Wednesday, March 23, 2011 11:54 PM

All replies

  • Doing that all in 1 library would concern me greatly.  it is also 341GB which is above the recomended limit. (although it is not a hard limit)  Doing what you want is possible, nothing will prevent it. But i would say it is not a good idea.

     

    Will this be an "active" library with people coming in and consuming the info or will it be just an archive of some type?

    Have you looked into using RBS or the like to externalize the content?

    Wednesday, March 23, 2011 7:27 PM
  • In a word: No - what you are suggesting is typically not supported due to the total size of the site collection that contains the document library.

    Generally speaking a site collection should be a maximum of 100GB.

    However, there are a couple of exceptional scenarios where you can exceed this limit:

    • It is the only site collection in the content database (in which case the maximum supported limit is 200GB).
    • It is a non-collaborative, single site repository (e.g. a records centre), in which case the limit is up to 1 terabyte.

    Given that you are storing PDFs, you may fall into the second category assuming this will be the only Web in your site collection.

    Assuming your SQL server is tuned and maintained correctly you should have no issues around corruption.

    See "SharePoint Server 2010 capacity management: Software boundaries and limits" for more info.

     


    Benjamin Athawes - MCTS & MCITP SharePoint 2010
    Twitter
    SharePoint Blog

    • Marked as answer by KeFang Chen Thursday, March 31, 2011 1:31 AM
    Wednesday, March 23, 2011 11:54 PM
  • I know this is an old thread but I just wanted to chime in and say we have been using document libraries this size and larger for years (since SP 2003 in fact). My recommendations would be:

    - Put this library in a dedicated site collection in a dedicated content database. We have web apps that house only large libraries, with their own app pools and identities (we have web apps for 'Records' which may contains a dozen or so site collections each with libraries containing 100-500k items each, some of the files are several MB each with multiple versions).

    - It helps to use the information management capabilities to delete files at a set time period if possible; often these libraries are created to house documents that have a certain retention requirement. This way the library will not just grow indefinitely.

    - Ensure planning goes into the creation of views; the performance the users who need to interface with this library will experience is primarily dependent on the views; the number of items returned and the complexity of the query required to build that view. Indexed columns can help with view performance. I had to have a conversation with one of our departments about their views, when they came to terms with the fact that a view returning 300k+ items in a datasheet view was not going to perform very well and realized they could return items in batches of 5-10 thousand or so with much better performance, they were happy. you may need to tweak those 2010 throttling settings. Another reason to organize them into a special web application (view throttling settings are defined per web app).

    - Make sure you can backup a database of this size as the native SharePoint backups/exports/imports will have trouble with this, they should work eventually but they just take an exceptionally long time and need a lot of extra disk space to complete.

    - Keep an eye on your SQL servers performance, a properly configured SQL server should not care about a database of this size at all... SQL can handle very large databases. Disk performance, memory etc are all important of course, watch those BCHR and PLE counters to get a sense of the SQL server performance.

    -If you use file folders remember views can flatten them, in some cases we use file folders containing about 5k items each and flatten the folder structure with views. Using explorer view can be problematic without this but in most of these libraries explorer view is not a requirement for us, and we do have libraries that are hundreds of thousands of items that are in a completely flat library.

    - We used to make search indexes for each of our large libraries, since 2010 I've just reduced the number of indexes significantly. I still like to have a separate index for the records web app, but I on longer try to have one for each large library as the search crawler is vastly improved, though if you need to use the pdf ifilter to do text searchable pdfs at this scale you might find a point where you need multiple search index servers, though this really only matters during full crawls, which are not needed frequently.

    The other responders talk about support for large libraries, even before SP1 for 2010 the 'limits' were recommendations, not hard limits. Sure you might have a support engineer fall out of his chair, but SP is quite capable of handling very large libraries, and they should know this.

    • Proposed as answer by bgmeek Monday, March 25, 2013 4:57 AM
    Thursday, August 18, 2011 12:40 PM