locked
Dictionary Out Of Memory Exception RRS feed

  • Question

  • I'm running into a problem with a dictionary in my application throwing an out of memory exception.

    Message: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
       at System.Collections.Generic.Dictionary`2.Resize()
       at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
       at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)

    My application and libraries have been compiled to run on any cpu. 

    The OS is 2008 R2 64bit. 

    The server has 64 gigabytes of RAM. 

    Dictionary key is a string that averages around 10 characters.  The value is a string that averages around 25 characters.  I am just adding elements to the dictionary, no removes.  I have had success with tens of millions of elements using this application.  Now I'm running into problems in the new environment where the application is being asked to handle hundreds of million elements.  Right now the count is around 325 million.

    The behavior I have seen using task manager is that the application memory (private working set) will increase steadily to around 8.5 gigabytes and then cease to grow any larger.  Then after five or ten minutes of no growth in consumed memory the application will throw the exception above.  The server averages has around 40 gigabytes of memory available before starting the application so there is plenty of memory to grow.  When the exception is thrown there is normally about 28 gigabytes of available memory.

    I am under the impression that there is no size limit for system.collections.generic.dictionary.  I am also under the impression that there is no size limit for .net applications on 64bit servers.  Is this not correct?  Looking at the exception, my guess is that since there are so many elements in the dictionary the resize method is not able to work properly.

    Any insight would be appreciated.

    Monday, June 25, 2012 7:57 PM

Answers

  • There's probably no hard coded limit to the Dictionary, but there's going to be limits based off of how the Large Object Heap has its memory allocated and the fact that internally the Dictionary stores everything in a big array (or two). I would suspect because of that, that Dictionary would have an upper limit of Int32.MaxValue (~2 billion).

    If you're just adding values to the Dictionary, every time it grows it will double the size of the internal array. Once that crosses over to the Large Object Heap, the location of the arrays in memory stop shifting around. So each new array will end up taking up new memory addresses, because it will be bigger than the rest of the previously allocated Large Object Heap addresses. This will quickly create really poor memory fragmentation. So you might be able to get away initializing the Dictionary to a couple hundred million elements that you aren't able to do with adding one at a time and having the internal structure grow on demand.

    I think you're at the point where you should be looking at a data structure specialized for larger data sets. You could try the PersistentDictionary.

    • Marked as answer by Big Hoss Tuesday, June 26, 2012 12:40 PM
    Monday, June 25, 2012 9:09 PM
  • You might be surprised at how performant ESE (what PersistentDictionary is wrapped around) is. Of course if there's pattern to your access of items then it would have to hit the disk allot.

    There was one quick and dirty program I wrote for fun one night where I was running into this issue. I was able to squeeze out more memory from the Large Object Heap by having an array of 10 Dictionary<>'s. My keys were evenly distributed so I wasn't worried about any one of the collections getting too much bigger than the others. So for all of my accesses I just indexed into the array by modding the key by 10 and grabbed that Dictionary. It really helped with the memory fragmentation issue.

    • Marked as answer by Big Hoss Tuesday, June 26, 2012 12:40 PM
    Tuesday, June 26, 2012 2:18 AM

All replies

  • There's probably no hard coded limit to the Dictionary, but there's going to be limits based off of how the Large Object Heap has its memory allocated and the fact that internally the Dictionary stores everything in a big array (or two). I would suspect because of that, that Dictionary would have an upper limit of Int32.MaxValue (~2 billion).

    If you're just adding values to the Dictionary, every time it grows it will double the size of the internal array. Once that crosses over to the Large Object Heap, the location of the arrays in memory stop shifting around. So each new array will end up taking up new memory addresses, because it will be bigger than the rest of the previously allocated Large Object Heap addresses. This will quickly create really poor memory fragmentation. So you might be able to get away initializing the Dictionary to a couple hundred million elements that you aren't able to do with adding one at a time and having the internal structure grow on demand.

    I think you're at the point where you should be looking at a data structure specialized for larger data sets. You could try the PersistentDictionary.

    • Marked as answer by Big Hoss Tuesday, June 26, 2012 12:40 PM
    Monday, June 25, 2012 9:09 PM
  • There's probably no hard coded limit to the Dictionary, but there's going to be limits based off of how the Large Object Heap has its memory allocated and the fact that internally the Dictionary stores everything in a big array (or two). I would suspect because of that, that Dictionary would have an upper limit of Int32.MaxValue (~2 billion).

    If you're just adding values to the Dictionary, every time it grows it will double the size of the internal array. Once that crosses over to the Large Object Heap, the location of the arrays in memory stop shifting around. So each new array will end up taking up new memory addresses, because it will be bigger than the rest of the previously allocated Large Object Heap addresses. This will quickly create really poor memory fragmentation. So you might be able to get away initializing the Dictionary to a couple hundred million elements that you aren't able to do with adding one at a time and having the internal structure grow on demand.

    I think you're at the point where you should be looking at a data structure specialized for larger data sets. You could try the PersistentDictionary.

    Thanks for the reply. 

    I checked out the PresistentDictionary and looks very useful.  I really liked what I saw on the codeplex page.

    However I am really trying to keep this data in memory and not on disk (if possible).  I know that doesn't sound wise with the number of elements I'm dealing with but the hope is that keeping it in memory will be faster for adding and looking up what i need.  The application will perform lookups on the dictionary at least tens of millions of times within a single hour.

    I completely forgot about initializing the dictionary with a capacity (dumby == me) so I will try that and go from there.  I assume that would prevent the memory fragmentation since the dictionary will be filled and won't need to grow?

    Monday, June 25, 2012 11:19 PM
  • You might be surprised at how performant ESE (what PersistentDictionary is wrapped around) is. Of course if there's pattern to your access of items then it would have to hit the disk allot.

    There was one quick and dirty program I wrote for fun one night where I was running into this issue. I was able to squeeze out more memory from the Large Object Heap by having an array of 10 Dictionary<>'s. My keys were evenly distributed so I wasn't worried about any one of the collections getting too much bigger than the others. So for all of my accesses I just indexed into the array by modding the key by 10 and grabbed that Dictionary. It really helped with the memory fragmentation issue.

    • Marked as answer by Big Hoss Tuesday, June 26, 2012 12:40 PM
    Tuesday, June 26, 2012 2:18 AM
  • Well I tried initializing the dictionary with the number of elements I expected to add.  However it is now throwing the out of memory exception during the initialization of the dictionary.  Going to try your other suggestions now.
    Tuesday, June 26, 2012 2:02 PM
  • Guys, this commonly happens when you access a Dictionary simultaneously by multiple threads. It is doubtfull this is an actually memory issues.
    Thursday, July 14, 2016 12:12 PM