locked
Memory Management on Foreach RRS feed

  • Question

  • Hello,

      I have a list of objects returned via linqtoSql from the DB.  When I iterate the list using

    foreach(myobject x in myListofOjbects)

    {

    }

    the memory usage goes up to close to 4GB and then I get an out of memory exception.  There are no objects inside the {} other than x.  I am guessing the iteration does not get rid of the previous objects.  Before I recode to exit the foreach before this happens, I wonder if there is any other way to fix it?

    Tuesday, November 26, 2013 2:27 PM

Answers

  • Hi Mataniah,

    when you are using LINQ To SQL fetched objects are tracked (cached) by default. To disable the caching set the DataContext.ObjectTrackingEnabled Property to false. Be aware, that will prevent changing the data later as the result will be read-only.

    You should also check if you can limit the query as using 4 GB and more indicates that you are fetching a lot of data. If possible use a Where clause to limit the result instead of comparing the values locally.

    It doesn't matter if you are using foreach or GetEnumerator. A for loop on the other hand that uses the Count property may force populating the whole collection, the same applies if you are using ToList().

    Regards, Elmar

    • Marked as answer by Mattaniah Monday, December 2, 2013 9:53 AM
    Wednesday, November 27, 2013 3:51 PM
  • >I do call a toList() before I iterate; 

    That will bring the whole collection into memory.  Either limit the results by using a more selective query, or don't call ToList().  You can run the query with AsEnumerable() to get a streaming collection or iterate it directly with foreach, which does the same thing.

    Using an Enumerator or a for loop won't help. 

    David


    David http://blogs.msdn.com/b/dbrowne/

    • Marked as answer by Mattaniah Monday, December 2, 2013 9:54 AM
    Wednesday, November 27, 2013 6:16 PM

All replies

  • Can you share the LINQ and what you're doin with x in the loop? My guess is that the actual list may be taking up the space since the query isn't evaluated until you try to iterate myListofObjects (unless of course you called ToList to something similar) you don't see the memory usage go up until you get to your foreach loop. And no the iteration has no side effect on your list.
    Tuesday, November 26, 2013 2:40 PM
  • Hi,

    Maybe you can do a for-loop instead?

    for(int i=0;i<myListOfObjects.Count;i++)
    {
    //do something with myListOfObjects[i]..
    }


    Tuesday, November 26, 2013 3:17 PM
  • Use an enumerator instead of enumerable.  See if something like below works

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    
    namespace ConsoleApplication1
    {
        class Program
        {
            static void Main(string[] args)
            {
                List<int> numbers = new List<int> { 1, 2, 3, 4, 5 };
    
                IEnumerator<int> read = numbers.GetEnumerator();
                while (read.MoveNext())
                {
                    Console.WriteLine(read.Current.ToString());
                }
            }
        }
    }
    


    jdweng

    Tuesday, November 26, 2013 3:29 PM
  • List<myOjbectResult> _file = dc.LimitFileCount().toList();

    foreach(myObjectResult x in _file)
    {

    }


    I do call a toList() before I iterate; 

    Wednesday, November 27, 2013 2:47 PM
  • Look at my code.  I don't use a foreach, instead I use a while.

    jdweng

    Wednesday, November 27, 2013 3:12 PM
  • Hi Mataniah,

    when you are using LINQ To SQL fetched objects are tracked (cached) by default. To disable the caching set the DataContext.ObjectTrackingEnabled Property to false. Be aware, that will prevent changing the data later as the result will be read-only.

    You should also check if you can limit the query as using 4 GB and more indicates that you are fetching a lot of data. If possible use a Where clause to limit the result instead of comparing the values locally.

    It doesn't matter if you are using foreach or GetEnumerator. A for loop on the other hand that uses the Count property may force populating the whole collection, the same applies if you are using ToList().

    Regards, Elmar

    • Marked as answer by Mattaniah Monday, December 2, 2013 9:53 AM
    Wednesday, November 27, 2013 3:51 PM
  • OK well if that's the case you are evaluating the query and bringing back all the data so your query is probably returning a result set that is too large, try using a where clause or something to return the results in smaller chunks that are more workable
    Wednesday, November 27, 2013 5:29 PM
  • >I do call a toList() before I iterate; 

    That will bring the whole collection into memory.  Either limit the results by using a more selective query, or don't call ToList().  You can run the query with AsEnumerable() to get a streaming collection or iterate it directly with foreach, which does the same thing.

    Using an Enumerator or a for loop won't help. 

    David


    David http://blogs.msdn.com/b/dbrowne/

    • Marked as answer by Mattaniah Monday, December 2, 2013 9:54 AM
    Wednesday, November 27, 2013 6:16 PM
  • I think the result set is actually Lazily Created. Wich means nothing will be behind those reference before you actually use them for the first tome. See Lazy<T> class for such an example.

    There is nothing the Runtime could get "rid off". As long as you hold a reference to the isntance, the garbage collector cannot collect it. The GC already tried to do all the collection work possible before ever giving you a OOM Exception.

    I agree with others that the result set is very likely too large. 4 GiB (that also have to be transfered over network or loaded from disk first) implies that you are retrieving much too much data.
    When working with any Database keep in mind that all queries are costly. Retrieve as little data as you actually need and as much as you can on one go. Do all filtering on the DB side using a where clause.
    If you need to do filtering on the Client side, something is likely wrong with your DB design. Better ask in the DB forum to properly resutrcture it so significant WHERE clauses can be written for filtering.


    Let's talk about MVVM: http://social.msdn.microsoft.com/Forums/en-US/wpf/thread/b1a8bf14-4acd-4d77-9df8-bdb95b02dbe2 Please mark post as helpfull and answers respectively.

    Wednesday, November 27, 2013 6:58 PM
  • What you do inside the loop is crucial, show us the code.

    foreach (var x in SomeCollection) {
       CallFaultyMethodWhichAllocatesLotsOfMemeoryAndDoesntReleaseIt(x);
    }

    this will give you a problem.

    Paul Linton

    • Proposed as answer by Jason Dot Wang Friday, November 29, 2013 9:27 AM
    Wednesday, November 27, 2013 10:19 PM