locked
Variables in LINQ Lambda Expressions cause expression to re-evaluate dynamically if changed.

    Question

  • I found an issue with LINQ dynamically re-evaluating the results of an expression when a variable used in the expression is changed after the expression is evaluated.

    I put sample code on my blog.


    Is this intended behaviour, or a quirky bug? It is certainly something to be wary of when using lambda expressions in recursion.


    SAMPLE CODE - changing the indx value after the count is evaluated changes the result set immediately!


    private void button1_Click(object sender, EventArgs e)
    {
    System.Diagnostics.Debug.WriteLine("Testing:" + DateTime.Now.ToString());
    List<Dictionary<int, int>> ld = new List<Dictionary<int, int>>();

    ld.Add(new Dictionary<int, int>() { { 1, 10 }, { 2, 20 }, { 3, 30 } });
    ld.Add(new Dictionary<int, int>() { { 1, 20 }, { 2, 300000}, { 3, 40000000 }, { 4, 500000000 }});

    int indx = 10;

    var ds = ld.Where(itm => itm[1] == indx);

    if (ds.Count() > 0)
    {
    System.Diagnostics.Debug.WriteLine("Count=" + ds.First().Count());
    indx = ds.First()[2]; //this is a value I want to keep for later
    int valueIwant = ds.First()[3]; //this is the value I want from this result

    System.Diagnostics.Debug.WriteLine("Value I Want=" + valueIwant);
    //oh no ??? that's not the value I wanted ???
    System.Diagnostics.Debug.WriteLine("Count=" + ds.First().Count());
    //and now the count changed ???
    }
    }
     
    • Moved by Noam Ben-Ami - MSFT1 Tuesday, November 10, 2009 6:35 PM (From:ADO.NET Entity Framework and LINQ to Entities)
    • Changed type BBeattie Friday, November 13, 2009 3:24 AM Lingzhi Sun asked
    Tuesday, November 10, 2009 3:31 AM

Answers

  • Hi Brett,

     

    Really interesting question!  However, I think it is more related to LINQ to Objects instead of LINQ to Entities.  J

     

    Such behavior is caused by the design of IEnumeable<T> and Enumerable.  The variable ds is in type of IEnumerable<Dictionary<int, int>>.  The Where extension method of the List ld is implemented in the Enumerable.Where():  (Here is its implementation in .NET Reflector)

    ====================================================================
    public static IEnumerable<TSource> Where<TSource>(this IEnumerable<TSource> source, Func<TSource, bool> predicate)

    {

        if (source == null)

        {

            throw Error.ArgumentNull("source");

        }

        if (predicate == null)

        {

            throw Error.ArgumentNull("predicate");

        }

        if (source is Iterator<TSource>)

        {

            return ((Iterator<TSource>) source).Where(predicate);

        }

        if (source is TSource[])

        {

            return new WhereArrayIterator<TSource>((TSource[]) source, predicate);

        }

        if (source is List<TSource>)

        {

            return new WhereListIterator<TSource>((List<TSource>) source, predicate);

        }

        return new WhereEnumerableIterator<TSource>(source, predicate);

    }
    ====================================================================

     

    For List<TSource>, the method returns WhereListIterator<TSource>.  Also, the Func<> is passed in to filter the data.  However, after the line of codes var ds = ld.Where(item => item[1] == indx);, the collection is not actually generated.  C# uses GetEnumerator and MoveNext methods to loop the IEnumerable<T> objects.  

     

    Here are the implementation of your sample codes in Reflector:

    ====================================================================
        int indx = 10;

        IEnumerable<Dictionary<int, int>> ds = from item in ld

                                                                        where item[1] == indx

                                                                        select item;

        if (ds.Count<Dictionary<int, int>>() > 0)

        {

            Debug.WriteLine("Count=" + ds.First<Dictionary<int, int>>().Count<KeyValuePair<int, int>>());

            indx = ds.First<Dictionary<int, int>>()[2];

            Debug.WriteLine("Value I Want=" + ds.First<Dictionary<int, int>>()[3]);

            Debug.WriteLine("Count=" + ds.First<Dictionary<int, int>>().Count<KeyValuePair<int, int>>());

    }
    ====================================================================

    Each time we use the ds variable, certain extension method is executed and the Func<Dictionary<int, int>, bool> is used to filter the data.  So if the indx variable is modified, the subsequent codes may get different values. 

     

    The compiler will auto-generated such a private helper class for the Func<Dictionary<int, int>, bool>,

    ====================================================================
    [CompilerGenerated]

    private sealed class <>c__DisplayClass3

    {

        // Fields

        public int indx;

     

        // Methods

        public bool <button1_Click>b__2(Dictionary<int, int> item)

        {

            return (item[1] == this.indx);

        }

    }
    ====================================================================

    For detailed information about why we have such a helper class, please see
    http://blogs.msdn.com/oldnewthing/archive/2006/08/02/686456.aspx.  

     

     

    You may have found it that converting the IEnumerable<T> to List<T> or T[], the problem can be solved.  (var ds = ld.Where(item => item[1] == indx).ToList();)   However, due to such design, C# does not support tail recursion.  Here is another related article for your references:

    http://www.atalasoft.com/cs/blogs/stevehawley/archive/2009/06/02/more-ienumerable-t-fun.aspx

     

     

    Thanks again for your posting!  If you have any questions, I’d love to discuss with you.

     

     

    Have a nice day!

     

     

    Best Regards,
    Lingzhi Sun

    MSDN Subscriber Support in Forum

    If you have any feedback on our support, please contact msdnmg@microsoft.com


    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    Welcome to the All-In-One Code Framework! If you have any feedback, please tell us.
    • Marked as answer by BBeattie Friday, November 13, 2009 3:24 AM
    Tuesday, November 10, 2009 9:35 AM
    Moderator

All replies

  • Hi Brett,

     

    Really interesting question!  However, I think it is more related to LINQ to Objects instead of LINQ to Entities.  J

     

    Such behavior is caused by the design of IEnumeable<T> and Enumerable.  The variable ds is in type of IEnumerable<Dictionary<int, int>>.  The Where extension method of the List ld is implemented in the Enumerable.Where():  (Here is its implementation in .NET Reflector)

    ====================================================================
    public static IEnumerable<TSource> Where<TSource>(this IEnumerable<TSource> source, Func<TSource, bool> predicate)

    {

        if (source == null)

        {

            throw Error.ArgumentNull("source");

        }

        if (predicate == null)

        {

            throw Error.ArgumentNull("predicate");

        }

        if (source is Iterator<TSource>)

        {

            return ((Iterator<TSource>) source).Where(predicate);

        }

        if (source is TSource[])

        {

            return new WhereArrayIterator<TSource>((TSource[]) source, predicate);

        }

        if (source is List<TSource>)

        {

            return new WhereListIterator<TSource>((List<TSource>) source, predicate);

        }

        return new WhereEnumerableIterator<TSource>(source, predicate);

    }
    ====================================================================

     

    For List<TSource>, the method returns WhereListIterator<TSource>.  Also, the Func<> is passed in to filter the data.  However, after the line of codes var ds = ld.Where(item => item[1] == indx);, the collection is not actually generated.  C# uses GetEnumerator and MoveNext methods to loop the IEnumerable<T> objects.  

     

    Here are the implementation of your sample codes in Reflector:

    ====================================================================
        int indx = 10;

        IEnumerable<Dictionary<int, int>> ds = from item in ld

                                                                        where item[1] == indx

                                                                        select item;

        if (ds.Count<Dictionary<int, int>>() > 0)

        {

            Debug.WriteLine("Count=" + ds.First<Dictionary<int, int>>().Count<KeyValuePair<int, int>>());

            indx = ds.First<Dictionary<int, int>>()[2];

            Debug.WriteLine("Value I Want=" + ds.First<Dictionary<int, int>>()[3]);

            Debug.WriteLine("Count=" + ds.First<Dictionary<int, int>>().Count<KeyValuePair<int, int>>());

    }
    ====================================================================

    Each time we use the ds variable, certain extension method is executed and the Func<Dictionary<int, int>, bool> is used to filter the data.  So if the indx variable is modified, the subsequent codes may get different values. 

     

    The compiler will auto-generated such a private helper class for the Func<Dictionary<int, int>, bool>,

    ====================================================================
    [CompilerGenerated]

    private sealed class <>c__DisplayClass3

    {

        // Fields

        public int indx;

     

        // Methods

        public bool <button1_Click>b__2(Dictionary<int, int> item)

        {

            return (item[1] == this.indx);

        }

    }
    ====================================================================

    For detailed information about why we have such a helper class, please see
    http://blogs.msdn.com/oldnewthing/archive/2006/08/02/686456.aspx.  

     

     

    You may have found it that converting the IEnumerable<T> to List<T> or T[], the problem can be solved.  (var ds = ld.Where(item => item[1] == indx).ToList();)   However, due to such design, C# does not support tail recursion.  Here is another related article for your references:

    http://www.atalasoft.com/cs/blogs/stevehawley/archive/2009/06/02/more-ienumerable-t-fun.aspx

     

     

    Thanks again for your posting!  If you have any questions, I’d love to discuss with you.

     

     

    Have a nice day!

     

     

    Best Regards,
    Lingzhi Sun

    MSDN Subscriber Support in Forum

    If you have any feedback on our support, please contact msdnmg@microsoft.com


    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    Welcome to the All-In-One Code Framework! If you have any feedback, please tell us.
    • Marked as answer by BBeattie Friday, November 13, 2009 3:24 AM
    Tuesday, November 10, 2009 9:35 AM
    Moderator
  • Hi,

    First of all, this forum is for Entity Framework questions, your question is about Linq to Objects and therefore would probably be moved from this forum sometime in the future.

    As to your question - the Where method doesn't execute the filter, it just creates a deffered query that will be executed once you call first/count, but note that each time you use the First & Count methods the where is re-evaluated - this is by design.

    If you want to keep the results in ds, use the following syntax :

    var ds = ld.Where(itm=>itm[1] == indx).ToList();

    Using the ToList will execute the where statement, and will create a list to hold the items, after that, and usage of first/count will be used on the created list and will not invoke the where method again
    Please mark posts as answers/helpful if it answers your question
    Tuesday, November 10, 2009 9:37 AM
  • Thanks for the explanation guys.

    Sorry about the wrong thread - I found this issue initially using LINQ to EF, and dumbed down the code to make it simpler to understand. 

    I had a look for a general LINQ forum but the "LINQ Project General" forum that this post is now in is in the ARCHIVED section of the forums. It does appear to be still active though - this is confusing to me.

    I understand the Where only constructs the query, however assumed (as most people would) that once the query is constructed, changing variable afterwards would not effect results. This is quite a philosophical difference when moving from SQL to ADO to LINQ.  I also did not realise First() would cause the query to re-evaluate - but it does make sense when thinking about it in a data driven sense.

    Lingzhi Sun - you are a legend. That explanation is awesome. Well done.

    Thanks again.

    Tuesday, November 10, 2009 9:16 PM
  • Variables in LINQ Lambda Expressions cause expression to re-evaluate dynamically if changed.
    Variables in LINQ Lambda Expressions cause expression to re-evaluate dynamically if changed.
    Variables in LINQ Lambda Expressions cause expression to re-evaluate dynamically if changed.Variables in LINQ Lambda Expressions cause expression to re-evaluate dynamically if changed.
    Users MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
    Wednesday, November 11, 2009 10:46 AM
  • Hi Brett,

     

    Thank you very much for your kind words!  J

     

    The LINQ Project General is still used and we are still supporting here for general LINQ to Objects issues. 

     

    BTW, may I modify this thread to a question?  The question thread in MSDN forums are easier to be searched, so that the information in this thread can benefit more community members.  

     

     

    Have a nice weekend!

     

     

    Best Regards,
    Lingzhi Sun

    MSDN Subscriber Support in Forum

    If you have any feedback on our support, please contact msdnmg@microsoft.com


    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    Welcome to the All-In-One Code Framework! If you have any feedback, please tell us.
    Friday, November 13, 2009 1:25 AM
    Moderator
  • no worries - done!
    Friday, November 13, 2009 3:26 AM
  • Thanks,  Brett!


    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    Welcome to the All-In-One Code Framework! If you have any feedback, please tell us.
    Friday, November 13, 2009 3:28 AM
    Moderator
  • Rahhe

    Janet just completed level 2 of Coffee mastery in FarmVille!
    Janet earned a huge reward for being such a dedicated farmer and wants to share their success with you!
    Friday, November 20, 2009 11:14 AM