none
EF6.1 Proxy off, virtual ICollection<T> still loading a lot of data RRS feed

  • Question

  • Hi there,

    I'm probably missing something basic, but I thought that if you turned lazyloading off and/or disabled proxy creation that you would HAVE to .Include("Entity")...

    However I have elements recursively loading:

    eg:

    MyObject.someproperty = "Bob" -- expected
    MyObject.listOfThings = <lots of things> -- expecting empty unless I called "Include("Things")

    do I need to also remove the virtual or stop using the ICollection (left overs from when I was trying to use proxy and lazyloading...)? (and if so why?)


    - sure I'm noJedi but that's no reason to stop trying to make stuff levitate! -

    Wednesday, April 15, 2015 4:10 PM

Answers

  • >This is what I'm seeing/stuck on:

    Oh,  that's just because that item has the same Location as its HomeLocation and CurrentLocation.  So the DbContext fixed up the reference.

    If you change the data so that the items have a different home and current location, it won't load.

        using (var db = new Db())
                {
                    db.Database.Log = (s) => Console.WriteLine(s);
    
                    foreach (var locName in new[] { "A", "B", "C" })
                    {
                        var loc = db.Locations.Create();
                        loc.Name = locName;
                        db.Locations.Add(loc);
    
                        foreach (var i in Enumerable.Range(1, 10))
                        {
                            var item = db.Items.Create();
                            item.HomeLocation = loc;
                            db.Items.Add(item);
                        }
                    }
    
                    var newLocation = db.Locations.Create();
                    newLocation.Name = "D";
                    db.Locations.Add(newLocation);
                    foreach (var i in db.ChangeTracker.Entries<Item>())
                    {
                        i.Entity.CurrentLocation = newLocation;
                    }
                    db.SaveChanges();
    
                }

    David


    David http://blogs.msdn.com/b/dbrowne/

    • Marked as answer by noJedi Friday, April 24, 2015 5:40 AM
    Saturday, April 18, 2015 2:14 PM
  • >Makes it difficult for me to trust things like AutoMapper NOT to traverse all objects and take an awful lot more data from my "source" object to my "target" than I had intended

    You just turn of Proxy generation (and hence Lazy Loading), and keep your DbContext lifetime short to control this.

    >does this also mean that if I have a LIST in something that has a reference to an ITEM that has been loaded then I could get a LIST with 1 items in it when in actuality that list might REALLY (in DB) have 2 or more items

    Interesting question.  I'll try to get a more definitive answer, but in my testing, no.  EG if you turn off LazyLoading and explicityly load "HomeLocation" then item.HomeLocation.HomeItems still be empty.

    >So what, now I have to carefully construct my test cases based on the relationships in data...?

    >What is the recommended process/pattern to cope with this?!

    Scope your DbContext to a single unit-of-work.  So a test case would bound the lifetime of a DbContext instance. 

    David


    David http://blogs.msdn.com/b/dbrowne/

    • Marked as answer by noJedi Friday, April 24, 2015 5:40 AM
    Sunday, April 19, 2015 1:07 PM
  • Okay so my actual issue turns out to be multifaceted (damn you real world scenarios!!!).

    So just to outline this for any other poor bugger struggling with this:

    • If you are "taking your data elsewhere" then definately go with davids advice and limit things by not using proxy/lazy loading and keeping your context lifetime as short as you possibly can.
    • be careful about use of automapping tools - because they use reflection you can get more than you bargain for and see my next point
    • be sure (unit tests perhaps?) that you "INCLUDE stuff you USE and only USE stuff you INCLUDE", because although David thinks this isnt' teh case from what I've seen you will get ALL REFEERENCES of an ENTITY populating things if it fits the model... this means you COULD get partial lists of things being populated even if you don't load things - this might depend on how your POCOs are initalized... (eg: ctor news up Hash<item>) and might not be an issue for DB first or if you let EF do the ICollection thing on its own... I'm not sure as I haven't played around with it enough...

    Cheers to David and Fred for input and happy coding!


    - sure I'm noJedi but that's no reason to stop trying to make stuff levitate! -

    • Marked as answer by noJedi Friday, April 24, 2015 5:40 AM
    Friday, April 24, 2015 5:39 AM

All replies

  • Hello noJedi,

    >>do I need to also remove the virtual or stop using the ICollection (left overs from when I was trying to use proxy and lazyloading...)? (and if so why?)

    It should be what you want by disabling the LazyLoadingEnabled as “db.Configuration.LazyLoadingEnabled = false;”, however, from your side, it is strange that it fails to work. For helping you check what causes this scenario, please tell us information as:

    What database you are using?

    How do you code your tables? If possible, please share it with us.

    Please also share code which is used to load the data from database in your project.

    Here I made a sample which shows it could work by disabling the LazyLoadingEnabled only, you could try it to see if it could work as you except:

    Model:

    public partial class Order
    
        {
    
            public Order()
    
            {
    
                this.OrderDetail = new HashSet<OrderDetail>();
    
            }
    
        
    
            public int OrderID { get; set; }
    
            public string OrderName { get; set; }
    
        
    
            public virtual ICollection<OrderDetail> OrderDetail { get; set; }
    
        }
    
    public partial class OrderDetail
    
        {
    
            public int ODID { get; set; }
    
            public string ODName { get; set; }
    
            public int OrderID { get; set; }
    
        
    
            public virtual Order Order { get; set; }
    
        }
    

    Query:

    using (DFDBEntities db = new DFDBEntities())
    
                {
    
                    db.Configuration.LazyLoadingEnabled = false;
    
                    var MyObject = db.Order.FirstOrDefault();
    
                    
    
                }
    

    Result:

    Regards.


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Thursday, April 16, 2015 3:22 AM
    Moderator
  • Hmmm... well thanks for your info. I have figured out the cause, but now I'm stuffed if I know what to do about it... any advice would be appreciated.

    I think "Include()" doesn't really work the way my mental model of it was rigged.

    the above is correct and lazy loading off does "turn off" most of the data, as you've implied.

    However I think we've shot ourselves in the foot in this project by the way we've constructed the model.

    This is a snippet of it:

    Items > Locations(Office/Countries)

    Items belong to (and have both) Current Locations and Home Locations

    Items belong to locations. and have lists associated with them...

    So if I have lazy loading OFF and I say:

    cxt.Items.First();

    then I get

    Name=Item1, CurrentLocationId = 1, CurrentLocation = NULL, HomeLocationId = 2, HomeLocation = NULL

    no problem...

    however (you can probably see where this is going already):

    if EITHER lazyloading is off OR I hit "Include("Locations")"

    then ALL CHAINS of the Locations table are retrieved.

    So Items will have current and home locations loaded, and each location will have its list of ITEMS loaded...

    I'm guessing this is SOMEHOW by design... So what are you supposed to do about this design... are you supposed to keep your lists "one way" (ie somehow heirachically important) and just have a separated funtionality when you want to get "Items listed at a location" (or vice versa if locations are my important)?

    UPDATE: so it seems to load all the info into the CONTEXT and so if its there, then its populated in the object tree... I think this causes the same issue as is illustrated by the DataContractSerializer and I think that uses the [IgnoreDataMember] attribute to stop recursive type stuff... maybe I shoudl use something like that... rather than mapping tools I'm currently using to copy objects around...?


    - sure I'm noJedi but that's no reason to stop trying to make stuff levitate! -


    • Edited by noJedi Friday, April 17, 2015 9:11 AM
    Friday, April 17, 2015 8:54 AM
  • >if EITHER lazyloading is off OR I hit "Include("Locations")" then ALL CHAINS of the Locations table are retrieved.

    No.  If LazyLoading is off, then navigation will just fail.  If you Include("Locations"), that will fail because you "include" navigation properties, not entity types.  You would need to Include("HomeLocation") or Include("CurrentLocation").

    >UPDATE: so it seems to load all the info into the CONTEXT and so if its there, then its populated in the object tree...

    Yes.  The DbContext caches entities and will "fix up" entity graphs if it already has the related entity. 

    Here's a little sample:

    using System;
    using System.Collections.Generic;
    using System.ComponentModel.DataAnnotations;
    using System.Data.Entity;
    using System.Data.Entity.ModelConfiguration.Conventions;
    using System.Linq;
    using System.Threading.Tasks;
    
    namespace eftest
    {
    
    
        public class Location
        {
            public Location() 
            {
                CurrentItems = new HashSet<Item>();
                HomeItems = new HashSet<Item>();
            }
            public int ID { get; set; }
            public string Name { get; set; }
    
            public virtual ICollection<Item> CurrentItems { get; set; }
            public virtual ICollection<Item> HomeItems { get; set; }
        }
        public class Item
        {
            public int ID { get; set; }
            public string Name { get; set; }
            public int CurrentLocationID { get; set; }
            public int HomeLocationID { get; set; }
    
            public virtual Location HomeLocation { get; set; }
            public virtual Location CurrentLocation { get; set; }
    
        }
    
        class Db : DbContext
        {
    
            public DbSet<Location> Locations { get; set; }
            public DbSet<Item> Items { get; set; }
    
            protected override void OnModelCreating(DbModelBuilder modelBuilder)
            {
                modelBuilder.Conventions.Remove<OneToManyCascadeDeleteConvention>();
                base.OnModelCreating(modelBuilder);
            }
    
        }
    
    
        class Program
        {
    
            static void Main(string[] args)
            {
                Database.SetInitializer(new DropCreateDatabaseAlways<Db>());
    
                using (var db = new Db())
                {
                    db.Database.Log = (s) => Console.WriteLine(s);
    
                    foreach (var locName in new[] {"A","B","C"})
                    {
                        var loc = db.Locations.Create();
                        loc.Name = locName;
                        db.Locations.Add(loc);
    
                        foreach (var i in Enumerable.Range(1,10))
                        {
                            var item = db.Items.Create();
                            item.HomeLocation = loc;
                            db.Items.Add(item);
                        }
                    }
    
                    var LocA = db.ChangeTracker.Entries<Location>().Where(l => l.Entity.Name == "A").Single().Entity;
                    foreach (var i in db.ChangeTracker.Entries<Item>())
                    {
                        i.Entity.CurrentLocation = LocA;
                    }
                    db.SaveChanges();
    
                }
    
                using (var db = new Db())
                {
                    db.Database.Log = (s) => Console.WriteLine(s);
    
                    //eagerly loads HomeLocation but not CurrentLocation
                    //and does not eagerly load any of HomeLocations other Items.
                    var item = db.Items.Include("HomeLocation").First(); 
                    /*
        SELECT TOP (1)
        [Extent1].[ID] AS [ID],
        [Extent1].[Name] AS [Name],
        [Extent1].[CurrentLocationID] AS [CurrentLocationID],
        [Extent1].[HomeLocationID] AS [HomeLocationID],
        [Extent2].[ID] AS [ID1],
        [Extent2].[Name] AS [Name1],
        [Extent1].[Location_ID] AS [Location_ID],
        [Extent1].[Location_ID1] AS [Location_ID1]
        FROM  [dbo].[Items] AS [Extent1]
        INNER JOIN [dbo].[Locations] AS [Extent2] ON [Extent1].[HomeLocationID] = [Extent2].[ID]
    
                     * */
    
                }
    
               
                using (var db = new Db())
                {
                    db.Database.Log = (s) => Console.WriteLine(s);
                    db.Configuration.LazyLoadingEnabled = true;
                    db.Configuration.ProxyCreationEnabled = true;
    
                    var item = db.Items.First();
                    /* runs
        SELECT TOP (1)
        [c].[ID] AS [ID],
        [c].[Name] AS [Name],
        [c].[CurrentLocationId] AS [CurrentLocationId],
        [c].[HomeLocationId] AS [HomeLocationId],
        [c].[Location_ID] AS [Location_ID],
        [c].[Location_ID1] AS [Location_ID1]
        FROM [dbo].[Items] AS [c]
                    
                     * */
                    Console.WriteLine(item.GetType().Name);
                    var homeLocationName = item.HomeLocation.Name;  //Lazy Loads the HomeLocation
                    /* 
        SELECT
        [Extent1].[ID] AS [ID],
        [Extent1].[Name] AS [Name]
        FROM [dbo].[Locations] AS [Extent1]
        WHERE [Extent1].[ID] = @EntityKeyValue1
                     
                     
                     */
    
                    var otherItems = item.HomeLocation.HomeItems; ////Lazy Loads the HomeLocation's HomeItems
                    /* runs
        SELECT
        [Extent1].[ID] AS [ID],
        [Extent1].[Name] AS [Name],
        [Extent1].[CurrentLocationID] AS [CurrentLocationID],
        [Extent1].[HomeLocationID] AS [HomeLocationID],
        [Extent1].[Location_ID] AS [Location_ID],
        [Extent1].[Location_ID1] AS [Location_ID1]
        FROM [dbo].[Items] AS [Extent1]
        WHERE ([Extent1].[Location_ID1] IS NOT NULL) AND ([Extent1].[Location_ID1] = @EntityKeyValue1)
                     * */
                }
            }
    
    
          
                
        }
    }

    David


    David http://blogs.msdn.com/b/dbrowne/




    Friday, April 17, 2015 4:09 PM
  • Hi David... okay lets say I believe you ;) - I should have been more careful with my contrived example because yes you need to specify the actual property, however...

    Either I'm misunderstanding something essential or there is something odd going on...

    This is what I'm seeing/stuck on:

                using (var db = new Db())
                {
                    db.Database.Log = (s) => Console.WriteLine(s);
                    db.Configuration.LazyLoadingEnabled = false;
                    db.Configuration.ProxyCreationEnabled = true;
    
                    var item = db.Items.Include("HomeLocation").First();
    
    //item.CurrentLocation is a proxy and is populated with data...
                    Console.WriteLine(item.GetType().Name);
    
    AND...
    
    
    
                using (var db = new Db())
                {
                    db.Database.Log = (s) => Console.WriteLine(s);
                    db.Configuration.LazyLoadingEnabled = false;
                    db.Configuration.ProxyCreationEnabled = false;
    
                    var item = db.Items.Include("HomeLocation").First();
    
    //item.CurrentLocation is a NOT a Proxy and is populated with data...
                    Console.WriteLine(item.GetType().Name);

    - sure I'm noJedi but that's no reason to stop trying to make stuff levitate! -

    Saturday, April 18, 2015 1:29 PM
  • >This is what I'm seeing/stuck on:

    Oh,  that's just because that item has the same Location as its HomeLocation and CurrentLocation.  So the DbContext fixed up the reference.

    If you change the data so that the items have a different home and current location, it won't load.

        using (var db = new Db())
                {
                    db.Database.Log = (s) => Console.WriteLine(s);
    
                    foreach (var locName in new[] { "A", "B", "C" })
                    {
                        var loc = db.Locations.Create();
                        loc.Name = locName;
                        db.Locations.Add(loc);
    
                        foreach (var i in Enumerable.Range(1, 10))
                        {
                            var item = db.Items.Create();
                            item.HomeLocation = loc;
                            db.Items.Add(item);
                        }
                    }
    
                    var newLocation = db.Locations.Create();
                    newLocation.Name = "D";
                    db.Locations.Add(newLocation);
                    foreach (var i in db.ChangeTracker.Entries<Item>())
                    {
                        i.Entity.CurrentLocation = newLocation;
                    }
                    db.SaveChanges();
    
                }

    David


    David http://blogs.msdn.com/b/dbrowne/

    • Marked as answer by noJedi Friday, April 24, 2015 5:40 AM
    Saturday, April 18, 2015 2:14 PM
  • Okay...

    So does this also explain why lists are also populated in a similar manner - ie: if the context has already loaded a list of "Items" elsewhere, does my Items object CurrentLocation then have a "Items" list that is populated?!

    Makes it difficult for me to trust things like AutoMapper NOT to traverse all objects and take an awful lot more data from my "source" object to my "target" than I had intended... does this also mean that if I have a LIST in something that has a reference to an ITEM that has been loaded then I could get a LIST with 1 items in it when in actuality that list might REALLY (in DB) have 2 or more items, but they just haven't been loaded?! (I hope not, but now I'm so worried that I'm going to have to test this is not the case...!!!) -- UPDATE -- Holy @#%# this is exactly what happens... sigh... so now I really need to ask WTF and put lots of asterisks next to the key point in my question... see below... *sniff*

    WHY EF designers have done this makes sense to me (from an architectural view), but also seems dangerous... Dangerous as in potential for "weird bugs" to surface because the behaviour of a bit of code is now tied to the relationships in the data.

    *******************************************************************************

    So what, now I have to carefully construct my test cases based on the relationships in data...?

    What is the recommended process/pattern to cope with this?!

    *******************************************************************************


    - sure I'm noJedi but that's no reason to stop trying to make stuff levitate! -



    • Edited by noJedi Sunday, April 19, 2015 11:44 AM Shock horror revelation that NOTHING is as I thought... thanks Dave... ;P
    Sunday, April 19, 2015 6:26 AM
  • >Makes it difficult for me to trust things like AutoMapper NOT to traverse all objects and take an awful lot more data from my "source" object to my "target" than I had intended

    You just turn of Proxy generation (and hence Lazy Loading), and keep your DbContext lifetime short to control this.

    >does this also mean that if I have a LIST in something that has a reference to an ITEM that has been loaded then I could get a LIST with 1 items in it when in actuality that list might REALLY (in DB) have 2 or more items

    Interesting question.  I'll try to get a more definitive answer, but in my testing, no.  EG if you turn off LazyLoading and explicityly load "HomeLocation" then item.HomeLocation.HomeItems still be empty.

    >So what, now I have to carefully construct my test cases based on the relationships in data...?

    >What is the recommended process/pattern to cope with this?!

    Scope your DbContext to a single unit-of-work.  So a test case would bound the lifetime of a DbContext instance. 

    David


    David http://blogs.msdn.com/b/dbrowne/

    • Marked as answer by noJedi Friday, April 24, 2015 5:40 AM
    Sunday, April 19, 2015 1:07 PM
  • Yes, I think ditching proxys is definitely the way to go here... thanks for clarifying.

    Possibly my model is more complex than initially implied... and that may be the difference

    Lets add another entity "Owner"

    Item.CurentLocation

    Item.Owner.AvailableLocations

    However from my testing I'm definitely seeing two issues:

    a) I'm definitely seeing limited lists returned - as implied in my "interesting question"

     - eg: db.Items.Include(Owner).Include(CurrentLocation).ToList();
    
    I see item1 has ownerx with list of available locations being highly populated.
    
    vs 
    
    db.Items.Include(Owner).Include(CurrentLocation).Take(2).ToList();
    
    I see item1 has ownerx with list of only a few locations (I think) because "current location" only has two elements loaded, because of the take)

    b) even with proxies and lazy loading off I'm seeing "tolist()" create recursive structures because "an ITEM has a current LOCATION which belongs to an OWNER, which has a list of Locations which have a list of ITEMS and so on..."

    THIS IS A PROBLEM (for me at least) because if I put this code into a webservice and want to simply return the object graph as a block, then the quantity of data is almost always more than I want it to be.

    eg:  I may want items > owners > locations, but I dont' want the LIST OF AVAILBLE locations with this "smart loading" of existing items, if I load all items, I will get a VERY large serialized set.

    eg: I may want this:

    item1
    --Owner1
    item2
    --Owner2
    item3
    --Owner2
    item4
    --Owner1

    but with either Proxy or not if I get all items, include Owner I would get this serialized...

    item1
    --Owner1
    ----item1
    ----item2
    item2
    --Owner2
    ----item2
    ----item3
    ----item4
    item3
    --Owner2
    ----item2
    ----item3
    ----item4
    item4
    --Owner1
    ----item1
    ----item2

    worst is when I get a list that ISN'T all of the ITEMS... eg:

    item3
    --Owner2
    ----item3
    ----item4
    item4
    --Owner1





    • Edited by noJedi Thursday, April 23, 2015 4:31 PM
    Sunday, April 19, 2015 1:42 PM
  • Okay so my actual issue turns out to be multifaceted (damn you real world scenarios!!!).

    So just to outline this for any other poor bugger struggling with this:

    • If you are "taking your data elsewhere" then definately go with davids advice and limit things by not using proxy/lazy loading and keeping your context lifetime as short as you possibly can.
    • be careful about use of automapping tools - because they use reflection you can get more than you bargain for and see my next point
    • be sure (unit tests perhaps?) that you "INCLUDE stuff you USE and only USE stuff you INCLUDE", because although David thinks this isnt' teh case from what I've seen you will get ALL REFEERENCES of an ENTITY populating things if it fits the model... this means you COULD get partial lists of things being populated even if you don't load things - this might depend on how your POCOs are initalized... (eg: ctor news up Hash<item>) and might not be an issue for DB first or if you let EF do the ICollection thing on its own... I'm not sure as I haven't played around with it enough...

    Cheers to David and Fred for input and happy coding!


    - sure I'm noJedi but that's no reason to stop trying to make stuff levitate! -

    • Marked as answer by noJedi Friday, April 24, 2015 5:40 AM
    Friday, April 24, 2015 5:39 AM