EF Code First with RIA Services / a difficult combination....

Question

I am working on an application using both EF5 and RIA Services, and am about to abandon this stack and look elsewhere....

• 1 - On the server side things appears simple, and in my plain CLR object model, I am adding the “virtual”  keyword to my various collections in my domain classes allows entity framework at runtime to create dynamic proxies to allow me to support lazy loading of these navigation properties.  (If my understanding is correct, without virtual, lazy loading will not be supported period).
• 2 - Now, there also is the ability to decorate my domain class navigation properties with the [Include], which if I understand things correctly should force eager loading of the collection (server side).   I am not yet 100% sure this is the purpose of the[Include]

Now by adding RIA Services to the mix things start getting complicated and I am told this is a result of RIA development being out of sync with EF development.   I find myself adopting many workarounds just to get these two technologies to work together.

• 3. Once such example:  RIA Services seems to force me to decorate domain model navigation properties with the [Include] attribute just to allow me to properly load the collection on the client side.   (In other words if I don’t have the [Include] attribute on navigation properties (Server side), RIA's client side proxies won’t be able to load them).

If my understanding is true, points (2 & 3) taken together sound like a bind..... the way to get RIA to handle navigation properties properly (on the client side) is to decorate my domain classes with [Include] attribute, which will also as a result force eager loading of my collections (on the server side)

This sounds like a bind indeed, since for example, if I have a collection of 1000 ASSETS entities and each asset has a child collection representing its prices history LIST<ASSETPRICE> of 1000 entries, eager loading would mean loading 1 million objects into my context (server side) just to be able to access them on the client side.

Two technologies marketed as complimentary (and released by the same company), why is it so hard to get them to work together ?  I must be missing something.

HELP !

Answer 1

There are two different Includes involved here. One is the RIA Services IncludeAttribute ([Include]) and the other is the Include that is part of the EF query. They are two separate things.

The [Include] tells the RIA Services serializer that if the associated entity(s) exists in memory at the time of serialization that it should also be sent to the client. The [Include] acts as a security mechanism to make sure that entities don't leak to the client that are not supposed to and as a performance mechanism for the same reasons.

There is a second Include that goes inside the EF query itself that tells it to eager load the related entities as part of the query. Using your objects, that would look something like:

return DbContext.ASSETs.Include("ASSETPRICEs");

In that case, if the [Include] was not on the ASSETPRICEs property of ASSETT then the ASSETPRICEs would all be loaded into memory on the server but would not be sent on to the client. If the [Include] was there then they would be sent to the client. If the .Include("ASSETPRICEs") was not part of the query then the existance or absence of the [Include] will have no effect.

Going through your list of three points;

1. By default, RIA Services disables proxy generation and lazy loading for its DbContext, so the virtuals in your POCO objects have no effect on RIA Services. This is because lazy loading can be dangerous in a serialization situation even with the [Include] safety mechanism and has no benefit. The proxies are also not needed because change tracking is performed client side.
2. I know I am restating what I have already said, but the [Include] attribute is for RIA Services and just enables serialization of eager loaded related objects, it does not force the eager loading of those related objects.
3. RIA Services only creates client side entities for those entities that are exposed by the server. For an entity to be exposed it must either have a query method in the DomainService or be marked as being serialized along with an entity that does have a query method using the [Include]. For example, if you added a query method to get the ASSETPRICE objects from the DomainService and didn't have anything marked with the [Include] the navigation properties for related ASSETPRICE objects would still appear in ASSETS. Also, because RIA Services uses foreign keys the loading the ASSETPRICE and ASSETS entities through two separate loads would still result in the entities being related to each other client side, the objects match up automatically based on the foreign keys.

I am not sure what you are asking at the end of your message, if you want to load 1 million objects into your client then you would have to load them into your server first. If your complaint was that you only wanted to load the ASSETS entities and only load the ASSETPRICE entities that you need later, don't put the .Include("ASSETPRICE") in the query.

---

http://www.riaservicesblog.net | RIA Services and MVVM http://bit.ly/pgL97k

Answer 2

Dear Colin 

<I apologize for the long post, but I am still confused >.

Thank you so much for your detailed response and for clarifying some of my confusion.    Let me then recap what I understood to make sure I am using EF / RIA in the right way:

Entity Framework

When my server side code is accessing my domain model via EF directly (i.e. has a direct reference to the DbContext object) such as when I am writing the code inside the CRUD methods of the DomainServices), the following rules apply

1. If methods inside my domain classes are marked as virtual, EF will replace those methods with those of dynamically generated proxy classes that know how to perform lazy loading when I use navigation properties in my object model.   
2. Unless there is a compelling performance reason that requires me to eager-load those navigation properties by embedding .Include(“property-name”) in my LINQ queries, there seems to be no downside for me to mark all my navigation properties everywhere in my object model with ‘virtual’ (as ‘default’ setting).
3. Change tracking is done automatically for me when I have a direct reference to DbConext

Remark: there is no [Include] attribute that is for EF understands, EF only understands the .Include() extension method as part of a LINQ Query.  Is this right?

RIA Services – Server Side

When talking about the server side of RIA Services, various attributes that are needed in my POCO classes to help WCF RIA Service to work properly.   For example, for the code in DomainService methods), the following is true:

1. Foreign keys fields must be added to all reference properties of domain classes
2. [ForeignKey] must also be used to decorate those properties that do not follow naming convention
3. [Association] attributes should also be used to match foreign keys and primary keys taking part in a relationship in cases where multiple relationships exist between entities.  (RIA Services currently only supports foreign key associations.  Direct many-many relationships are not supported).
4. Any class hierarchy should have its parent node by marked with [KnownType] attribute to tell RIA Services about its subclasses, so that it can also track these subclasses on the client.
5. Any domain model entity that is not explicitly exposed via methods in my domain service should be decorated with the [Include] attribute which acts as an ‘opt-in’ security switch (to tell the RIA serializer to also transfer these entities) and MUST ALSO be eager loaded in the EF query via the .Inlcude() extension method. 
6. In other words, If I specify the [Include] attribute for an association, RIA Services will serialize the related objects to the client, but I would still be obliged to manually load these related objects prior to serialization. 

RIA Services – Client Side

Understanding this side of the stack is the most confusing for me. I am struggling most with learning how to exactly use my client side RIA Generated navigation properties to get to related data.

Here is what I understood:

1. In an earlier post, you had explained that serialization is incompatible with lazy loading since serializers need to know (at build time) the type they are transferring, while lazy loading is typically implemented via dynamically generated proxy classes, whose type is not known before runtime.
2. I also understand that for the reason above, LazyLoading should be disabled.  Does that mean setting the property of the Entity Framework context?   So using RIA Services will indirectly also force me to give up (the nice) lazy loading feature on the server?

In my case, I have a collection of ASSETS (exposed by Domain Service), and each ASSET has a child collection of ASSETPRICES.   What I need is to display the assets in VIEW on the client, and when the user clicks an asset to load those prices into some sort of chart control.

1. For performance reasons I don’t wish to load all the prices of all my assets at the same time.  I need RIA to make the first call to ASSETS, and only make a separate call to ASSETPRICES when user clicks a particular asset.
2. Since there is no way to implement lazy loading with WCF RIA Services, is my only option to expose two separate methods in my Domain Servcies as for example: GetAssets() and GetPricesforAsset(),  and call them myself at the right time?
3. If the point (2) above is true, wouldn’t that also mean that I would also be giving up the desirable benefit of having client side navigation properties which promised me ability to drill down from a parent to get to child collections ?
4. Where does the self-tracking feature of these client side classes fit into all of this?

I apologize for the long post, but I am still confused.

Answer 3

I want to make sure we are on the same page so lets set a baseline. You are using EF5 so you should have the RiaServices.EntityFramework NuGet package installed and you should be using the DbDomainService. Your CRUD methods in your DomainService should look something like the ones at http://varunpuranik.wordpress.com/2011/06/29/wcf-ria-services-support-for-ef-4-1-and-ef-code-first/

Entity Framework

1. Your server side code should be using the DbContext provided by the DbDomainService. That instance of the DbContext will have lazy loading and proxy generation turned off. If you directly construct an instance of your DbContext then it would have change tracking and proxies enabled.
2. There is never any downside to using the Virtual on your properties.
3. Since the proxies are turned off, automatic change tracking is not enabled. If you want to modify the entities server side outside of what RIA Services already done then you would need to override the PersistChangeset method of the DomainService and manually call DbContext.ChangeTracker.DetectChanges before calling the base.PersistChangeset.
4. Correct on the last remark there

RIA Services - Server Side

1. Correct
2. Either the attribute or using the fluent configuration.
3. Association should not be needed for Entity Framework. The DbDomainService will extract the association information from EF for you. You would only need association if you were using a PresentationModel entity (i.e. a POCO object not tracked by the DbContext).
4. Correct, this is required for any type of serialization.
5. Generally yes. The related entities just have to be there at serialization time to be sent to the client so there are tricks you can play other than the .Include, but for our purposes yes should suffice.
6. Correct

RIA Services - Client Side

1. Correct, almost forgot about that post. That is the general reason why all services should turn off proxies and lazy loading, another reason is that during serialization you could end up accidentally serializing the entire database as each property is followed. That is mostly avoided in RIA Services because of the [Include] but it can still happen with [Include] in place.
2. In Silverlight (and WinRT for that matter) all communications with the server are asynchronous. This means that it is impossible to call currentAsset.AssetPrices and have it lazy load the entities from the server before returning your call. The best RIA Services could have done is on the first call to currentAsset.AssetPrices an empty collection would return and then at some point in the future currentAsset.AssetPrices would be populated once the server replied. That would not be a good design. This is not really a RIA Services specific limitation.

Last Section:

1. OK
2. Yes, a GetAssets and a separate GetPricesForAsset would be the best solution.
3. Yes and no, you can't directly call currentAsset.Prices and have it auto-fill from the server. However, after you have called GetPricesForAsset the currentAsset.Prices collection will be populated and you can drill down into it from that point on.
4. I am not sure I understand that question. Every entity knows its current values, its original values, and if it is Unmodified, Detached, New, Deleted, or Modified.

OK, after all of that lets try answering your question. First, there is a link in my signature about MVVM and RIA Services. That link takes you to best practices on how to architect your application. It is a bit.ly bundle that starts with a John Papa video and then continues through some refinement added by Kyle McClellan (former member of the RIA Services team at Microsoft who currently works for XBox) and myself.

The client side entities are marked as Partial classes so you can extend them with your own Partial class. I would extend Asset like this:

partial public class Asset
{
    bool _assetPricesLoaded = false;
    public bool AssetPricesLoaded {get{return _assetPricesLoaded;}}
    public void SetAssetPricesLoaded()
    {
        _assetPricesLoaded = true;
        RaisePropertyChanged("AssetPricesLoaded");
    }
}

The reason that AssetPricesLoaded is not just a read only property is to prevent the deserializaer from overwriting it during deserialization. It is a behavior I am planning to change for Open RIA Services but we are stuck with it for now.

In my ViewModel, I would have a CurrentAsset and a CurrentAssetPrices property exposed. The CurrentAssetPrices property would look something like:

public IEnumerable<AssetPrice> CurrentAssetPrices
{
    get
    {
        if (currentAsset == null)
            return null;
        else
        {
            if (!currentAsset.AssetPricesLoaded)
            {
                //The following code would be in my DataService class and but putting it here for simplicity
                myContext.Load(myContext.GetPricesForAsset(currentAsset.AssetId);
            currentAsset.SetAssetPricesLoaded;
            }
            return currentAsset.AssetPrices;
        }
    }
}

Now, the actual Load would need error checking and if it failed I would need to unset the SetAssetPricesLoaded so that the UI could try and reload it, and stuff like that but I wanted to get your the general shape of the code. The EntityCollection returned by currentAsset.AssetPrices implements INotifyCollectionChanged so once the prices loaded from the server the UI would automatically update itself to display the prices.

---

http://www.riaservicesblog.net | RIA Services and MVVM http://bit.ly/pgL97k

Answer 4

Dear Colin,

Yes, I am using EF 5 and the DbDomainService as outlined in the article you mentioned, with the only difference being that in my case, I am not checking the condition (HttpContext.Current == null) prior to invoking Database.     Does that make any difference?

Entity Framework

1. In my solution, I have a loosely coupled design and have additional two layers between EF and my DbDomainService. 
   1. The first layer above EF is a DataAccessLayer which implements the repository pattern and it constructs the DbContext directly.
   2. The second layer is the BusinessLayer and just has various validation checks and business rules.
   3. I suppose that means, in my case that “change tracking” and “proxies” are enabled.    Is that alright?
2. OK there is no down side, but after reading the entire email below, we can conclude that it does not add any value either since proxy classes on the server side will need to be turned off?
3. OK
4. OK

RIA Services – Server Side

1. OK
2. OK
3. I suspect the [Association] attribute may still needed in cases where multiple relationships exist are in place between two entities, and these are not bidirectional associations.   For example: a ForeignExchangeDeal may reference two CURRENCIES (PrincipalCurrency and CounterCurrency).
4. OK
5. OK
6. OK

RIA Services – Client Side

1. OK
2. So, having a web services enabled solution, always means giving up the lazy loading features of an ORM technology?   Put differently, the magic of EF proxy classes would only be useful in a single tier (example: WPF application)?

Last Section

1. OK
2. OK
3. I am confused….
   1. You mention “…you can't directly call currentAsset.Prices and have it auto-fill from the server….”.   I thought the who ‘magic’ of expression trees and deferred execution.  The IQeryable interface was supposed to precisely solve that issue and allow me via the EntityQuery<T> to shape the expression tree before sending it to the server for execution.

 

1. You mention …” after you have called GetPricesForAsset the currentAsset.Prices collection will be populated and you can drill down into it from that point on….”.  

How exactly does RIA know to which ASSET child collection, the result-set from calling   GetPricesForAsset is to go.   I understand a replica of my domain model relationships exist on the client, but how does RIA figure out that the prices being retrieved via another service method should go into a particular instance of the asset (the one selected) don’t I explicitly need to populate that child collection myself ?

1. Change Tracking… let me paraphrase my initial question…..
   1. In my mind “Change Tracking” (server side) referred to the ability for EF to track all changes made by a user to an object graph and then be able persist chose changes when SubmitChanges() is called.
   2. So, it would follow that when proxy classes are turned off on the server side, then EF is not able to track changes.
   3. RIA Services, adds the ability to track changes on the client via its DomainContext class and this is completely unrelated to EF’s ability to track changes, which would normally be turned off.

Am I correct?

Your Example

I will check-out the best practices outlined in the link you sent me.   Upon reading the code, I think it sort of responds to the question I pose in the preceding section 3.1 and 3.b in respect of how exactly RIA is able to figure out to what ASSET instance the separately retrieved prices should go…..

The answer it appears from my reading your code, is that RIA does NOT know (and probably CAN NOT know), and therefore should be handled by my code, which loads the child collection in the same block of code as loading the asset manually (as in the sample code you provided).   Am I correct?

I apologize for covering many inter-related issues in a single post, but getting a holistic view of this stack and understanding these interactions is precisely what I am struggling with.

Answer 5

Entity Framework

Yes, that probably means that proxies and lazy loading are turned on and you should be turning it off. There are two general ways to architect your repositories. What I would recommend is not having your repository construct the DbContext, instead let the DomainService manage the DbContext and pass it into your repository. The second way of using repositories is to use the base DomainService and decorate it with the [DbDomainServiceDescriptionProvider]. In that case you have to manage the DbContext yourself.

What you need to understand is how the SubmitChanges works. When you call SubmitChanges all of the changes on the client are packaged together and sent to the DomainService through the Submit method. The DomainService then loops through the ChangeSet checking security and validation before calling your CUD methods. Each CUD method's job is to move the entity and its change state from the ChangeSet into the DAL. The database is not supposed to be updated in the CUD methods. Once all of the CUD methods have been called then the DomainService calls PersistChangeset. That is where the DAL is told to update the database. Using a plain DomainService, you would need to override the PersistChangeset and tell your repositories to update the database at that time. The CUD methods of the DomainService are not called in any particular order, it is expected that the DAL itself doesn't care in what order entities are attached to it and that the DAL will update the database in the correct order during PersistChangeset.

The big piece of functionality you lose by not letting the DbDomainService manage the DbContext and the PersistChanges is handling of concurrency errors. The DbDomainService marks entities in the ChangeSet that failed concurrency checking and places a copy of the current database version of the entity in the ChangeSet as well to send back to the client. That lets you write client side code to resolve concurrency errors, using the plain DomainService you will not have that code.

RIA Services Client Side

Correct, lazy loading is really only useful in a single tier situation.

Last Section

3: The entities have no path back to the DomainContext, there is a separation that occurs there to prevent server access code from leaking from one layer of the client side code to another. So, nothing you do to an entity can trigger data to load from the server, but once an outside force like a ViewModel triggers the load of data from the server then the data will appear.

Which gets into your next question, as I have said before RIA Services is based on relational data and uses foreign keys. The general concept of how it works is exactly how your original database works. The AssetPrice has an AssetId which matches to the primary key of the Asset entity. Each Asset entity has an EntityCollection<AssetPrice> object that acts sort of like a view into the EntitySet<AssetPrice> of every AssetPrice object that matches based on the foreign key. Each AssetPrice object has an EntityRef which does the reverse to find the matching Asset. In a one to many relationship, the object on the One side has an EntityCollection an the entity on the many side has an EntityRef object.

So, when you loaded additional data into the DomainContext the Assets get notified that there were changes in the EntitySet and those get refiltered into the various EntityRef/EntityCollections. This same process happens even when you eager load by the way, what gets sent back to the client from the server is just a loose collection of all of the entities that were loaded filtered to make sure there are no duplicates. All of the relationships are rebuilt on the client based on the foreign keys. This is one of the greatest strengths of RIA Services, how data was loaded by the DomainContext does not matter to how the data works. The only thing that matters is if the data was loaded at all.

---

http://www.riaservicesblog.net | RIA Services and MVVM http://bit.ly/pgL97k

Answer 6

Dear Colin,

I conclude by relisting my 'lessons learned' from our over-all discussion in this thread

Just to be 100% sure, I got it all, would it be possible to confirm the correctness of if the underlined phrases are 100% correct ?

1. In a RIA enabled solution using EF, there nothing to be gained by marking methods in my domain classes with the keyword virtual and therefore this should not done (while using virtual will not have any side effects, it is likely to add to the confusion).  Note: virtual would normally turn on lazy loading on the server, which as we’ve discussed will not happen.
2. Any server side code should be using the DbContext provided by the DbDomainService. That instance of the DbContext will have lazy loading and proxy generation turned off.   We should architect the repository in a why so that the DbContext is not created by it.  Instead  we should let the DomainService manage the DbContext and pass it into your repository
3. In the domain model, any class hierarchy should have its parent node by marked with [KnownType] attribute to tell RIA Services about its subclasses, so that it can also track these subclasses on the client
4. Any domain model entity that is not explicitly exposed via methods in domain service should be decorated with the [Include] attribute which acts as an ‘opt-in’ security switch (to tell the RIA serializer to also transfer these entities).  If the entity is directly exposed via domain service using any CRUD methods, then there is no need to use the [Include] attribute.  It is therefore only necessary to use the [Include] attribute in the case where that entity is not also directly exposed in the Domain Service
5. My DomainService code (or lower layers like repository etc.) in doing data loading, must always explicitly eager load any child collections needed on the client via the .Inlcude() extension method.    
6. RIA Services only creates client side entities for those entities that are exposed by the server. For an entity to be exposed it must either have a query method in the DomainService or be marked as being serialized along with an entity that does have a query method using the [Include].
7. The DomainService does not send serialized object graphs to the client, but rather just a ‘loose’ (flat) collection of all the entities loaded (to remove duplicates).   This loose collection of things is rebuilt client side by the DomainContext to create the object graph.  

Answer 7

1. Yes, although there is no problem leaving the virtuals in place. This is important because if you are doing database first and then using EF's reverse engineer wizard to generate the initial entities the virtuals will be in the generated entities and nobody should feel like they need to strip them out.
2. Yes
3. Yes, if you are using inheritance and you want that inheritance on the client side as well.
4. ...or if you want to eager load the entity.
5. ....if you want to eager load. Sidebar on this at the end.
6. Yes. This is also true for inheritance as well. If you have three classes inheriting one from another  (A, B:A, C:B) but you only expose entities A and C  then B will be flattened into C and on the client you would only see A and C with C inheriting A.
7. Yes

Number 5 sidebar:

The actual rule is that the entities need to be in memory to be serialized with the [Include]. Because EF doesn't have any ability to filter the .Include it sometimes makes sense to manually eager load the entities. In doing this, it is hard to make the result IQueryable so we tend to use an IEnumerable return with the filter in the parameters. Here is an example:

IEnumerable<Person> GetPeopleWithMailingAddress(int companyId)
{
    List<Address> tempAddresses = DbContext.Address.Where(a=>a.AddressType=="M" && a.Person.CompanyId == companyId).ToList()); 
    return DbContext.Persons.Where(p=>p.CompanyId == companyId);
}

I use the list for illustration, all you really have to do is make sure that the query returning addresses is enumerated. That will load the addresses into the DbContext so when the Persons are returned the address will be there in memory and be serializaed to the client as long as the [Include] is there.

One thing I missed way up at the beginning of the thread, M2M support is available through the M2M4RIA project on codeplex. We are planning to add direct support for M2M in Open RIA Services, however I personally prefer having explicit linking tables.

---

http://www.riaservicesblog.net | RIA Services and MVVM http://bit.ly/pgL97k

Answer 8

@Colin: You give a strong motivation for letting the DomainService manage the DbContext and not the repositories. I'm using the repository pattern and use dependency injection to inject my dbcontext in the repositories. Would this be possible as well when DomainService is managing the DbContext?

Answer 9

The DbContext has a protected virtual method named CreateDbContext. If you override the CreateDbContext and return the same DbContext that the repositories use then the DbDomainService will use that DbContext as its own. However, the DbDomainService will assume that it is the owner of the DbContext and when the DbDomainService is disposed of it will dispose the DbContext as well so keep that in mind. The important thing is just to make sure that the repositories are not trying to update the database before PersistChangeset is called.

The code that turns off lazy loading and proxy generation is what calls the CreateDbContext so those will be set for you. Long term as part of Open RIA Services I want to break the Context setup code and the concurrency handling into a separate public utility class to make it easier to use that logic in a plain DomainService.

---

http://www.riaservicesblog.net | RIA Services and MVVM http://bit.ly/pgL97k