none
Implement IDataServicePagingProvider?

    שאלה

  • I am trying to determine if I need to implement IDataServicePagingProvider. My implementation required creating an untyped custom data provider as is described in this article (http://msdn.microsoft.com/en-us/library/ee960143.aspx). I am running into scalability issues because the data set that I need to construct whenever

         protected override DSPContext CreateDataSource()

    is called is a very large data set. The default paging works (e.g. calling SetEntitySetPageSize), however it is applied after CreateDataSource has already been called. Is there a good way to improve the performance of the CreateDataSource method by not having it create a "complete" data source and instead utilize paging?

    Thanks

    Cory

    יום שלישי 28 פברואר 2012 01:16

תשובות

  • Hi,

    IDataServicePagingProvider won't help you make your CreateDataSource better. I think the problem lies in that you're likely doing to much in the CreateDataSource.

    First a comment on the sample custom provider you're using as the bases of your code (DSPContext is the name of the context class in that sample provider): It is just a sample, and to simplify it's code it creates/gets all the data in the CreateDataSource, but that is not what real world providers would do.

    The general design should look like this:

    1) CreateDataSource simply creates a representation of the data source. For example, if your underlying data source is a database, then this would just open a connection to that database (login and so on), but it would NOT load any data from it.

    2) The IDataServiceQueryProvider which you implement is called to GetQueryRootForResourceSet. From this method you return a custom implementation of IQueryable for the specified resource set. Again, this object is just a description of the resource set, it doesn't load any data yet. (In the DB case this would return IQueryable with an expression which is the representation of the entire table).

    3) WCF DS uses the IQueryable API to construct the query. There's pretty much nothing to do here, just construct the right Expression. Any stock implementation of IQueryable does this. Again, no data is loaded from anywhere yet, just a description of the query is constructed.

    4) WCF DS calls IQueryable.GetEnumerator. Now this is the place where the real work is being done. The method takes the Expression from the IQueryable which describes the entire query (so all the filters, sortings and so on) and turns it into a query against the underlying data source (in the DB case this would produce the SQL to run). It then takes the results and reports them through the enumerator.

    Note that #4 has all the information necessary to only load the data needed to fullfil the request, since it has the entire query described in the Expression. So with standard built-in server driven paging (SetEntitySetPageSize) the query Expression will contain the right filters and limits (Take) to only get the necessary number of results from the underlying data source.

    IDataServicePagingProvider is only meant to be used if you want a completely different behavior of the paging. The built-in one limits each page to a certain size, if you want something different, like for example limit the amount of time it takes to run the query to get that one page, then you would need to implement IDataServicePagingProvider. Otherwise the built-in one should be just fine.

    Thanks,


    Vitek Karas [MSFT]

    • סומן כתשובה על-ידי CoryKissinger יום שלישי 28 פברואר 2012 16:56
    יום שלישי 28 פברואר 2012 11:48
    מנחה דיון

כל התגובות

  • Hi,

    IDataServicePagingProvider won't help you make your CreateDataSource better. I think the problem lies in that you're likely doing to much in the CreateDataSource.

    First a comment on the sample custom provider you're using as the bases of your code (DSPContext is the name of the context class in that sample provider): It is just a sample, and to simplify it's code it creates/gets all the data in the CreateDataSource, but that is not what real world providers would do.

    The general design should look like this:

    1) CreateDataSource simply creates a representation of the data source. For example, if your underlying data source is a database, then this would just open a connection to that database (login and so on), but it would NOT load any data from it.

    2) The IDataServiceQueryProvider which you implement is called to GetQueryRootForResourceSet. From this method you return a custom implementation of IQueryable for the specified resource set. Again, this object is just a description of the resource set, it doesn't load any data yet. (In the DB case this would return IQueryable with an expression which is the representation of the entire table).

    3) WCF DS uses the IQueryable API to construct the query. There's pretty much nothing to do here, just construct the right Expression. Any stock implementation of IQueryable does this. Again, no data is loaded from anywhere yet, just a description of the query is constructed.

    4) WCF DS calls IQueryable.GetEnumerator. Now this is the place where the real work is being done. The method takes the Expression from the IQueryable which describes the entire query (so all the filters, sortings and so on) and turns it into a query against the underlying data source (in the DB case this would produce the SQL to run). It then takes the results and reports them through the enumerator.

    Note that #4 has all the information necessary to only load the data needed to fullfil the request, since it has the entire query described in the Expression. So with standard built-in server driven paging (SetEntitySetPageSize) the query Expression will contain the right filters and limits (Take) to only get the necessary number of results from the underlying data source.

    IDataServicePagingProvider is only meant to be used if you want a completely different behavior of the paging. The built-in one limits each page to a certain size, if you want something different, like for example limit the amount of time it takes to run the query to get that one page, then you would need to implement IDataServicePagingProvider. Otherwise the built-in one should be just fine.

    Thanks,


    Vitek Karas [MSFT]

    • סומן כתשובה על-ידי CoryKissinger יום שלישי 28 פברואר 2012 16:56
    יום שלישי 28 פברואר 2012 11:48
    מנחה דיון
  • Vitek, thanks for the great explanation. This gives me a good idea of which pieces of that sample I should go back and take a closer look at. Currently, the data source that I am creating in CreateDataSource (which I will be migrating into the GetEnumerator call as recommended above) is utilizing a custom built store procedure that does not take in any paging parameters. Would you recommend updating that stored procedure to accept the entire Expression from the IQueryable?

    Thanks again,

    Cory


    Cory

    יום שלישי 28 פברואר 2012 17:00
  • Hi,

    I think the "pass expression from IQueryable to the stored procedure" is... well very hard. The expression is not the string you see the URL, it's the expression tree from Linq (System.Linq.Expression.Expression and it derived classes). Not counting that processing it is non-trivial. I have a series of blog posts about this if you're interested: http://blogs.msdn.com/b/vitek/archive/2010/02/25/data-services-expressions-part-1-intro.aspx

    Thanks,


    Vitek Karas [MSFT]

    יום רביעי 29 פברואר 2012 10:35
    מנחה דיון
  • Vitek,

    I finally got a chance to go through the blog you referenced, but I am having a little trouble applying the concepts to what I am trying to accomplish. The way I am visualizing this (probably incorrectly) is that I have the query that comes in, say:

    http://services.odata.org/OData/OData.svc/Products('99')

    Where the query specifically identifies the product it wants using the Key Property. And what I want to do is parse the relevant parts of the expression tree to be able to pull out what I need to call a store procedure, say:

    exec usp_GetProduct 99

    And then translate the result of running the stored procedure back into a list of the base Resource Type and that enumeration is returned when the IQueryable.GetEnumerator() is called as you describe in 4) above. If that is way off, please let me know. Otherwise, I think my biggest question is where in the IDataServiceQueryProvider implementation would I put the call to the store procedure?

    Thanks again

    Cory


    Cory

    יום שלישי 31 יולי 2012 00:24
  • Hi,

    I think there's a little bit of confusion in this. The paging provider allows you to implement custom limits on the amount of data returned by a query which returns collection of entities. So for example ~/Products, or ~/Category(0)/Products.

    For queries which return singletons, the paging provider does get invoked, but in the end it should not have any effect on the result. In fact it can't since singleton results can't be paged, so such a query must always return one result (the WCF DS service will fail if it returns 0 or more than one).

    If you want to turn the ~/Products(99) into a call which takes 99 as its parameter, you need to do this in your IQueryable implementation. The expression tree passed into it will have a filter over the key property and testing it for equality to 99. This is described in more detail here: http://blogs.msdn.com/b/vitek/archive/2010/06/16/data-services-expressions-part-6-key-lookup.aspx

    You can recognize this pattern and extract the necessary information from the expression tree, then execute your stored procedure and return an enumerable with the single result as the result of the query. This though has nothing to do with paging provider, since you're dealing with a single result.

    Thanks,


    Vitek Karas [MSFT]

    יום שלישי 31 יולי 2012 08:33
    מנחה דיון