locked
Limit and log the number of records returned RRS feed

  • Question

  • I need to:
    1st: limit the maximum number of records returned to users per query, but when i set it using MaxResultsPerCollection, the atom stream returned contains some error that results in the stream cannot be deserialized into objects on the client end; when i set it using SetEntitySetPageSize, it pops out an request error. How could I solve this problem.

    2nd: to log how many records being returned to users. I am trying to intercept queries using query interceptor and register an event handler for the RequestProcessingPipeline's ProccessedRequest event, but unable to read the response stream out of it to count the number of records in the response. Is there any other way I can achieve this?

    Anyone can help me on these issue?

    Thursday, January 7, 2010 5:55 PM

Answers

  • Hi,

    In the V2 we now have a feature called "Server Driven Paging": Take a look here http://blogs.msdn.com/astoriateam/archive/2009/03/19/ado-net-data-services-v1-5-ctp1-server-driven-paging.aspx.
    The simple way is to use what is described that will solve your problem #1.
    Problem #2 is a bit more complicated. You could use two approaches:
    a) Wrap the IQueryable returned from your service and implement the counting by overwriting the GetEnumerator method and return your own enumerator which will perform the counting and then call the original enumerator.
    b) Use custom paging provider (IDataServicePagingProvider) in which case you completely control the process. You can stop the enumeration of results at any time and provide a continuation token which can be used by the client to continue from that point on.

    The approach a) allows you to just count the entries, and is probably a bit simpler to implement.
    The approach b) allows you to do more than just counting. This is meant for example for scenarios where you don't know the number of entries you want to return, instead you for example say that the request must not take longer than 1 second, and so your enumerato will return "end here and a continuation" whenever the 1 second timeout expires. Or any other mechanism.

    Thanks,
    Vitek Karas [MSFT]
    • Marked as answer by Huang Sile Saturday, January 9, 2010 11:51 AM
    Thursday, January 7, 2010 11:32 PM
    Moderator
  • Hi,

    The API should be the same for CTP2. Note that you will need to add config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V2 into your InitializeService for this to work.
    The server driven paging requires data service version 2.0 to work (as it changes the meaning of the payload for the client, it returns less data than the client asked for).
    If you could post the error you're getting we should be able to help.

    Thanks,
    Vitek
    Vitek Karas [MSFT]
    • Marked as answer by Huang Sile Saturday, January 9, 2010 11:51 AM
    Friday, January 8, 2010 11:04 PM
    Moderator
  • Hi,

    To use IDataServicePagingProvider your DataService derived class needs to implement IServiceProvider and in its GetService method react for the type IDataServicePagingProvider by returning an instance of an object which implements that interface.
    The Data Service uses this mechanism (IServiceProvider) for almost all extensions (IDataServiceMetadataProvider, IDataServiceQueryProvider, IDataServiceStreamProvider, ...).

    Thanks,
    Vitek Karas [MSFT]
    • Marked as answer by Huang Sile Saturday, January 9, 2010 11:51 AM
    Saturday, January 9, 2010 11:35 AM
    Moderator
  • Hi Jason,

    IQueryable question:
    Note that wrapping IQueryable also means that you have to wrap the IQueryProvider retuned by its Provider property. During the processing of the request the data service will call Provider.CreateQuery to create a new IQueryable based on the one you provided but with some expression applied (like filters, projections and so on). You need to make sure that even those instances of IQueryable are wrapped by your wrapper as the data service will end up calling GetEnumerator only on the last one create (usually).
    Take a look at this blog post series, it's a great description of how to implement IQueryable (and you can pretty much copy paste the code from there). http://blogs.msdn.com/mattwar/archive/2007/07/30/linq-building-an-iqueryable-provider-part-i.aspx

    I will look into the server driven paging a bit more...

    Thanks,
    Vitek Karas [MSFT]
    Thursday, January 14, 2010 9:28 AM
    Moderator
  • Hi,

    Please note, that there's one limitation. WCF Data Services currently don't suppot custom paging (IDataServicePagingProvider) for Entity Framework based services. The limitation is not a big one as it's very likely that you need to implement your own IQueryable to be able to implement your behavior using IDataServicePagingProvider, and the built-in support for EF does not allow you to easily provide your own IQueryable.

    Thanks,
    Vitek Karas [MSFT]
    • Marked as answer by Huang Sile Tuesday, January 19, 2010 6:03 PM
    Tuesday, January 19, 2010 5:33 PM
    Moderator

All replies

  • Hi,

    In the V2 we now have a feature called "Server Driven Paging": Take a look here http://blogs.msdn.com/astoriateam/archive/2009/03/19/ado-net-data-services-v1-5-ctp1-server-driven-paging.aspx.
    The simple way is to use what is described that will solve your problem #1.
    Problem #2 is a bit more complicated. You could use two approaches:
    a) Wrap the IQueryable returned from your service and implement the counting by overwriting the GetEnumerator method and return your own enumerator which will perform the counting and then call the original enumerator.
    b) Use custom paging provider (IDataServicePagingProvider) in which case you completely control the process. You can stop the enumeration of results at any time and provide a continuation token which can be used by the client to continue from that point on.

    The approach a) allows you to just count the entries, and is probably a bit simpler to implement.
    The approach b) allows you to do more than just counting. This is meant for example for scenarios where you don't know the number of entries you want to return, instead you for example say that the request must not take longer than 1 second, and so your enumerato will return "end here and a continuation" whenever the 1 second timeout expires. Or any other mechanism.

    Thanks,
    Vitek Karas [MSFT]
    • Marked as answer by Huang Sile Saturday, January 9, 2010 11:51 AM
    Thursday, January 7, 2010 11:32 PM
    Moderator
  • Just want to confirm:
    The link that you provided is the data service v1.5 ctp1, and there is a recent release of data service v1.5 ctp2 that also supports Server Driven Paging if i am not wrong. I have the v1.5 ctp2 installed currently, and using SetEntitySetPageSize which results in a request error which i am still trying to debug. So should i revert back to v1.5 ctp1 in this case?

    Friday, January 8, 2010 6:06 PM
  • Hi,

    The API should be the same for CTP2. Note that you will need to add config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V2 into your InitializeService for this to work.
    The server driven paging requires data service version 2.0 to work (as it changes the meaning of the payload for the client, it returns less data than the client asked for).
    If you could post the error you're getting we should be able to help.

    Thanks,
    Vitek
    Vitek Karas [MSFT]
    • Marked as answer by Huang Sile Saturday, January 9, 2010 11:51 AM
    Friday, January 8, 2010 11:04 PM
    Moderator
  • Hi Vitek,

    Great thanks to your help on solving issue 1. Once i added config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V2, the request error is no longer there.

    And just want to verify if I fully understand ur description on solving issue 2:
    So if i were to use approach (a) i will write a inheritance of the IQuerable class with override methods for GetEnumerator, and use it as a returned type.
    If i were to use (b) i need to write a custom paging provider, but I don't see where the IDataServicePagingProvider is being called/used when i explored through the definitions of the references. Would you mind explaining more about this appraoch?

    Saturday, January 9, 2010 11:32 AM
  • Hi,

    To use IDataServicePagingProvider your DataService derived class needs to implement IServiceProvider and in its GetService method react for the type IDataServicePagingProvider by returning an instance of an object which implements that interface.
    The Data Service uses this mechanism (IServiceProvider) for almost all extensions (IDataServiceMetadataProvider, IDataServiceQueryProvider, IDataServiceStreamProvider, ...).

    Thanks,
    Vitek Karas [MSFT]
    • Marked as answer by Huang Sile Saturday, January 9, 2010 11:51 AM
    Saturday, January 9, 2010 11:35 AM
    Moderator
  • ...
    a) Wrap the IQueryable returned from your service and implement the counting by overwriting the GetEnumerator method and return your own enumerator which will perform the counting and then call the original enumerator.
    ...
    Vitek Karas [MSFT]

    Hi, just curious, does the filters (?$filter=[filters here]) get applied during this phase or does this counts the total number of results before filtering?

    The reason I'm asking is because I have tried and the count seems to represent the total number of entries in the datasource.

    Your help will be greatly appreciated. :)
    Wednesday, January 13, 2010 4:25 AM
  • Hi,

    To use IDataServicePagingProvider your DataService derived class needs to implement IServiceProvider and in its GetService method react for the type IDataServicePagingProvider by returning an instance of an object which implements that interface.
    The Data Service uses this mechanism (IServiceProvider) for almost all extensions (IDataServiceMetadataProvider, IDataServiceQueryProvider, IDataServiceStreamProvider, ...).

    Thanks,
    Vitek Karas [MSFT]

    Hi,

    I tried this with DataServiceConfiguration.SetEntitySetPageSize config but it dosent seems to call the custom paging provider. It only catches the type IDataServiceMetadataProvider and I returned null for that.

    Is it right to return null if I don't have a provider for IDataServiceMetadataProvider?

    Thanks Alot!
    Wednesday, January 13, 2010 6:50 AM
  • Hi,

    The first question (about filters):
    Filters ($filter) are applied to the IQueryable as Where calls. Since the enumeration is started only once the entire query is built (including the filters) it should only report the number of results after filtering. Our serializer does throw aways results. Whatever the query returns will be sent as a response, so the query must contain everything.
    Note that you need to react to the GetEnumerator call on your IQueryable, other methods may be called during the process of building the query.

    The second question (paging providers):
    We support two ways:
    1) Built in server driven paging. This is set by using the SetEntitySetPageSize method. We won't use the IDataServicePagingProvider to implement that. We implement this on our own (it will add OrderBy, Take to the query).
    2) Custom server driven paging. This is when you provide an implementation of IDataServicePagingProvider. In that case you're on your own, we give complete control over the paging to you and we don't do anything on our own.

    Thanks,
    Vitek Karas [MSFT]
    Wednesday, January 13, 2010 12:53 PM
    Moderator
  • Hi Vitek,

    Thanks for your prompt response.

    1. How do I use/apply my Custom server driven paging?

    I have tried to implement IDataServicePagingProvider in a class called CustomPagingProvider and implemented the IServiceProvider in my service class.

    In the GetService method I have also reacted to the type IDataServicePagingProvider.

    Am I missing out anything?

    Thanks in advance.

    Thursday, January 14, 2010 1:17 AM
  • Hi Vitek,

    I have written a wrapper for the IQueryable returned from my service but it dosent call the GetEnumerator method at all. Any idea why does that happen?

    Regards,
    Jason Poh
    Thursday, January 14, 2010 2:43 AM
  • Hi Jason,

    IQueryable question:
    Note that wrapping IQueryable also means that you have to wrap the IQueryProvider retuned by its Provider property. During the processing of the request the data service will call Provider.CreateQuery to create a new IQueryable based on the one you provided but with some expression applied (like filters, projections and so on). You need to make sure that even those instances of IQueryable are wrapped by your wrapper as the data service will end up calling GetEnumerator only on the last one create (usually).
    Take a look at this blog post series, it's a great description of how to implement IQueryable (and you can pretty much copy paste the code from there). http://blogs.msdn.com/mattwar/archive/2007/07/30/linq-building-an-iqueryable-provider-part-i.aspx

    I will look into the server driven paging a bit more...

    Thanks,
    Vitek Karas [MSFT]
    Thursday, January 14, 2010 9:28 AM
    Moderator
  • Hi Vitek,

    Thanks for the clarification on IQueryable will try that.
    Thursday, January 14, 2010 9:43 AM
  • Hi Vitek,

    I have tried the above link you posted and I'm facing this error when I look at the tracelog in IntelliTrace everytime I try to access my data service:

    Exception:Thrown: "No method 'OrderBy' on type 'System.Linq.Queryable' is compatible with the supplied arguments." ("System.InvalidOperationException")
    A System.InvalidOperationException was thrown: No method 'OrderBy' on type 'System.Linq.Queryable' is compatible with the supplied arguments.

    Any idea why is this happening? Thanks in advance.
    Friday, January 15, 2010 6:15 AM
  • Hi,

    To support orderby, your IQueryable has to in fact implement IOrderedQueryable interface. It's basically just a different name (no new methods or properties).

    Thanks,
    Vitek Karas [MSFT]
    Saturday, January 16, 2010 3:03 AM
    Moderator
  • Hi Vitek,

    Thanks for your reply.

    I have implemented IOrderedQueryable but still I get the following error:

    Exception Thrown: "No method 'OrderBy' on type 'System.Linq.Queryable' is compatible with the supplied arguments." ("System.InvalidOperationException")


    public class Query<T> : IQueryable<T>, IQueryable, IEnumerable<T>, IEnumerable, IOrderedQueryable<T>, IOrderedQueryable 
    This is how my IQueryable looks like.

    Monday, January 18, 2010 1:35 AM
  • Hi Vitek,

    Thanks a million for your help for the past week.

    I've finally figured out how to do this. :) There seems to be some misunderstand I had.

    Regards,
    Jason Poh
    Monday, January 18, 2010 12:16 PM
  • Hi Vitek,

    I refer to the blog: http://blogs.msdn.com/alexj/archive/2010/01/07/creating-a-data-service-provider-part-2-iserviceprovider-datasources.aspx
    for implementation of IDataServicePagingProvider:

    In my DataService inherited class, I have:

    namespace Nimbus.Data.Services
    {
        public class DataService<T>: System.Data.Services.DataService<T>, IServiceProvider
        {
            public object GetService(Type serviceType)
            {
                if (serviceType == typeof(IDataServicePagingProvider)){
                    return new Nimbus.Data.Providers.IDataServicePagingProvider();
                }else{
                    return null;
                }

    I realized that data service request does go to GetService(), however, it is not trigering the serviceType == typeof(IDataServicePagingProvider) condition. May I know how to triger this condition to be true?

    Thanks and best regards,

    Sile :)

    Monday, January 18, 2010 6:24 PM
  • Hi,

    I just tried and it works for me just fine. I tried with a simple reflection based service as well as full IDataServiceMetadataProvider and in both cases the IDataServicePagingProvider gets instantiated and the SetContinuationToken on it gets called when I issue a request for an entity set.
    Does your GetService method get called at all?

    Note that it won't get called for requests which don't need it. So for example the service document request (the root of the service) will not call this. Neither will a request which looks for a single result. It should only get called for requests which migth return multiple results.

    Thanks,
    Vitek Karas [MSFT]
    Monday, January 18, 2010 9:50 PM
    Moderator
  • Hi,

    I understand that for the request that list name of data sets in a data service won't call for the IDataServicePagingProvider but IDataServiceMetadataProvider instead. However, when i tried on a data set with filter condition, only IDataServiceMetadataProvider gets into GetService() method again.

    Do I need to set anything on the config in order to trigger IDataServicePagingProvider?

    Thanks,

    Sile
    Tuesday, January 19, 2010 1:15 AM
  • Hi,

    Please note, that there's one limitation. WCF Data Services currently don't suppot custom paging (IDataServicePagingProvider) for Entity Framework based services. The limitation is not a big one as it's very likely that you need to implement your own IQueryable to be able to implement your behavior using IDataServicePagingProvider, and the built-in support for EF does not allow you to easily provide your own IQueryable.

    Thanks,
    Vitek Karas [MSFT]
    • Marked as answer by Huang Sile Tuesday, January 19, 2010 6:03 PM
    Tuesday, January 19, 2010 5:33 PM
    Moderator
  • Hi Vitek,

    I did the implementation of the IQueryable using the InterceptProvider from the link you posted.

    Everything works fine until I use a projection and I get this error:

    <innererror>
      <message>Unable to create a constant value of type 'System.Data.Services.Internal.ProjectedWrapper2'. Only primitive types ('such as Int32, String, and Guid') are supported in this context.</message>
      <type>System.NotSupportedException</type>
      <stacktrace>  at System.Data.Objects.ELinq.ExpressionConverter.ConstantTranslator.TypedTranslate(ExpressionConverter parent, ConstantExpression linq)&#xD;
      at System.Data.Objects.ELinq.ExpressionConverter.TranslateExpression(Expression linq)&#xD;
      at System.Data.Objects.ELinq.ExpressionConverter.ConditionalTranslator.TypedTranslate(ExpressionConverter parent, ConditionalExpression linq)&#xD;
      at System.Data.Objects.ELinq.ExpressionConverter.TranslateLambda(LambdaExpression lambda, DbExpression input)&#xD;
      at System.Data.Objects.ELinq.ExpressionConverter.TranslateLambda(LambdaExpression lambda, DbExpression input, DbExpressionBinding&amp; binding)&#xD;
      at System.Data.Objects.ELinq.ExpressionConverter.MethodCallTranslator.SelectTranslator.Translate(ExpressionConverter parent, MethodCallExpression call)&#xD;
      at System.Data.Objects.ELinq.ExpressionConverter.MethodCallTranslator.TypedTranslate(ExpressionConverter parent, MethodCallExpression linq)&#xD;
      at System.Data.Objects.ELinq.ExpressionConverter.TranslateExpression(Expression linq)&#xD;
      at System.Data.Objects.ELinq.ELinqQueryState.GetExecutionPlan(Nullable`1 forMergeOption)&#xD;
      at System.Data.Objects.ObjectQuery`1.GetResults(Nullable`1 forMergeOption)&#xD;
      at System.Data.Objects.ObjectQuery`1.System.Collections.Generic.IEnumerable&lt;T&gt;.GetEnumerator()&#xD;
      at System.Data.Objects.ObjectQuery`1.GetEnumeratorInternal()&#xD;
      at System.Data.Services.Internal.ProjectedWrapper.EnumerableWrapper.GetEnumerator()&#xD;
      at System.Data.Services.WebUtil.GetRequestEnumerator(IEnumerable enumerable)</stacktrace>
     </innererror>
    
    

    it seems that Linq to Entities cannot translate to the wrappers. I tried to implement IProjectionProvider but it's internal.

    Anyway to work around this?

    Thanks

    Frank

    Friday, December 10, 2010 10:28 PM
  • Hi,

    If your underlying provider is EF you need to disable null propagation (IDataServiceQueryProvider.IsNullPropagationRequired == false), otherwise the expression trees generated by WCF DS won't work with EF. There are other potential issues with custom provider over EF, again due to limitations in EF in regard which expression trees it can consume. You might need to modify the expression tree a little bit before passing it down to EF.

    Thanks,


    Vitek Karas [MSFT]
    Saturday, December 11, 2010 12:50 PM
    Moderator
  • Any link you can point on how can I modify the expression tree for the EF?

    Thanks

    Frank

    Monday, December 13, 2010 3:13 PM
  • Expression trees are usually modified using the Visitor pattern, derive from ExpressionVisitor. There are bunch of articles on the web on how to do that. But ultimately it's your choice. As to what to modify and how that depends on the query and on your settings and data model.

    I would start by making sure that the IsNullPropagationRequired is set to false, that should enable most of the queries to run just fine. You will have to test the rest. The one thing which might be a bit problematic for EF is: When ProjectedWrapper is instantiated in the query (used when $select is used), its properties are assigned values. These values are some expressions which are wrapped in a Convert node which casts them to System.Object. EF doesn't always like this cast, so removing it might help in certain cases. But if that would be the problem you would see a failure along the lines "Can't convert to System.Object" or something like that.

    Thanks,


    Vitek Karas [MSFT]
    • Proposed as answer by FrankXP Monday, December 13, 2010 4:41 PM
    Monday, December 13, 2010 3:49 PM
    Moderator
  • Exactly as you said, making IDataServiceQueryProvider.IsNullPropagationRequired = false removed the previous error but it's complaining now that it cannot cast System.String to System.Object.
    I will try to modify the tree and post my results.

    Thanks a lot

    Frank

    Monday, December 13, 2010 4:32 PM