Reading lots of entries

Question

Hi all;

Lets say I do a select like http://services.odata.org/OData/OData.svc/Categories except it returns 10,000 entries. And in my code I need to read one Category, then do a bunch of writing to the report, often including another select to the OData server (such as all Products in that Category), before I need the next Category entry.

For a SQL database I just read a row at a time, do all the report building for that row, then read the next row. Works great, only have 1 result set in memory at a time, and the data and connection stay good.

Should I do the same in this case? Or do OData servers expect to return all requested data ASAP and will timeout if I'm too slow to pull all the data. And/or, will the close out one request if another comes in?

thanks - dave

---

Who will win The International Collegiate Programming Championships?

Answer 1

In theory you can keep the connection open for a long time, but I would expect the server to eventually cut it (This probably depends on the web server being used, but I think proxies on the way may cut it as well). So I would not rely on it.

That said most services will not return 10 000 entries in one response. First of all such response would be huge which usually doesn't play too nice with the HTTP infrastructure (most web server have limits for this, and thus the service would have to disable them and so on). Typically the service will use server driven paging to split the response into chunks of manageable size. Each page is then transmitted as a separate response. Here's a (rather old) description how it works: http://blogs.msdn.com/b/astoriateam/archive/2009/03/19/ado-net-data-services-v1-5-ctp1-server-driven-paging.aspx.

So you could either rely on that, or force it by using so called "client side paging". Basically don't ever request entire entity set, always append $top=100 to the query. And when it returns append $skip=100&$top=100 to get the next page and so on. It's not as robust as the server driven paging, but it will work as well. The problem is if the result set changes in between the pages. The client side paging has no way to handle that nicely. The server driven paging usually handles that much better.

So I guess my advice would be to make sure that the request you issue will not return too much data in one response. Then you can safely buffer the response, process it one by one and move on to the next page. This advice only really applies if you need to do heavy processing of each entity in a streaming fashion. It's quite the opposite if you would do only light processing of each entity (then it's better to simply stream through them, since you don't have to pay for the roundtrips for each page).

As for having multiple connections opened to the server... it used to be the case that HTTP spec itself limited the number of connections to a specific server from one client to 2. And lot of client side libraries had this limit coded into them (including .NET). Nowadays this limit is usually not enforced by the server, but it's still considered a bad practice. The limit still exists in the .NET library for example, but it can be changed. But unless you really need to, I would try to avoid too many concurrent connections (I'm guessing proxies wont' be happy about it either).

Thanks,

---

Vitek Karas [MSFT]

Answer 2

Ok, I think I half get this.

First off, for anyone else reading this the url is http://blogs.msdn.com/b/astoriateam/archive/2009/03/19/ado-net-data-services-v1-5-ctp1-server-driven-paging.aspx (the editor included the closing . in Vivek's url).

Reading that link it looks like the server sets the paging and returns just the first N entries. Is that correct? That I do not set that or in any way tell it to page it?

I then have to get that next link. Do I just look for it in all the NavigationLinkEnd objects? Or is there a specific property to return it?

And how do I know when I've received the last set of data?

thanks - dave

---

Who will win The International Collegiate Programming Championships?

Answer 3

Hi again;

Another question that just occurred to me. When I pass your library a select, do you immediately query the server and pull the entire response back down immediately? I think with http you have to don't you? So I might as well copy all the entries over to my cached object immediately in that case.

??? - thanks - dave

---

Who will win The International Collegiate Programming Championships?

Answer 4

Server driven paging question:

Yes - the server just sends the first page of data back and includes a next link in it. If you want more of the data you need to send another request to the next link. That returns the next page and so on. You reach the end of it, when there's no next link in the response.

Using ODataLib you can access the next link through the ODataFeed.NextPageLink property. It will return null if there's no next link, otherwise it returns the URL of the link. Note that you have to first read through the entire feed and only access the property after your read it all. Otherwise it will likely return null even though the payload does have a next link. The safest way is to access this property only once the reader reports the FeedEnd state. Which is by the way true for everything else, only access the data on ODataEntry once the reader reports EntryEnd and so on.

HTTP query question:

Note that you don't pass the "select" (really a query) to the library, you pass it to your implementation of the message interface to get a response. And it's not true that you will always get the entire response back in one go. That depends on the configuration of the HTTP client and server. There are definitely scenarios where the payload is truly streamed (chunked HTTP encoding for one).

Thanks,

---

Vitek Karas [MSFT]

Answer 5

A follow-up to let you know this API makes it almost too easy to work with our system. We'll have a beta out soon and I'll let you know so you can look at it if you want. But this works perfectly.

Good job on the design (both OData itself and the library).

thanks - dave

---

Who will win The International Collegiate Programming Championships?

Answer 6

This is great to hear - thanks a lot!

---

Vitek Karas [MSFT]