locked
Why there is limit on 100000 on skip ? RRS feed

  • Question

  • 1. How can we retrieve or page through more than 100000 records?  

    2.Sometimes we might retrieve more than 100000 records Geography information and create customer asset foot prints on the map. With 100000 limit we are not able to do that ?

    Wednesday, July 26, 2017 2:54 AM

Answers

  • You can retrieve more than 100000 documents from an index by using a combination of $filter and $orderby. You will need a filterable field that you can use to partition the data (you can use facets to find good candidates for this). You will also need a sortable field that has unique values in the index, otherwise the order of documents is not guaranteed to be stable.

    Note that there are no consistency guarantees. If documents are added or deleted from the index while paging, you may observe duplicate or missing results. This StackOverflow thread explains why.

    All that said, it sounds like your scenario is to display geographic search results to the end user. Will your end users be able to make sense of more than 100000 points on a map?

    Wednesday, July 26, 2017 4:48 AM
    Moderator

All replies

  • You can retrieve more than 100000 documents from an index by using a combination of $filter and $orderby. You will need a filterable field that you can use to partition the data (you can use facets to find good candidates for this). You will also need a sortable field that has unique values in the index, otherwise the order of documents is not guaranteed to be stable.

    Note that there are no consistency guarantees. If documents are added or deleted from the index while paging, you may observe duplicate or missing results. This StackOverflow thread explains why.

    All that said, it sounds like your scenario is to display geographic search results to the end user. Will your end users be able to make sense of more than 100000 points on a map?

    Wednesday, July 26, 2017 4:48 AM
    Moderator
  • Will your end users be able to make sense of more than 100000 points on a map?

    If the user have to work with individual points on the map , answer to your question is NO.

    But our intention is to show them the heat map of their assets in a given geographical bounds, then filtering /retrieving more than 1000000 record should be valid.

    Thursday, July 27, 2017 5:47 PM
  • Heat maps are a good way of presenting aggregate geospatial data. Unfortunately, Azure Search does not support such aggregations in the search request itself. The problem with aggregating on the client side is that eventually the amount of data you'd have to read would make the latency unacceptably high. If 100000 is not enough, what about one million? Ten million? Without aggregations, this way of building a heat map will not scale.

    Do you have a sense of how many points you'd need to present for a single request?

    Friday, August 4, 2017 7:31 PM
    Moderator
  • This is multi tenant application. It various from customer to customer 50,000 to 4 million. If all the result document's geospatial data is aggregated to one geojson reponse would be an awesome feature. This would allow us not to retrieve all the records on the client side ( or server side aggregation)  and aggregate. 
    • Edited by CBaski Friday, August 11, 2017 2:45 AM
    Friday, August 11, 2017 2:44 AM