locked
Azure Application Insights REST API - returning only 500 rows RRS feed

  • Question

  • Hi 

    When i use Azure Application Insights REST API to fetch data from Application insights custom events i am only getting back 500 rows. Is there any setting or parameters to remove this limit of 500 rows.

    thanks 

    Naresh 

    Wednesday, March 27, 2019 10:25 AM

Answers

  • Hello Naresh!

    Sorry for not getting back sooner here. I did reach out to our internal Teams regarding this query, and here is our suggestion:

    So, it turns out that the pagination logic is not an out of the box capability that we offer as of today. Our current recommendation is for you to build your own iteration/pagination logic to pull data in time intervals tuned to the overall volume. For some it’s an hour, for some it’s 5 minutes. The current limits are 64MB/500K rows per query.

    Meanwhile, you could also use the Analytics query API to get the desired events. There is still a limit, but it will be quite a bit higher than the current /events API.

    Hope this helps! Let us know if there is anything else related to this thread that we can assist you with, else please feel free to mark and close this out if any of the above responses have been of help! Thanks again for reaching out!

    Monday, April 15, 2019 5:01 PM

All replies

  • Hello Naresh,

    Yes, there is a default limit of 500 rows added on the result set of the API. However, you can use the $top parameter to circumvent this. As quoted in the API Documentation:

    The $top parameter returns the first "n" result records of a query:

      GET /v1/apps/{app-id}/events/customEvents?timespan=2019-03-01%2F2019-03-31&$top=1000

    The above fetches the first 1000 records of a query. Check this example out.

    Additional Info:

    You can also add $count=true to get a count of the records matching a search or filter, but ignoring $top. For example, to get the number of customEvents collected:

    GET /v1/apps/{app-id}/events/customEvents?timespan=2019-03-01%2F2019-03-31&$top=1000&$count=true

    This will append the following at the end of the result set:

     ...
        }
      ],
      "@odata.count": <value>
    }

    Here is another detailed example for your reference.

    Hope this helps!


    Sunday, March 31, 2019 5:38 AM
  • Thank you Bhargavi. 

    after using top filter it did bring more than 500 records. But is there a hard limit of 5000. I put top 10000 but it only brought back 5000 records. I know that there are more than 5000 records.

    Tuesday, April 2, 2019 9:44 AM
  • thank you Bhargavi.

    after using top filter it did bring back more than 500 records. but is there a hard limit of 5000. I put top 10000 but it only brought back 5000 records.

    "@ai.messages": [
        {
          "code": "AddedLimitToQuery",
          "message": "The query was limited to 5000 rows"
        }
      ],
      "@odata.count": 18279,

    Tuesday, April 2, 2019 9:55 AM
  • Thanks for providing additional information!

    I'm investigating this further, and will keep you posted on my findings.

    Tuesday, April 2, 2019 9:57 AM
  • You could use the $skip parameter along with $top to iterate over blocks of data, as $skip returns records starting at the nth. Have you checked that out yet?

    For example:

    To get 10 records starting at the 20th:

        GET /v1/apps/{app-id}/events/$all?$top=10&$skip=20

    The same is mentioned in the API Documentation as well. Please give it a try and let me know if it works for you!

    Tuesday, April 2, 2019 12:14 PM
  • thanks. 

    I have to use that workaround to do the iteration if there is a hard limit of 5000 records.

    Can you please check and confirm the 5000 limit.

    thanks 

    naresh 

    Tuesday, April 2, 2019 4:19 PM
  • Yes, I was also able to fetch only 5000 records at a time using $skip and $top parameters, and believe this is the set limit AFAIK.

    I shall, however, check with our internal teams and get back if I have additional information on this.

    Thanks for reaching out!

    Wednesday, April 3, 2019 2:27 PM
  • Hello Naresh!

    Sorry for not getting back sooner here. I did reach out to our internal Teams regarding this query, and here is our suggestion:

    So, it turns out that the pagination logic is not an out of the box capability that we offer as of today. Our current recommendation is for you to build your own iteration/pagination logic to pull data in time intervals tuned to the overall volume. For some it’s an hour, for some it’s 5 minutes. The current limits are 64MB/500K rows per query.

    Meanwhile, you could also use the Analytics query API to get the desired events. There is still a limit, but it will be quite a bit higher than the current /events API.

    Hope this helps! Let us know if there is anything else related to this thread that we can assist you with, else please feel free to mark and close this out if any of the above responses have been of help! Thanks again for reaching out!

    Monday, April 15, 2019 5:01 PM