locked
I have 5m dependency logs per day with a small website RRS feed

  • Question

  • I installed AI on my site and run of quota in one day :( And, no, I do not have TONS of users.

    I got 5.56m DEPENDENCY of which 5.4m for SQL. What?? This in about 8 hours. Something like 187 hits to the DB per second. No way.

    So the question is: what is logged under dependency/SQL ? Could this be a bug or what?

    AI looks great but I can't generate 5m per 1/2 day when Google Analytics says 7k sessions.

    Thanks

    Tuesday, April 5, 2016 9:36 AM

Answers

  • Hello pmonte,

    Filtering is released with 2.0 Stable SDK, so you should be able to use it right away without installing Beta. E.g. you can filter out Dependencies with small duration that are of no interest to you: it will limit the amount of data points that are sent to AI; it will still report slow dependencies that you care about.

    I would expect that LINQ + EF calls into a lot of SQL Methods that are considered as dependencies in AI (the list is in my reply above), but they should be really light-weight. You can check how many of these calls are actually making it to the SQL server via some network traffic tool or some sort of SQL profiling. If all of them do - then it's your actual dependencies and will include network time to SQL, if most of them do not go to SQL - then you can try to filter them out based on some condition/common pattern they have to avoid reporting them.

    Regarding Page Views, even if any action is the Page View, you might still have much more requests than views if there are AJAX calls that perform the additional queries to backend from the page, AJAX won't trigger a page view telemetry, but might trigger AJAX dependency telemetry collection.

    I understand that you reached quota already, so I can suggest you to create another app in AI with new Instrumentation Key and use it.


    Dmitry Matveev

    Friday, April 8, 2016 10:32 PM

All replies

  • Hello pmonte,

    This is the code that listens to these SQL ETW events in case you're running .NET 4.6+.

    This is the code that instruments these SQL calls in case you're running .NET 4.0+ with AI Status Monitor on the box.

    Could you please double check that 190 dependencies per second can be produced by the respective method for your code?

    I understand that even if this amount turns out to be expected based on the instrumentation approach - that hits free tier quota really fast. I would recommend to use sampling or adaptive sampling in this case to limit the amount of the telemetry items collected per application instance and stay within the quota with the statistically correct data.


    Dmitry Matveev

    Tuesday, April 5, 2016 3:21 PM
  • Hello Dmitry,

    thanks for the hint but I use LINQ with EF and pair my code with this is quite complicated (I mean I have a .Where and here a ExecuteReader, probably it's a lack of knowledge by me but...).

    Said that, it sounds REALLY strange that our app could generate such an amount of data for sql dependency. However I set sampling to 2%. I have now to wait 1st of May to see how it goes :(

    My question now is: I'm interested to diagnose slow requests. It was nice to be able to drill down to sql part of the request to understand if the slow down was connected to the DB or not. With a sampling of 2% can I see and examine a single slow request? If so can I examine calls to dependencies made by that requests?

    Thanks a lot for you help


    Wednesday, April 6, 2016 2:35 PM
  • Hello pmonte,

    Glad to help here.

    You should be able to examine slow requests even with a small sampling rate - the telemetry collected per request is sent or dropped altogether, so the related items will be there for the requests that are not sampled.

    You can read about this here in Azure Docs. The small snippet from there:

    "The sampling decision is based on several rules that aim to preserve all interrelated data points intact, maintaining a diagnostic experience in Application Insights that is actionable and reliable even with a reduced data set. For example, if for a failed request your app sends additional telemetry items (such as exception and traces logged from this request), sampling will not split this request and other telemetry. It either keeps or drops them all together. As a result, when you look at the request details in Application Insights, you can always see the request along with its associated telemetry items."


    Dmitry Matveev

    Wednesday, April 6, 2016 3:54 PM
  •  It either keeps or drops them all together. 

    Ouch, does this mean that I have a 2% chance that my slow requests are logged?

    Wednesday, April 6, 2016 3:57 PM
  • Hello pmonte,

    Sampling means that only certain percent of the data is collected.

    So, yes, approximately only 2% of all telemetry that AI WEB SDK collects is going to be sent to AI in your case. Currently, it does not take "slowness" into the consideration, so slow requests are sampled just as fast requests. Statistically, over a period of time, if the application has an issue with the requests being slow under certain conditions - you'd notice it even with 2% sampling and all the related data will be collected for that request: exceptions/dependencies and etc. to conduct the analysis of why it's slow.

    I think that you can open a User Voice request (if there is no such request already) to always provide at least one diagnostics telemetry example for slow request (above some certain configurable threshold) regardless of sampling.


    Dmitry Matveev

    Wednesday, April 6, 2016 4:06 PM
  • Thanks again Dmitry things are getting clearer and clearer.

    Sampling is not the preferred option for us, the main reason to use AI is to examine and look into "problems" (exceptions, slow requests, timeouts etc) so we'd need to have them all.

    Can we get back to the original problem? How can I better understand why in one day I got

    Dependency 79.2%

    Request 20.1%

    Pageview 0.3%

    others 0.4%

    Please note that it is not a single page application, on the other hand, each action is a different page view (it is a real estate portal so even when you change search parameters we modify the url for SEO and reload the page).

    How can I dig deeper? 
    What I have done so far is to select a single request to the home page and what I get when I show all telemetry for this opeartion:
    1 Request
    153 Dependecy all of them SQL
    If I click on any of the 153 SQL Dependecy I get very few detailsbasically timestamp, if successful (yes) and duration.

    Is it possible to filter without using beta sdk?


    • Edited by pmonte Thursday, April 7, 2016 7:26 AM
    Thursday, April 7, 2016 7:17 AM
  • Hello pmonte,

    Filtering is released with 2.0 Stable SDK, so you should be able to use it right away without installing Beta. E.g. you can filter out Dependencies with small duration that are of no interest to you: it will limit the amount of data points that are sent to AI; it will still report slow dependencies that you care about.

    I would expect that LINQ + EF calls into a lot of SQL Methods that are considered as dependencies in AI (the list is in my reply above), but they should be really light-weight. You can check how many of these calls are actually making it to the SQL server via some network traffic tool or some sort of SQL profiling. If all of them do - then it's your actual dependencies and will include network time to SQL, if most of them do not go to SQL - then you can try to filter them out based on some condition/common pattern they have to avoid reporting them.

    Regarding Page Views, even if any action is the Page View, you might still have much more requests than views if there are AJAX calls that perform the additional queries to backend from the page, AJAX won't trigger a page view telemetry, but might trigger AJAX dependency telemetry collection.

    I understand that you reached quota already, so I can suggest you to create another app in AI with new Instrumentation Key and use it.


    Dmitry Matveev

    Friday, April 8, 2016 10:32 PM