none
Duplicate datapoints

    Question

  • Hi,

    Sorry I'm a little new with AI, so apologies if this question lacks detail!

    We have just deployed AI version 2.0.0 to our existing asp.net site and it's been ticking over impressively for 3 days, then on the fourth day the datapoint count went from around 300k per day to 8 million. All day there were warnings that throttling was occurring because we were exceeding our 30k 5 minute limit. Finally we exceeded our free plan quota so now we're not reporting telemetry.

    Checking some of the datapoints, it looks like there are duplicates. So filtering on just page views for one example I see 8 page view datapoints for one matching IIS log entry.

    Are there recommendations for further diagnosing the problem? I could do with knowing whether the application is generating the duplicates or whether they are being mirrored at some later point. I've been hunting around forums and can't see anyone who has this problem where the problem appeared overnight. It's just a single server setup, no web farm or anything like that which might explain it.

    Anyway - any help much appreciated!

    Thanks.


    Friday, May 20, 2016 9:13 AM

Answers

All replies

  • Please clarify the following:

    • Are you using the web app extension or have you recompiled your application with the AI SDK?
    • If you are using the SDK, which library packages are you using? (I presume Microsoft.ApplicationInsights and Microsoft.ApplicationInsights.Web)
    • What do you have configured in your ApplicationInsights.config file?
    • What AI modules are configured in your Web.config file?

    Are you making your own trace calls in addition to the automatic telemetry capture?

    Friday, May 20, 2016 6:31 PM
  • Hi,

    Thanks for replying.

    I'm not 100% sure what you mean by a web app extension, so I think we've not done that. We've added AI to our solutions via NuGet.

    Yes we're using those packages you mention plus 

    .Agent.Intercept
    .DependencyCollector
    .PerfCounterCollector
    .WindowsServer
    .WindowsServer.TelemetryChannel

    Our configuration file is basically the same as the example so all modules are configured...

    <Add Type="Microsoft.ApplicationInsights.DependencyCollector.DependencyTrackingTelemetryModule, Microsoft.AI.DependencyCollector"/>

    <Add Type="Microsoft.ApplicationInsights.Extensibility.PerfCounterCollector.PerformanceCollectorModule, Microsoft.AI.PerfCounterCollector"> [[no counters set]] </Add>

    <Add Type="Microsoft.ApplicationInsights.WindowsServer.DeveloperModeWithDebuggerAttachedTelemetryModule, Microsoft.AI.WindowsServer"/>
    <Add Type="Microsoft.ApplicationInsights.WindowsServer.UnhandledExceptionTelemetryModule, Microsoft.AI.WindowsServer"/>
    <Add Type="Microsoft.ApplicationInsights.WindowsServer.UnobservedExceptionTelemetryModule, Microsoft.AI.WindowsServer"/>
    <Add Type="Microsoft.ApplicationInsights.Web.RequestTrackingTelemetryModule, Microsoft.AI.Web">

    <Handlers>
    <Add>System.Web.Handlers.TransferRequestHandler</Add>
    <Add>Microsoft.VisualStudio.Web.PageInspector.Runtime.Tracing.RequestDataHttpHandler</Add>
    <Add>System.Web.StaticFileHandler</Add>
    <Add>System.Web.Handlers.AssemblyResourceLoader</Add>
    <Add>System.Web.Optimization.BundleHandler</Add>
    <Add>System.Web.Script.Services.ScriptHandlerFactory</Add>
    <Add>System.Web.Handlers.TraceHandler</Add>
    <Add>System.Web.Services.Discovery.DiscoveryRequestHandler</Add>
    <Add>System.Web.HttpDebugHandler</Add>
    </Handlers>
    </Add>
    <Add Type="Microsoft.ApplicationInsights.Web.ExceptionTrackingTelemetryModule, Microsoft.AI.Web"/>


    In web.config we have Microsoft.ApplicationInsights.Web.ApplicationInsightsHttpModule configured.

    We are not currently capturing our own trace calls, we do have some but they are configured off at the moment, so it's basically the automated telemetry only.

    I hope this all makes sense and thank you for your help.

    Tuesday, May 24, 2016 11:02 AM
  • Hi,

    More information - we host our website as an IIS Application below the Default Website. I spotted that there was an ApplicationInsights.config file in the wwwroot folder and that it was for a previous version of AI (1.2.3). We were running an older version of AI prior to deploying v2.0.0 so this must be left over from that old deployment. I don't think we actually copied the file into there but either way I have removed it.

    I'm hoping that there was a problem with the website and an application under it having two different configurations.

    I will update this once I know but for now I have to wait for our free plan to renew at the end of this month to see whether it's been cured or not.

    Just thought I'd keep this question up to date :)

    Thursday, May 26, 2016 9:05 AM
  • OK I know what's happening, but not why.

    In IE (not Chrome) if you run Fiddler you can see that there is a call to dc.services.visualstudio.com. It contains a couple of interesting data in the request header of type PageviewData and PageviewPerformanceData. The response contains an error from AI - Too much data code 429. Fair enough - it's getting hammered.

    But then 5 seconds later, it posts again, and this time the request header contains an extra bit of data - ExceptionData - from http://az416426.vo.msecnd.net/scripts/a/ai.0.js. If you break on script errors you can see this is a problem in the AI script - "Object doesn't support property or method 'forEach'". The call stack seems to indicate that this code is executed when a script error is encountered (t.onerror). 

    So it looks very much like if you leave the browser open on any page that every 5 seconds it posts to dc.services.visualstudio.com with an ever increasing request with one extra ExceptionData entry in it and the response from that post causes an error which triggers the onerror which fails which causes a post to dc.services.visualstudio.com with extra ExceptionData in and the response causes an error which triggers the onerror which... I mean I could go on :)

    In Chrome, this doesn't happen - it does receive a response containing a 429 but it doesn't then trigger onerror so the error request / response loop doesn't happen.

    Has http://az416426.vo.msecnd.net/scripts/a/ai.0.js changed at all? Maybe sometime around a fortnight ago?

    Ta.


    Wednesday, June 1, 2016 12:59 PM
  • Hi,

    thank you for providing all details. This is very helpful. Yes, we released a new version of ai.0.js a couple days ago. I'll disable those changes to mitigate this behavior and release a fixed version soon.

    Edit: we are tacking this issue on GitHub: https://github.com/Microsoft/ApplicationInsights-JS/issues/217

    Thanks,

    Kamil



    Wednesday, June 1, 2016 5:29 PM
  • Hi Kamil,

    Thanks for replying.

    I'll see how it goes in GitHub!

    Thanks :)

    Thursday, June 2, 2016 9:12 AM
  • Gah - monthly quota gone again - on day 2. I won't be able to tell whether this has worked until July :(
    Thursday, June 2, 2016 9:15 AM
  • Hi Chris,
    would you mind sending your instrumentation key to aidevsupport@microsoft.com ? 

    Thank you,
    Kamil 


    Thursday, June 2, 2016 6:28 PM
  • Sent :)

    I did raise a support request asking for the quota to be reset and they are looking into it I think.

    Ta.

    Friday, June 3, 2016 2:38 PM
  • Thank you for providing the instrumentation key. We found the issue and we are in the process of release a new version of JavaScript SDK. 

    We are investigating multiple options to re-enable your monitoring this month. You may also consider initiating your free Premium trial for the next 30 days to quickly unblock your monitoring. 

    Thank you!


    Tuesday, June 7, 2016 5:51 PM