none
Filtering/grouping external depdenencies

    Question

  • My service has a number of genuine external dependencies. It also has a feature that spiders a website.

    Typically I spider 1000's of websites a week.

    For each website I spin up a HTTP client and make the necessary requests before closing it.

    Each website (session) can have anything from 1 to 2000 hits.

    I want to separate all this traffic out in to a spider

    • All "Spider" trafic
    • Particular session/ website
    • particular request in a session

    How can I instrument my code to do this?

    Then I want to filter it all out as a critical dependency / automated alerts etc.

    Monday, May 9, 2016 1:12 PM

Answers

All replies

  • If we are talking about autocollected dependencies you can enrich data by adding TelemetryInitializer: http://apmtips.com/blog/2014/12/01/telemetry-initializers/

    If you call TrackDependency you create DependencyTelementry manually so you can add any properties you want.

    You can consider setting Context.Operation.SyntheticSource. By default will filter all synthetic traffic out in most views.

    You can add custom properties so you can search for specific websites. You may be able also to group if custom property value does not reach a limit of unique value that I think now is 2000.


    Anastasia

    Monday, May 9, 2016 10:47 PM
    Moderator
  • Thanks that is a step in the right direction.

    That example seems to set the property globally.

    Once I've defined the telemetry property how do I set that for a given process?

    e.g. my website will create a

    Spider x = new Spider(SessionGuid);

    For all the calls within the Spider class I would want the application insights property setting to the SessionGuid.

    Monday, May 9, 2016 11:11 PM
  • TelemetryInitializer sets properties on each telemetry item independently. It is not global. When you call telemetryClient.Track (or our autocollection module does it) all telemetry initializers all called. Do you have HttpContext or some other container from where you can get context specific information like SessionGuid? In that case you would take it from there and apply. You can check what initializers we have as an example: https://github.com/Microsoft/ApplicationInsights-dotnet-server/blob/master/Src/Web/Web.Shared.Net/OperationNameTelemetryInitializer.cs

    If you can create telemetry manually everything is much simpler. You would create DependencyTelemetry and set its properties using local members of you spider class. And than you just call Track(your_telemetry_item) 

     

    Anastasia

    Monday, May 9, 2016 11:33 PM
    Moderator
  • That is another part of the problem I think.

    The page initial request spins off another thread in the background.

    The user is taken to a  "please wait" page.

    Once the thread completes in the background the waiting page wakes up and redirects the user.

    There is no context to the background thread.

    Tuesday, May 10, 2016 12:09 AM
  • Can you track your dependencies manually?

    var dependency = new DependencyTelemetry();

    dependency.Context....= this....;

    telemetryClient.Track(dependency);

    And than initialize context from local members so you do not need telemetry initializers?


    Anastasia

    Tuesday, May 10, 2016 5:49 PM
    Moderator
  • Yes that could work if it is a safe and accepted pattern.

    It already seems to pick up the dependency in terms of the outbound requests .

    Is that being instrumented elsewhere? 

    Tuesday, May 10, 2016 6:41 PM
  • I just came across this example which might do the job .... 

    http://stackoverflow.com/questions/32650200/how-to-use-a-custom-operationid-without-ending-up-with-memory-leaks-and-it-being

    TelemetryConfiguration config = TelemetryConfiguration.CreateDefault();
    
    var loggingContext = //new logging context with operation name and correlation id
    config.TelemetryInitializers.Add(new MiTelemetryInitialiser(loggingContext));
    
    while(!cancellationToken.IsCancellationRequested) {
        // 1. Get message from the queue
        // 2. Extract operation name and Correlation Id from the message
        TelemetryClient telemetryClient = new TelemetryClient(config);        
    
        // do work
    
        client.Flush();
    }

    Tuesday, May 10, 2016 6:46 PM
  • Great. Just make sure you create config once (or use TelemetryConfiguration.Active). It is expensive.

    Anastasia


    Tuesday, May 10, 2016 7:04 PM
    Moderator