none
Periodically receiving numerous response errors (errors: [{index: XX statusCode: 500, message: "Processing Error"}, {}, ...]) returned while sending batch telemetry

    Question

  • While sending a batch of telemetry (100+ items), I have been recently receiving errors in the response payload for some of the telemetries.  The errors I am receiving in the response payload are always:

    {index: XX statusCode: 500, message: "Processing Error"}

    What does this mean?  Is the AI server temporarily refusing request?  Throwing errors?  The errors happen on Request, Dependencies, and Exceptions.  When I submit them to the validate endpoint (https://dc.services.visualstudio.com/v2/validate), they all succeed.

    Here is a sample of what I am sending:

    REQUEST:

    POST https://dc.services.visualstudio.com/v2/track

    BODY (formatted and truncated for clarity)

    {
        "ver": 1,
        "name": "Microsoft.ApplicationInsights.YYYYY49f08787d0eXXXXXX.RemoteDependency",
        "time": "2016-07-12T18:42:17.861Z",
        "sampleRate": 100,
        "tags": {
            "ai.application.ver": "1.0.9-build.1",
            "ai.application.build": "application server.",
            "ai.device.id": "node",
            "ai.device.machineName": "Mac-mini.local",
            "ai.device.os": "Darwin",
            "ai.device.osVersion": "15.5.0",
            "ai.device.type": "server",
            "ai.device.osArchitecture": "x64",
            "ai.device.osPlatform": "darwin",
            "ai.internal.sdkVersion": "node:0.15.14",
            "ai.operation.id": "0F57808D-31B3-4214-B36A-D8238FAD21FD",
            "ai.operation.name": "CREATE Rows",
            "ai.operation.parentId": "0F57808D-31B3-4214-B36A-D8238FAD21FD",
            "ai.operation.rootId": "0F57808D-31B3-4214-B36A-D8238FAD21FD"
        },
        "data": {
            "baseType": "RemoteDependencyData",
            "baseData": {
                "ver": 2,
                "kind": 0,
                "dependencyKind": 2,
                "success": true,
                "dependencySource": 0,
                "properties": {
                    "schemaId": "m",
                    "tableId": "g",
                    "tableName": "participants",
                    "commandName": "findDatasetTableRows"
                },
                "measurements": {
                    "datasetId": 1
                },
                "id": "D786E371-5323-485E-B301-388E851212D7",
                "name": "DatasetCrud.findDatasetTableRows",
                "commandName": "findDatasetTableRows",
                "value": 12.187575817108154,
                "dependencyTypeName": "DatasetCrud",
                "async": true
            }
        },
        "appVer": "1.0.9-build.1",
        "iKey": "XXXXXXXX",
        "os": "Darwin",
        "osVer": "15.5.0",
        "seq": "Yg9hiY+FmegEPv0LdZ7KXQ:3001"
    }

    RESPONSE:

    { errors: Array[79], itemsAccepted: 171, itemsReceived: 250}

    Each of the error elements are:

    {index: XX statusCode: 500, message: "Processing Error"}

    Tuesday, July 12, 2016 7:17 PM

Answers

  • 500 is s server side error. Server side may have intermittent issues while storing data. SDK will save this data locally and resend if you use ServerTelemetryChannel. If you use InMemoryChannel data will be lost.

    Anastasia

    • Marked as answer by tmcgannon Thursday, July 14, 2016 7:00 PM
    Wednesday, July 13, 2016 5:17 PM
    Moderator

All replies

  • 500 is s server side error. Server side may have intermittent issues while storing data. SDK will save this data locally and resend if you use ServerTelemetryChannel. If you use InMemoryChannel data will be lost.

    Anastasia

    • Marked as answer by tmcgannon Thursday, July 14, 2016 7:00 PM
    Wednesday, July 13, 2016 5:17 PM
    Moderator
  • Thanks for your answer.  As you can see from my example, I am using the AI Node client which does not handle these type of errors so I will have to deal with them myself.

    Are the potential list of errors documented anywhere?  Since the Node client is not complete nor well documented, I have been looking at the other client implementations (.net, asp.net, java, etc) and have found their documentation lacking as well - but I can at least look at the .net version for how they handle errors by reading the code.

    Thursday, July 14, 2016 7:12 PM
  • Right now status codes are not documented.

    You can take a look here: https://github.com/Microsoft/ApplicationInsights-dotnet/tree/master/src/TelemetryChannels/ServerTelemetryChannel/Shared/Implementation

    Files: ErrorHandlingTransmissionPolicy.cs, ThrottlingTransmissionPolicy.cs, PartialSuccessTransmissionPolicy.cs

    I am creating flowcharts for these 3 classes and will post it in the repo mentioned above. It should be available early next week.


    Anastasia

    Thursday, July 14, 2016 7:55 PM
    Moderator
  • Along those same lines, is it documented anywhere what 'ai.operation.parentId', 'ai.operation.rootId', and 'ai.operation.correlationVector' are specifically intended for? The only mention of them states 'support end-to-end telemetry items correlation'.  They don't appear to be used at all in the Azure portal AI interface - as far as I can tell it only uses 'ai.operation.id' to join Requests to one level of Dependencies.  I've used them in the analytics.applicationinsights.io interface while creating my own queries that join Requests to Dependencies to child Dependencies. But its quite cumbersome to join request.id or dependency.id to dependency.parentId or dependency.rootId and get useful output.  What I'm trying to accomplish is a profiler-like output with hierarchical view of my layers and the time spent in each layer (In-coming Requests -> Business Layer -> CRUD layer -> Table Storage layer).  Further, it would be nice to tie them into AI telemetry from client requests presumably using the correlationVector.
    Thursday, July 14, 2016 8:52 PM