locked
Process Events: Node shutdown due to high memory utilization RRS feed

  • Question

  • Stream analytics job taking two small <g class="gr_ gr_38 gr-alert gr_spell gr_inline_cards gr_run_anim ContextualSpelling ins-del multiReplace" data-gr-id="38" id="38">json</g> objects every 10 minutes as input failed unexpectedly with Process Events: Node shutdown due to high memory utilization

    SU % spiked, with no associated increase in input or output events.  SU setting is 1.    Can anyone provide advice on why this could have failed?   

    Thanks,

    Ed

    

    Friday, August 10, 2018 5:08 PM

All replies

  • Hi Ed,

    Can you please provide the following details about your job:

    - Region

    - Input(s) and Output(s) types (e.g., Event Hub, Power BI)

    - The query you're using

    Thanks,

    Sam


    This posting is provided "AS IS" with no warranties, and confers no rights.

    Friday, August 10, 2018 10:07 PM
  • Hi,

    From the screenshot, it looks like the job was not running for few days. Did you have to manually restart the job after Aug 10? Can you provide details of your subscription id and resource name/id to us via DM on Twitter?

    https://twitter.com/AzureStreaming

    Monday, August 13, 2018 4:26 PM
  • <g class="gr_ gr_221 gr-alert gr_gramm gr_inline_cards gr_run_anim Grammar only-ins doubleReplace replaceWithoutSep" data-gr-id="221" id="221">Region</g> is East US 2

    Data is coming into IoT hub every 10 minutes and going into stream analytics, this is the only input.  The query is sending each sample to both Azure SQL database and Table storage.  These are the only 2 outputs.

    Could this issue possibly be related to the "TIMESTAMP BY EventEnqueuedUtcTime" statement in the query?

    This is the query that is running in SA:

    /*

    This query supports data coming from the gateway in the following JSON format:

    { "data":

        [ 

            {

                "device": "{{{PARTICLE_DEVICE_ID}}}",

                "ts": "{{{PARTICLE_PUBLISHED_AT}}}",

                "ch": "ain0",

                "name": "Differential Pressure",

                "eu": "inWC",

                "value": "{{{dp}}}" 

            },

            {

                "device": "{{{PARTICLE_DEVICE_ID}}}",

                "ts": "{{{PARTICLE_PUBLISHED_AT}}}",

                "ch": "sysRunning",

                "name": "Collector Running",

                "eu": "",

                "value": "{{{r}}}"  

            },

            {

                "device": "{{{PARTICLE_DEVICE_ID}}}",

                "ts": "{{{PARTICLE_PUBLISHED_AT}}}",

                "ch": "sysTemp",

                "name": "Gateway Temperature",

                "eu": "F",

                "value": "{{{f}}}"  

            },

            {

                "device": "{{{PARTICLE_DEVICE_ID}}}",

                "ts": "{{{PARTICLE_PUBLISHED_AT}}}",

                "ch": "sysRSSI",

                "name": "Cellular RSSI",

                "eu": "dB",

                "value": "{{{c}}}"  

            }

        ]

    }

    */

    WITH

    Gw2Samples AS

    (

    SELECT System.Timestamp AS datetime, arrayElement.ArrayValue.device AS gatewayid, arrayElement.ArrayValue.ch AS channelid, arrayElement.ArrayValue.name AS name, arrayElement.ArrayValue.eu AS eu, 'avg1day' AS calculation, 'Ok' AS status, TRY_CAST(arrayElement.ArrayValue.value AS float) as value

    FROM [gwiothub2-in] as event TIMESTAMP BY EventEnqueuedUtcTime

    CROSS APPLY GetArrayElements(event.data) AS arrayElement 

    ),

    Raw AS

    (

    SELECT CONCAT ( arrayElement.ArrayValue.device, '-', arrayElement.ArrayValue.ch, '-', arrayElement.ArrayValue.name) AS partitionKey, system.timestamp as rowKey, 'raw' AS Type, arrayElement.ArrayValue.device, arrayElement.ArrayValue.ch, arrayElement.ArrayValue.name, arrayElement.ArrayValue.eu, TRY_CAST(arrayElement.ArrayValue.value AS float) as value

    FROM [gwiothub2-in] as event TIMESTAMP BY EventEnqueuedUtcTime

    CROSS APPLY GetArrayElements(event.data) AS arrayElement

    )

    SELECT * INTO [table-out-raw] FROM Raw

    SELECT * INTO [sqldb-out-runtime] FROM Gw2Samples

    Thx,

    Ed

    Monday, August 13, 2018 4:45 PM
  • Yes, the job was running for several days, then quit for a few days before we noticed it.  I did have to manually restart it after Aug 10, and it has been running ok since then.    I can send you id details, but don't have twitter, do you have <g class="gr_ gr_428 gr-alert gr_gramm gr_inline_cards gr_run_anim Grammar only-ins doubleReplace replaceWithoutSep" data-gr-id="428" id="428">email</g> I can send it to or <g class="gr_ gr_429 gr-alert gr_gramm gr_inline_cards gr_run_anim Grammar multiReplace" data-gr-id="429" id="429">other method</g>?
    Monday, August 13, 2018 4:49 PM
  • Glad to know that the job is running fine after the restart. Apart from Twitter DM, the other way would be to open a support ticket on the portal and share the details with us.

    Thanks,

    Sid

    Tuesday, August 14, 2018 7:25 PM
  • I just sent you the id and details via twitter DM from @AnalyticsCharge
    Tuesday, August 14, 2018 11:21 PM