locked
Streaming Unit consumes more and more memory even when nothing is being processed RRS feed

  • Question

  • Hey,

    I'm just getting started with ASA and I've noticed some weird behavior with my SU Utilization %.

    Here is a picture of my performance monitor for the last 12 hours:

    I can't put in a picture due to my account not being verified, so for now this URL will have to do: https://social.msdn.microsoft.com/Forums/getfile/716272

    This shows that ASA has not been processing any events, yet the SU % Utilization keeps climbing to 99% and then drops back down to 75%.


    I checked the Operations Log for my ASA, and the result was somewhat worrying - apparently it started consuming too much memory and was terminated / restarted.

    I can't put in a picture due to my account not being verified, so for now this URL will have to do: https://social.msdn.microsoft.com/Forums/getfile/716275

    The timestamps correspond to when SU % Utilization dropped in the monitor.


    This is what the message says:

    Message:
    Resource usage is over the capacity for one or more of the query steps. Event processing may be delayed or stop making progress. This may be a result of large window in your query, large events in your input, large out of order tolerance window, or a combination of the above. Please try to partition your query, or break down your query to more steps, and add Streaming Unit resources from the Scale tab to avoid such condition., : 
    
    Message Time:
    9/2/2015 9:12:25 AM UTC
    
    Microsoft.Resources/EventNameV2:
    streamingNode0
    
    Microsoft.Resources/Operation:
    Process Events: Node shutdown due to high memory utilization.

    My queries are very simple:

    SELECT
        *
    INTO
        [EventPassthroughOutput]
    FROM
        [EventInput]
        
    SELECT 
    	Id,
    	Address.Street as Address_Street,
    	Address.Street2 as Address_Street2,
    	Address.State as Address_State,
    	Address.PostalCode as Address_PostalCode,
    	Address.City as Address_City,
    	Subscription.WhiteCount as Subscription_WhiteCount,
    	Subscription.RedCount as Subscription_RedCount,
    	Subscription.SubstatusId as Subscription_SubstatusId,
    	SourceTypeId,
    	CreateDate,
    	LeadSource,
    	Email,
    	EventProcessedUtcTime,
    	EventEnqueuedUtcTime,
    	PartitionId
    INTO
        [UserEvents]
    FROM
        [EventInput]


    Does anyone know what's going on?

    Thanks,

    Felix


    • Edited by FAKastner Wednesday, September 2, 2015 7:38 PM Images
    Wednesday, September 2, 2015 7:25 PM

Answers

  • Thank you for reporting this issue! We will address it promptly. Please note, that this issue is specific to the condition where there is no input. It should no affect jobs that are actually processing data.
    • Marked as answer by Janet Yeilding Thursday, October 8, 2015 11:36 PM
    Thursday, September 3, 2015 12:28 AM

All replies

  • You said no actual traffic (events) were going through? Is there any chance someone from test was load testing by accident there? Just checking

    Are these separate jobs (the two queries?)

    I don't specifically see any time window on your query, do you have a window at all, and maybe just didn't show it here?

    Wednesday, September 2, 2015 8:06 PM
  • Hey, thank you for your reply.

    I was testing a few hours prior to what the chart shows ( at 6-7 PM ), after that I am sure no traffic went through.

    At my company, I am the only dev working with Event Hubs and ASA at the moment, and my work in progress source code is still local for the time being. 

    The two queries are in the same job, and I copied them as they are. I am using ASA for the purpose of routing the raw event data to multiple outputs, so I have no need for windows.

    EventPassthroughOutput is a Blob Output that just logs the raw event, while UserEvents is a SQL Data Warehouse table.

    There's nothing else in the job - 1 Input, 2 Queries, 2 Outputs. 

    Thanks,

    Felix

    Wednesday, September 2, 2015 8:22 PM
  • Can you please provide Correlation Id? You can find it in the same Operations Log error message that you posted.
    Wednesday, September 2, 2015 8:59 PM
  • Of course. Apologies for not providing it from the beginning - I wasn't sure if it was safe to share.

    Correlation ID:
    5058800d-ad48-4e75-9de7-e8733ffb10be

    It seems to be the same for all the errors.

    Thanks,

    Felix

    Wednesday, September 2, 2015 9:08 PM
  • Thank you for reporting this issue! We will address it promptly. Please note, that this issue is specific to the condition where there is no input. It should no affect jobs that are actually processing data.
    • Marked as answer by Janet Yeilding Thursday, October 8, 2015 11:36 PM
    Thursday, September 3, 2015 12:28 AM