locked
Receive Events: Input processor failure RRS feed

  • Question

  • Our Stream Analytics job has been getting the following errors sporadically the last few weeks.

    I have restarted the Stream Analytics job, but that didn't seem to make a difference.

    The Event Hub shows no errors.

    Any idea how I should troubleshoot?

    {
        "Environment": "Prod",
        "Region": "North Europe",
        "category": "Execution",
        "level": "Error",
        "operationName": "Receive Events: Input processor failure",
        "properties": "{\"Error\":null,\"Message\":\"We are experiencing issues receiving events from input 'sigfoxincomingeventhub',
    partition 0 right now. We will try again soon. Details: - One or more errors occurred.
    - The operation was canceled. Reference:200e898a029746d2b6604bd6dfa1a7be_G26, TrackingId:N\\/A,
    Timestamp:4\\/30\\/2019 3:11:40 AM
    - The operation was canceled.
    ",\"Type\":\"InputProcessorFailure\",\"Correlation ID\":\"ddee7b2f-990e-4009-967a-ba9136777bb2\"}", "resourceId": "/SUBSCRIPTIONS/xxx/RESOURCEGROUPS/xxx/PROVIDERS/MICROSOFT.STREAMANALYTICS/STREAMINGJOBS/SIGFOXTOSQL", "status": "Failed", "time": "2019-04-30T03:11:41.1994828Z" }

    { "Environment": "Prod", "Region": "North Europe", "category": "Execution", "level": "Error", "operationName": "Receive Events: Input processor failure", "properties": "{\"Error\":null,\"Message\":\"We are experiencing issues receiving events from input 'sigfoxincomingeventhub',
    partition 1 right now. We will try again soon. Details: - One or more errors occurred.
    - This might indicate that entity associated with the operation does not exist or it has been deleted.
    Make sure your Event Hub namespace [sb:\\/\\/xxx.servicebus.windows.net\\/] still exists.
    - An error occurred during communication with '41060ab01b94481d91bed26dc64cc0f3_B6'.
    Check the connection information, then retry.\\u000d\\u000a
    - The AMQP object session2929 is aborted.\\u000d\\u000a\",

    \"Type\":\"InputProcessorFailure\",\"Correlation ID\":\"ddee7b2f-990e-4009-967a-ba9136777bb2\"}", "resourceId": "/SUBSCRIPTIONS/xxx/RESOURCEGROUPS/xxx/PROVIDERS/MICROSOFT.STREAMANALYTICS/STREAMINGJOBS/SIGFOXTOSQL", "status": "Failed", "time": "2019-04-30T03:21:48.4400122Z" }

    {
        "Environment": "Prod",
        "Region": "North Europe",
        "category": "Execution",
        "level": "Error",
        "operationName": "Receive Events: Input processor failure",
        "properties": "{\"Error\":null,\"Message\":\"We are experiencing issues receiving events from input 'sigfoxincomingeventhub',
        partition 1 right now. We will try again soon.
        Details: - One or more errors occurred.
        - The operation was canceled. Reference:405c4e3c9f7a425a91a0bbf58a068275_G21, TrackingId:N\\/A,
        Timestamp:4\\/26\\/2019 8:45:20 AM
        - The operation was canceled.",
        \"Type\":\"InputProcessorFailure\",\"Correlation ID\":\"c82d4405-90c1-4c98-8aa8-206a982db113\"}",
        "resourceId": "/SUBSCRIPTIONS/xxx/RESOURCEGROUPS/xxx/PROVIDERS/MICROSOFT.STREAMANALYTICS/STREAMINGJOBS/SIGFOXTOSQL",
        "status": "Failed",
        "time": "2019-04-26T08:45:20.7505319Z"
    }
    
    {
        "Environment": "Prod",
        "Region": "North Europe",
        "category": "Execution",
        "level": "Error",
        "operationName": "Receive Events: Input processor failure",
        "properties": "{\"Error\":null,\"Message\":\"We are experiencing issues receiving events from input 'sigfoxincomingeventhub',
        partition 0 right now. We will try again soon.
        Details: - One or more errors occurred.
        - This might indicate that entity associated with the operation does not exist or it has been deleted.
        Make sure your Event Hub namespace [sb:\\/\\/xxx.servicebus.windows.net\\/] still exists.
        - An error occurred during communication with '03183927cd924c709cf1f0ebeb67b70a_B39'.
         Check the connection information, then retry.
         - The AMQP object session2777 is aborted.
         This message has occurred 6 times in past 10 minutes\",
         \"Type\":\"InputProcessorFailure\",\"Correlation ID\":\"ddee7b2f-990e-4009-967a-ba9136777bb2\"}",
        "resourceId": "/SUBSCRIPTIONS/xxx/RESOURCEGROUPS/xxx/PROVIDERS/MICROSOFT.STREAMANALYTICS/STREAMINGJOBS/SIGFOXTOSQL",
        "status": "Failed",
        "time": "2019-04-30T02:46:14.9832720Z"
    }
    





    Tuesday, April 30, 2019 7:32 AM

Answers

  • Hi,

    These errors are often transient and do not cause data loss. The reason Stream Analytics shows them is to provide insights in case processing might be delayed.  If you do not observe increasing watermark delay they should not be of concern.

    We understand that the messages might be confusing and are revising the content, circumstances and frequency with which we output them. 

    Thanks,
    Kati Iceva, 

    Azure Stream Analytics


    This posting is provided "AS IS" with no warranties, and confers no rights.

    Saturday, May 18, 2019 12:23 AM

All replies

  • Hi mjose,

    This issue looks strange. For a deeper investigation and immediate assistance on this issue, if you have a support plan you may file a support ticket, else could you send an email to AzCommunity@Microsoft.com with your Subscription ID and thread link to this post, and I will enable a one-time free support request for your subscription. 

    Please reference this forum thread in the subject: “Receive Events: Input processor failure”. Thank you for your persistence.

    Tuesday, April 30, 2019 8:12 AM
  • Thanks. We're on a Basic subscription so we don't have a support plan but I've sent an email to AzCommunity.
    Hopefully we'll be able to provide the community with an update in this thread later.
    Tuesday, April 30, 2019 10:14 AM
  • Hi mjose,

    Thank you for reaching out.

    We have enabled for one-time free technical support.

    I hope you have received one on one support to work towards a resolution on this matter.

    Tuesday, April 30, 2019 10:20 AM
  • Support suggested I delete and recreate the Stream Analytics Input for the affected Event Hub.

    I have stopped the Stream Analytics job, removed the Event Hub input, added the Event Hub input with the same parameters and started the Stream Analytics job.

    Since this problem has been occurring sporadically over more than a month, there will be some time before we can guess whether this action fixed the problem. I'll report back when I have something to share.

    Thursday, May 2, 2019 8:03 AM
  • We are also experiencing this issue and we have started observing it since April 23.

    Based on our initial investigation, it is not caused by the following:
    1. Parallel Deployment
    2. EventHub Threshold limit exceeded
    3. Low Input Volume - which usually triggers operational type error "Receive Events: Input processor failure"
    Other probable causes:
    1. Azure Services Incidents - happening around the timeline where the Runtime Errors was encountered. 
    Our Timeline (UTC) of Eventhub-ASA Runtime Errors (Receive Events: Input processor failure):
    • Tuesday, April 23, 2019 | 1:29 | 1:32 |1:57 | 8:21
    • Tuesday, April 30, 2019 | 22:13 | 22:57 | 23:40
    • Wednesday, May 1, 2019 | 5:00 | 5:10 | 5:22 | 6:24 | 6:26 | 7:34 | 8:26 | 8:30 | 10:23
    • Wednesday, May 8, 2019 | 6:39 | 6:54 | 8:25
    • Monday, May 13, 2019 | 4:32 | 4:48 | 5:27 | 6:33 | 8:22 | 8:25


    • Edited by Angelica Quebec Thursday, May 16, 2019 8:08 AM Added/Updated Details
    Wednesday, May 15, 2019 12:38 PM
  • Thanks for your input Angelica. Good to see that we're not the only ones experiencing this problem.

    We tried to delete and recreate the Stream Analytics Input for the affected Event Hub on recommendation from Support, but that didn't make a difference. We still get occasional errors.

    Wednesday, May 15, 2019 2:21 PM
  • You're welcome Mikael. Noted also on your update.

    Btw, if I may ask, how does these occasional errors impacted your processes? Do you experience data loss in those timings?
    Thursday, May 16, 2019 9:20 AM
  • On 2019-05-11 we had "Input processor failure" events at T00:27:28 and T00:40:39.

    Around this time, Stream Analytics received messages at the following timestamps
    Epoch date     Human readable date (GMT)
    1557534324    2019-05-11T00:25:24+00:00
    1557534388    2019-05-11T00:26:28+00:00
    1557534767    2019-05-11T00:32:47+00:00
    1557535092    2019-05-11T00:38:12+00:00
    1557535098    2019-05-11T00:38:18+00:00
    1557535255    2019-05-11T00:40:55+00:00
    1557535303    2019-05-11T00:41:43+00:00
    1557535346    2019-05-11T00:42:26+00:00
    All these are present in the target database.

    Unfortunately, we don't have any logging on messages in the step before Stream Analytics, and we're only processing a low amount of messages so I can't tell whether the "Input processor failure" causes data loss.
    Thursday, May 16, 2019 9:48 AM
  • Hi,

    These errors are often transient and do not cause data loss. The reason Stream Analytics shows them is to provide insights in case processing might be delayed.  If you do not observe increasing watermark delay they should not be of concern.

    We understand that the messages might be confusing and are revising the content, circumstances and frequency with which we output them. 

    Thanks,
    Kati Iceva, 

    Azure Stream Analytics


    This posting is provided "AS IS" with no warranties, and confers no rights.

    Saturday, May 18, 2019 12:23 AM
  • Thank you Kati
    Saturday, May 18, 2019 11:49 AM