none
Azure Function with Event Hub Listener is missing events RRS feed

  • Question

  • How to handle missing events and make the azure functions with event hub trigger more reliable?

    The problem we are seeing is where we have around 10,000 events being pushed to the event hub and the listener is missing few events.

    Monday, May 29, 2017 4:31 AM

Answers

  • Azure Functions uses the EventProcessorHost (EPH) to retrieve messages from your Event Hub.  Each Function instance is backed by a single EPH.  An EPH will retrieve messages in batches from the last known successful checkpoint.  Upon successful execution of your Function code, the EPH checkpoints the batch for the partition as being successfully processed and those messages will never be retrieved again.  If your Function execution fails due to exceptions, timeouts or partition lease lost, the next Function instance that manages to get a lease on the partition will start retrieving messages from the last known checkpoint.

    There are a few possibilities that may explain why events are missing:

    1. The missing events are processed by your Function code but not forwarded to the downstream processing pipeline.  Kindly verify that every message processed in your Function code is accounted for either through logging, output binding or any other custom client store.

    2. You have try-catch logic in your Function code that handles one or more exceptions.  While this defensive programming approach is recommended, the side-effect is that it allows your Function code to complete its execution successfully, checkpoint the entire batch and move on to retrieve the next batch.  If this applies to your Function, verify that messages that cause exceptions are also accounted for. 

    3. The retention policy for your messages has expired.  The retention policy is configurable through Event Hub portal.  When the retention policy is enforced for the message block, all the messages in the block that is outside of the retention window will be discarded by Event Hub.

    Wednesday, May 31, 2017 6:12 PM