none
EventHub feedback on send? (Or: Reliability?) RRS feed

  • Question

  • We're currently switching to EventHub coming from Azure Queues in our project. Since reliability is a big factor for us, I am confused as to how EventHub works / is placed in the market.

    Currently I'd like to know what approaches there are for sending EventData to EventHub. I can't pass a timeout (Send / SendBatch will block for eternity when an EventHub is disabled). I don't get any information back on async calls.

    Basically I can only catch errors, but the errors that are possibly thrown are not specified in documentation. Any hints / pointers would be appreciated.

    Friday, January 9, 2015 1:32 PM

Answers

  • EventHub is a complete messaging system that has some unique characteristics. You really need to study it very carefully to see if it meets your requirements.

    http://azure.microsoft.com/en-us/documentation/services/service-bus/

    Having said that, I would like to answer other questions you raised here:

    1. Messages are stored in storage blobs, so long as your device can send the messages there, it is kept there for retrieval until expiry. It is as simple as it can get. So unless you have heard anything otherwise, you should trust its reliability.

    2. When you send a message to eventhub, if it doesn't arrive, the send will be timed-out and you do get an exception.

    3. It is not unusual that error messages that you get are not 100% clear enough, and sometimes a bit of detective work is required to figure out what exactly is happening, and sometimes it is just about lack of understanding of the mechanism, but that should not deter you from not using a suitable platform for your solution.


    Frank

    • Marked as answer by Stijn Spijker Monday, January 12, 2015 3:32 PM
    Friday, January 9, 2015 4:25 PM

All replies

  • EventHub is a complete messaging system that has some unique characteristics. You really need to study it very carefully to see if it meets your requirements.

    http://azure.microsoft.com/en-us/documentation/services/service-bus/

    Having said that, I would like to answer other questions you raised here:

    1. Messages are stored in storage blobs, so long as your device can send the messages there, it is kept there for retrieval until expiry. It is as simple as it can get. So unless you have heard anything otherwise, you should trust its reliability.

    2. When you send a message to eventhub, if it doesn't arrive, the send will be timed-out and you do get an exception.

    3. It is not unusual that error messages that you get are not 100% clear enough, and sometimes a bit of detective work is required to figure out what exactly is happening, and sometimes it is just about lack of understanding of the mechanism, but that should not deter you from not using a suitable platform for your solution.


    Frank

    • Marked as answer by Stijn Spijker Monday, January 12, 2015 3:32 PM
    Friday, January 9, 2015 4:25 PM
  • Hi Stijn,

    Queues and EventHub seems to be very similar, but there are few major differences.

    If I might express it this way, queues are designed for reliable messaging and hubs for speed. Queues commit to the caller successfully when message is persisted in durable store (by default, but it can be changed) and hubs commit when the message is received on the node. The later one is faster, because it does not require persisting, but it is in many messaging scenarios not reliable enough.

    Another difference is related to partitioning, which is default model when using EventHubs.

    Anyhow, if you want to build messaging system, which leverages integration patterns, the better choice would be probably queues/topics.

    If you are building IoT system or similar, where lot of data is in focus, then EventHubs will be probably better choice.

    However, if your system is messaging system requiring integration patterns, but needs a "speed" of EventHubs, it can be more difficult to make a right choice. In that case I would probably start with queues/topics with partitioning.

    Regarding documentation, I agree that it can be better. If you follow changes in last 6 months especially in context of EventHubs you will notice huge positive change.

    Regarding your "timeout" question, take a look on EventHubSender class property RetryPolicy.
    This one might be helpful: http://msdn.microsoft.com/en-us/library/azure/microsoft.servicebus.messaging.eventhubsender.aspx


    Damir Dobric
    developers.de
    daenet.de
    daenet.eu
    daenet.com

    Friday, January 9, 2015 9:47 PM
  • Thanks for these in-depth answer, they are very useful!

    Regarding Queues/Topics v.s. EventHub. We are focussing on a lot of data, our current goal is 20.000 messages / second, but we should be able to scale up to 100.000 messages/second in the coming 2 years.

    EventHub has all the clever partioning tricks baked in, where Queues do not. Also, Queues might be more reliable, but in the end, Queues can crash as well, thus you will always need to build the backup scenario.

    Also, EventHub does sticky partioning (a certain source / partitionkey will always land on the same partition). At the end of some of our architecture, we need to implement filters that require messages to be in order. By leveraging sticky partitioning we can avoid having to sort data later (which is very expensive).

    This is why we chose EventHub :).


    Monday, January 12, 2015 3:31 PM