locked
Scale out IoT solution after stream of data passes the hub RRS feed

  • Question

  • Hi,

    I am investigating Azure's IoT Suite for a very basic scenario of having field devices sending telemetry to Azure and storing data in azure storage tables/blobs/documentdb/whatever. I'd like the solution to be able to scale in future, just adding more resources and not requiring reenginering of the data pipeline for handling increasing data throughtput .

    The Hub itself is described as being able of ingesting a high mount of data, but I find more difficult to understand how to handle that data after it's moved out of the hub into other Azure based resources. As an example, Azure IoT documentation and samples show how to connect the IoT hub to some other backend services, normally passing through Event Hub. Then a reader gets the data using a Function and stores it in a table.

    My questions are:

    1. IoT Hub has some performance characteristics, but Event hub has different. I see no point in having a large ingestion point if the data then clogs up the rest of the pipeline. How is this solved? Can the event hub keep up to the event rate of the IoT hub?

    2. Some articles compare the IoT Hub and Event hub and their different scale targets. But them some solutions connect an Event Hub to the Iot Hub. But doesn't this just move the ingestion scale problems from the IoT hub to the Event hub?

    3. Samples show how to potentially ingest millions of events per day and store them using Functions and storage tables, but tables have very performance limits strictly related to partitions and account limits. It seems to me that proposed samples are totally ignoring this aspect. How are those handled if we want the solution not to drop data or clog up?

    4. After browsing some docs on IoT Hub I came up with the idea that events sent to the hub are automatically stored and preserved for a configurable amount of time, until a reader gets the data out. But in some solution I see an Event Hub connected to the IoT Hub. What's the point of doing this if the IoT Hub has the same purpose of the Event Hub but on a larger scale?

    Probably I am missing something, or samples I see in Azure's IoT documentation are just useless in showing a working ad scalable solution even for very simple scenarios...

    Thanks for enlightening me.

    Monday, May 8, 2017 9:38 AM