locked
IoT Hub Data Ingestion - Mapping to a standard data format RRS feed

  • Question

  • I am putting together an IoT solution using Azure IoT Hub.

    What I would like to do is have an application that performs some operations on the data; the application would expect event data to be formatted using a standard model.

    What are the best practices for ingesting events via IoT Hub across a variety of devices which may use different models to format their payloads?

    I'm aware there's things like Stream Analytics and Function Apps that can do this kind of thing, but do people typically write their own mapping to handle each kind of device talking to their IoT Hub using these?
    Wednesday, June 21, 2017 4:05 PM

All replies

  • Hi,

    The following screen snippet is an example of the typical IoT Stream Pipeline:

    The first stream job in the pipeline is dedicated for VET (Validation-Enrichment-Transformation) mediation using a reference data from the Blob Storage and the event mediator in the User Defined Functions (for simple primitives) and/or Machine Learning models. After this mediation the telemetry data can be ingested into the Event Hub and archive in the Blob Storage.   

    Thanks

    Roman



    • Edited by Roman Kiss Wednesday, June 21, 2017 7:10 PM
    Wednesday, June 21, 2017 7:10 PM
  • Hi Roman,

    Thanks for you help, and speedy response! I have a couple of follow up questions:

    What do you mean by this line:

    "using a reference data from the Blob Storage"

    The Blob storage appears to be separate from the IoT data stream; what is this reference data?

    "the event mediator in the User Defined Functions"

    As I understand, this means some custom JS-style parsing as an ASA function?

    Say I had (N) Gateways sending events to my IoT Hub; each Gateway sending a payload in their own format... would it be typical to have (N) User Defined Functions, each parsing the data stream in a different way, based on some meta-data attached to the payload?

    Thanks!


    • Edited by Stevie89 Thursday, June 22, 2017 10:42 AM
    Thursday, June 22, 2017 10:42 AM
  • Hi @Stevie89,

    - Reference Data is an optional (static) input for correlation and lookups used in the ASA stream process.

    - Yes, the ASA query can be extended by stateless custom functions written in JavaScript (UDF). The UDF is used for custom transformation of the input data to output data.

    - There is a limit for maximum number of functions per job such as 60, see this.

    Thanks

    Roman



    • Edited by Roman Kiss Thursday, June 22, 2017 7:03 PM
    Thursday, June 22, 2017 7:02 PM