locked
IOT Hub Data Transformation? RRS feed

  • Question

  • i'm working with IOT hub on Azure via MQTT.   This is working well.  The deivces are publishing raw data that needs to be transformed before it’s persisted.

    In doing my research, it seems that the only way to make this happen would be to use the IOT Hub Protocol Gateway (https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-protocol-gateway).  I’m trying to avoid setting up another system in the middle so that we can lower our recurring costs, and keep or infrastructure from getting larger.

    is there a way to apply simpl conversions to the MQTT messages that does not require setting up the protocol gateway?

    Here is an example.

    The device publishes a Jason string like…

    {"temperature":125, "humidity":40}

    And I would want to run a transform from a dotnet library (or dll) that I have so that the following is saved.  For simplicity, we’ll assume I have these two simply functions.

    temperature=temperature*2

    Humidity=Humidity/2

    So after I published the original message, I would like for azure to act on the message below.

    {"temperature":250, "humidity":20}

    Is that possible to do without having me having to set up the portal gateway?  i'm trying to be as cookie cutter as possible so that i can leverage stream analytics, power BI, etc.

    Tuesday, May 2, 2017 12:37 PM

Answers

  • Yes, that's the correct way and place for VET (Validate-Enlarge-Transform), such as the first ASA job behind the IoT Hub.

    Basically there are 3 implementation ways for your scenario:

    1. using an ASA job + Function (ML service)

    2. Instead of the ASA job to use an Azure Function (Event Hub Trigger) with your c# library

    3. Instead of the ASA job to use a Worker Role with EventProcessorHost and your c# library

    Thanks

    Roman




    Wednesday, May 3, 2017 7:50 AM

All replies

  • Hi,

    - have a look at the document Communicate with IoT Hub using the MQTT protocol.

    Thanks

    Roman


    • Edited by Roman Kiss Tuesday, May 2, 2017 4:16 PM
    Tuesday, May 2, 2017 4:15 PM
  • Thanks for the replay.  I've very familiar with this and we have implemented most of this.  Unless I'm missing something, this does not address my transformation question.

    I have not problem getting data into and out of IOT hub. 

    Tuesday, May 2, 2017 7:27 PM
  • Thanks for the replay.  I've very familiar with this and we have implemented most of this.  Unless I'm missing something, this does not address my transformation question.

    I have not problem getting data into and out of IOT hub. 

    Wednesday, May 3, 2017 2:02 AM
  • Hi,

    - as you mentioned, your telemetry data has been successfully ingested into the Azure IoT Hub, so they are ready to flow through the data stream pipeline. The following screen snippet shows an example of this data stream pipeline:

    The first stream job is dedicated to Validate, Enlarge and Transformation of the telemetry data (raw data) with an output to the Event Hub configured for archiving feature. After that, the telemetry data continue to the business stream pipeline part such as ASA Job, etc.

    - see also this thread

    Thanks

    Roman




    • Edited by Roman Kiss Wednesday, May 3, 2017 3:33 AM
    Wednesday, May 3, 2017 3:23 AM
  • Thanks.  I've read about the ASA Jobs.  That seems to allow for you apply transforms using the available functions allowed with the Steam Analytics Query Language.  While that has value in some cases, it's limited in others. 

    Is there not a way to apply a simply transform from a c# library that I have?  Our transforms are more complex, and may apply complex math functions.

    From what I can see, we may need to leverage Azure ML (or some sort of custom worker role) in place of the ASA job step to clean our data and output to a stream for the consumer app.

    Am I on the right track here?

    Wednesday, May 3, 2017 5:37 AM
  • Yes, that's the correct way and place for VET (Validate-Enlarge-Transform), such as the first ASA job behind the IoT Hub.

    Basically there are 3 implementation ways for your scenario:

    1. using an ASA job + Function (ML service)

    2. Instead of the ASA job to use an Azure Function (Event Hub Trigger) with your c# library

    3. Instead of the ASA job to use a Worker Role with EventProcessorHost and your c# library

    Thanks

    Roman




    Wednesday, May 3, 2017 7:50 AM