locked
IoT Hub data ingestion: custom application properties RRS feed

  • Question

  • Hi, is it possible to ingest the custom message properties in addition to the message body and if yes, how?
    Thursday, April 16, 2020 1:39 PM

All replies

  • Can you be more specific with regard to the custom message properties by providing an example? When configuring the connection to the ADX target ingestion table from IoT Hub (link) you have the option to include Event system properties. Please see: System Properties of D2C IoT Hub messages

    Property Description User Settable? Keyword for
    routing query
    message-id A user-settable identifier for the message used for request-reply patterns. Format: A case-sensitive string (up to 128 characters long) of ASCII 7-bit alphanumeric characters + {'-', ':', '.', '+', '%', '_', '#', '*', '?', '!', '(', ')', ',', '=', '@', ';', '$', '''}. Yes messageId
    iothub-enqueuedtime Date and time the Device-to-Cloud message was received by IoT Hub. No enqueuedTime
    user-id An ID used to specify the origin of messages. When messages are generated by IoT Hub, it is set to {iot hub name}. Yes userId
    iothub-connection-device-id An ID set by IoT Hub on device-to-cloud messages. It contains the deviceId of the device that sent the message. No connectionDeviceId
    iothub-connection-module-id An ID set by IoT Hub on device-to-cloud messages. It contains the moduleId of the device that sent the message. No connectionModuleId
    iothub-connection-auth-generation-id An ID set by IoT Hub on device-to-cloud messages. It contains the connectionDeviceGenerationId (as per Device identity properties) of the device that sent the message. No connectionDeviceGenerationId
    iothub-connection-auth-method An authentication method set by IoT Hub on device-to-cloud messages. This property contains information about the authentication method used to authenticate the device sending the message. No connectionAuthMethod



    Saturday, April 18, 2020 12:30 AM
  • An example would be the simDevice in the Azure-Samples. In this we set the "temperatureAlert" property:

    // Add a custom application property to the message.
    // An IoT hub can filter on these properties without access to the message body.
    message.Properties.Add("temperatureAlert", (currentTemperature > 30) ? "true" : "false");

    When i monitor the built-in event endpoint my message looks like:

    [IoTHubMonitor] [9:18:28 AM] Message received from [simDevice]:
    {
      "body": {
        "temperature": 23.900632392103148,
        "humidity": 75.06975383268193
      },
      "applicationProperties": {
        "temperatureAlert": "false"
      }
    }
    

    If i map the root path of my message ("path":"$") i only get the content of the body. how can i ingest what would be the temperatureAlert in this example?

    Saturday, April 18, 2020 7:38 AM
  • Can you try the following json ingestion mapping as per the following instructions: Ingest data from IoT Hub into Azure Data Explorer

    .create table TestTable ingestion json mapping "TestMapping"
    '['
    '  {"column": "temperature", "Properties":{"Path":"$.body.temperature"}},'
    '  {"column": "humidity", "Properties":{"Path":"$.body.humidity"}},'
    '  {"column": "temperatureAlert", "Properties":{"Path":"$.applicationProperties.temperatureAlert"}}'
    ']'

    The example in the documentation with the custom application property added.

    .create table TestTable ingestion json mapping 'TestMapping'

    '['

    '{"column":"temperature","path":"$.body.temperature","datatype":"real"}'

    '{"column":"humidity","path":"$.body.humidity","datatype":"real"},'

    '{"column":"temperatureAlert","path":"$.body.applicationProperties.temperatureAlert","datatype":"real"},'

    ']'

    If you have the ADX desktop client installed, you can Install and use LightIngest to test a sample file to workout the correct mapping, as I do not have a concrete example to provide you.

    Regards,

    Mike

    Wednesday, April 22, 2020 12:23 AM
  • I already tried multiple mappings. The problem is that "$" is the path to the body-object not the message-object.

    When i use:

    .create table TestTable (Event: dynamic)

    and

    .create table TestTable ingestion json mapping 'RawEventMapping' '[{"column":"Event","path":"$","datatype":"","transform":null}]'

    my Events look like: {"temperature":28.6557508975527,"humidity":78.324339603224928}

    "path":"$.body.xyz" doesn't exist/is empty because $ is body.

    I would have to map the parent-object of "$".

    Wednesday, April 22, 2020 8:10 AM

  • Thank you for this additional detail. I believe you are going to need to use the Avro file format and Azure Data Lake Analytics to extract any properties that fall outside the the message body path. Please see: Query Avro data by using Azure Data Lake Analytics

    System properties are an exception to this.

    Regards,

    Mike

    Wednesday, April 22, 2020 3:57 PM