locked
Consume IoT Hub file upload notifications as triggered events RRS feed

  • Question

  • Hey,

    So I started using the file upload facitiliy of the IoT Hub. From what I understand, the notification arrives to "messages/servicebound/fileNotifications". 

    I want to have an Azure Function that acts upon these file notifications. What would be an efficient way to do that ?

    The only example I could find for consuming file notifications, is this one : https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-csharp-csharp-file-upload#receive-a-file-upload-notification ( Unverified account, sorry ). 

    Which uses a sort of async busy-wait. However, wouldn't it be more efficient if I were just implement the file upload process with the request/SAS token/etc myself and have the client send me the file notification as a normal device message ( with some differntiating field) , since the normal device message endpoint behaves like a ServiceBus, which I can be triggered upon. 

    Or is this ServiceBus endpoint I'm exposed to is just the exact same busy-wait somewhere else, and so all of this matters not, except for the sake of readability and simplicity ?

    Thursday, March 9, 2017 9:44 AM

Answers

  • Hi Shrulik,

    The file notification endpoint (messages/servicebound/fileNotifications) currently only exposed using the AMQP protocol, so it will be necessary to create a worker role for receiving a notification message.

    However, when we want to use an Azure Function for notification file upload post-process in the cloud backend and using a file upload capability (AMQP State Machine) built in the Azure IoT Hub, the following "hybrid" scenario can handled it.

    Step 1. Device: Send a POST request to the IoT Hub for file upload reference.

    As you can see, the above response payload has a correlationId for AMQP State Machine and information for creating a storage url address with a temporary SaSToken. Note, that each device can have opened maximum 10 AMQP State Machines at the same time.

    Step 2. Device: Uploading Blob

    the following screen snippet shows an example of the uploading blob file:

    Step 3. Device: Send non-telemetry message (notification) to the Azure IoT Hub Custom Endpoint, for instance Service Bus Queue. The message payload should be the same as received in the step 1. The message is sent using a device protocol, for instance, the MQTT protocol can have a property bag for routing like this example:

      /messages/events/messageType=UploadNotify

    where the messageType=UploadNotify is a routing query string for forwarding the message to the Service Bus Queue.

    Step 4. Azure Function:  The AFN received a D2C message (notification), checking a uploaded blob, handling a necessary business logic and the end it will send a completion request to the IoT Hub (AMQP State Machine) to close the file uploading process. The following screen snippet shows this example:

    Note, that the Azure IoT Hub - Upload File notifications can be turn off, because the AFN already handled this post process.

      

    Thanks

    Roman





    • Edited by Roman Kiss Friday, March 10, 2017 12:05 AM
    • Marked as answer by Shrulik Saturday, March 18, 2017 8:18 PM
    Thursday, March 9, 2017 11:19 PM

All replies

  • Hi Shrulik,

    The file notification endpoint (messages/servicebound/fileNotifications) currently only exposed using the AMQP protocol, so it will be necessary to create a worker role for receiving a notification message.

    However, when we want to use an Azure Function for notification file upload post-process in the cloud backend and using a file upload capability (AMQP State Machine) built in the Azure IoT Hub, the following "hybrid" scenario can handled it.

    Step 1. Device: Send a POST request to the IoT Hub for file upload reference.

    As you can see, the above response payload has a correlationId for AMQP State Machine and information for creating a storage url address with a temporary SaSToken. Note, that each device can have opened maximum 10 AMQP State Machines at the same time.

    Step 2. Device: Uploading Blob

    the following screen snippet shows an example of the uploading blob file:

    Step 3. Device: Send non-telemetry message (notification) to the Azure IoT Hub Custom Endpoint, for instance Service Bus Queue. The message payload should be the same as received in the step 1. The message is sent using a device protocol, for instance, the MQTT protocol can have a property bag for routing like this example:

      /messages/events/messageType=UploadNotify

    where the messageType=UploadNotify is a routing query string for forwarding the message to the Service Bus Queue.

    Step 4. Azure Function:  The AFN received a D2C message (notification), checking a uploaded blob, handling a necessary business logic and the end it will send a completion request to the IoT Hub (AMQP State Machine) to close the file uploading process. The following screen snippet shows this example:

    Note, that the Azure IoT Hub - Upload File notifications can be turn off, because the AFN already handled this post process.

      

    Thanks

    Roman





    • Edited by Roman Kiss Friday, March 10, 2017 12:05 AM
    • Marked as answer by Shrulik Saturday, March 18, 2017 8:18 PM
    Thursday, March 9, 2017 11:19 PM
  • Hey Roman, thanks for the quick reply. 

    So if I understand correctly, you agree I shouldn't use the built in file notification mechanism. A bit of a shame the built in service is lacking. 

    You mentioned there is a limitation of 10 AMQP state machines ( That each represnt a file upload, I presume ) at the same time. If I don't send a notification of a complete file upload to the Iob Hub from the AFN, or just turn off file notifications completely, does it mean there is no way for me to tell the IoT Hub when the file upload is complete, and I will have to wait for the 1 hour expirey time until I can go over the 10 file limit ?  

    You mentioned message routing with the MQTT protocl. I have two questions regarding that:

    1. Can I add a message routing header via the REST api ? I couldn't find how to.

    2. What are the pros/cons of using message routing with custom endpoints, compared to sending all messages to a single endpoint with a differntiating field, and have a StreamAnalytics query split the messages to different places. 

    Sunday, March 12, 2017 7:29 PM
  • Hi Shrulik,

    The File Upload built in the Azure IoT Hub is a great feature providing a reliable queued notification (same like the C2D notifications). You can use two instances of the small (or extra small) worker role instances for listening, receiving and forwarding (such as "a generic pusher") notifications to the Azure Function for handling a specific business logic. This is an another scenario in opposite to my previously thread.

    - Yes, when the device will send 10 requests for file uploading, the next one will response an error until one of then will send a complete request with a valid correlationId or the Azure IoT Hub terminated. The correlationId is valid based on the TTL time (1 - 48 hrs).

    For instance, if the device file uploading is "dead" (glitches, shutdown, unconnected, reset, etc.), the new request for file uploading will be necessary to send to the Azure IoT Hub and the previously one will be closed by Azure IoT Hub after expiring its TTL time. Another workaround scenario can be used, when the device will persisted this correlationId into the device twin.

    - the following screen snippet shows an example of the send D2C message using a REST API:

    as you can see the highlighted headers shows the sys and app headers when the REST API is used for device-facing endpoint. For non-telemetry (D2C) messages can be used an application headers (such as the message properties), so there are need to be prefixed by iothub-app-xxxxx, where the xxxxx is the name of the application property for routing query string.

    - splitting the telemetry and non-telemetry messages within the Azure IoT Hub using a built-in Routes is giving to you better managing, less cost, less dependences, simple testing, etc. Basically, if this decision is not based on the data stream analytics, in other words, if the device decided that this event is a non-telemetry, it should be routed this kind of message into the custom endpoint. Beside that, still you have a capability to ingest those messages to the another Event Hub and forwarded them into the ASA job.

    Thanks

     Roman



    • Edited by Roman Kiss Monday, March 13, 2017 7:05 PM
    Monday, March 13, 2017 6:54 PM
  • Thank you, I apologize for the late reply, I really appreciate your help.

    I have confirmed the routing solution, and it looks pretty good. Before you answered I had another idea. What do you think about having a time triggered Azure Function that tries to pull file notifications every second ? How do you think it compares with regards to cost/performance ? 

    Another question, if I already have you here. This article : https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-device-twins Says that the device twin API is MQTT only for now. But there is a REST api : https://docs.microsoft.com/en-us/rest/api/iothub/devicetwinapi#DeviceTwinApi_UpdateDeviceTwin

    So what does this mean ? 

    Saturday, March 18, 2017 8:36 PM
  • - Having a one second time triggered AFN for polling a service-facing file notification AMQP endpoint (notificationReceiver.ReceiveAsync(TimeSpan.FromSeconds(1)) is not a good idea for production environment, where can be massive file uploads. I do recommend to use a multiple instances (extra small or small) of the worker roles for forwarding a file notification messages to the business logic (for instance to the AFN) and scaling them based on the needs. 

    - the Azure IoT Hub has built-in several endpoints divided basically into two groups such as device-facing endpoints and service-facing endpoints, see more details in the documents Reference - IoT Hub Endpoints.

    The device twins and methods from the device-facing endpoints are available only using MQTT v3.1.1 protocol. On the other side, such as service-facing endpoints we can use the REST API for device twins, direct methods and jobs, see the document IoT Hub REST.

    Thanks

    Roman

      

    • Edited by Roman Kiss Saturday, March 18, 2017 11:45 PM
    Saturday, March 18, 2017 11:43 PM
  • - I'm sorry, but I don't understand, what is the difference between an extra small worker role and an AFN ? Wouldn't the worker also do the exact same thing ? Is it the fact an AFN will go down and up and the workers won't ? Also, isn't your routing idea better anyway ? Or an always up "worker role" will offer better perfomance than an EventHub/Service Bus Queue ? 

    - Yes, of course, silly me. 

    Sunday, March 19, 2017 10:23 AM
  • - The worker role is an instance of the Azure Cloud Service hosted on VM in the PaaS infrastructure for performing a background task, see here. The azure worker role is "similarly" to the Windows NT Service on the OS, it is a start up service - task for endless loop.

    Using a worker role for receiving a file notification message is giving you more robustness solution, where the Azure IoT Hub will deliver this file notification message with a retry mechanism and ACK by your receiver. 

    Thanks

    Roman



    • Edited by Roman Kiss Sunday, March 19, 2017 3:20 PM
    Sunday, March 19, 2017 3:19 PM