locked
How can we load files from IoTHub to storage account (ADLS) by hour.? RRS feed

  • Question

  • Hi,

    We are planning to use IoTHub to load sensor data from our plant devices, but we like to keep a copy of these data in storage account for long term storage for future use cases. whats is the best way to do it.?

    i saw that we can setup routes in IoTHub to storage account, but it seems it batches by minute.

    But if we load sensor data from all our plant devices by minute in to storage account we will end up having millions of smaller files in few days\weeks. is there way we can either batch by Hour or size (ex : wait until all smaller files becomes 128 MB and load in at one file in to storage account). Can we do that.??

    if not is there any other option , like Azure streaming analytics? just want to get the opinion from experts.

    Regards,

    Sai


    Saturday, April 11, 2020 8:08 PM

All replies

  • Hi,

    - The Azure IOT Hub enables to configure a batch frequency between the 60 - 720 seconds for capturing events for blob storage, so the maximum blobs per day and partition is 120, when the batch frequency = 720 seconds. If your IoT Hub has 10 partitions you will have 1200 blobs per day (no millions).

    Another approach is using a custom schedule job (it's required coding and additional azure resources) for storing all IoT Hub events each day based on the needs. Note, that the events retained in the IoT Hub for max 7 days (the value is configurable).

    Thanks

    Roman 





    Sunday, April 12, 2020 1:42 PM
  • OK , thank you . i was thinking it will copy file to file instead of merging all the file at batch run.

    so when it executes at 12 min intervals,will it produce 1 blob file with all the small files combined.

    one other question , after the execution (moving in ADLS) will it remove the messages in IoTHub or will they still stay for 7 days.

    Regards,Sai

    Sunday, April 12, 2020 9:50 PM
  • Hello Sai,

    IoT Hub will write messages to a blob based on the batch frequency and block size specified. After either the batch size or the batch frequency is hit, whichever happens first, IoT Hub will then write the enqueued messages to the storage container as a blob. IoT Hub defaults to the following file naming convention:

    {iothub}/{partition}/{YYYY}/{MM}/{DD}/{HH}/{mm}

    I would suggest you, go through the documentation for more details to understand Upload files with IoT Hub.

    Regarding data retention:

    IoT Hub allows data retention in the built-in Event Hubs for a maximum of 7 days. You can set the retention time during creation of your IoT Hub. Data retention time in IoT Hub depends on your IoT hub tier and unit type. In terms of size, the built-in Event Hubs can retain messages of the maximum message size up to at least 24 hours of quota. For example, for 1 S1 unit IoT Hub provides enough storage to retain at least 400K messages of 4k size each. If your devices are sending smaller messages, they may be retained for longer (up to 7 days) depending on how much storage is consumed. We guarantee retaining the data for the specified retention time as a minimum. Messages will expire and will not be accessible after the retention time has passed.

    Hope this helps. Do let us know if you have further queries.

    Thursday, April 16, 2020 4:58 PM
  • Hello Sai,

    Just checking in to see if the above answer helped. If this answers your query, do click “Mark as Answer” and Up-Vote for the same, which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

    Monday, April 20, 2020 4:38 AM