locked
CSVs from third party app in the blob storage RRS feed

  • Question

  • Hi there!

    I have data in my Azure Data Lake that I ingest every hour from a third party app (this third party app makes the data available for me in batches of ~1min). Therefore, there is no stream of data, I get the data as CSV or JSON, and then I dump it to Azure Data Lake (I can also dump it to a Blob storage). Now, my question is, how can  I create  a stream of data with Azure Stream Analytics (or the best way) ? My ultimate plan is to user Power BI for reports. What would the format of the files need to be (I can change the names of the files). Summary:

    1.- Get data from the third party vendor

    2.- Dump it in to Blob storage in certain format (CSV or JSON with certain names)

    3.- Read this data using  Azure Stream Analytics and update every minute or so (not that I will have lots of small files, but I plan to purge the old ones)

    4.- Connect Azure Stream Analytics with Power BI (this option is still not available)

    Please, let me know if you guys have suggestion about the name of the files and their format, or the best way to do this.

    Thanks,

    Jose

    Tuesday, April 10, 2018 2:11 PM

All replies

  • Hi, this scenario can be supported easily with Stream Analytics.

    For this, you just need to use your blob storage as input as described here, CSV or JSON can be used.

    Then you just need to define a query, and set PowerBI as output.

    Let me know if it works for you.

    Thanks,
    JS

    Tuesday, April 10, 2018 6:31 PM
  • Hi!

    Thanks for your answer. This is very useful, do you know if I can start and stop the Stream Analytics to only ingest data for certain period of time? Let's say from 5pm to 9pm. I know this looks weird, but it would be helpful.

    Thanks,

    Jose

    Wednesday, April 11, 2018 8:11 PM