none
Need to stream JSON data in Data Lake Store to Event Hubs

    Question

  • Is there any way I can stream data directly from Data Lake Store instance as my primary Data Source to Event Hubs?

    My data source has JSON documents, and will keep coming in as batches or files from a different source and I want to trigger events by reading into each document, picking the first line and placing it into event hubs as a single event; every time a new document comes.

    I know this is not the architecture in mind behind using these services, but I've seen .Net implementation of data streaming from an Azure SQL instance to Event Hubs if I'm not wrong. That being said, even though it's an unusual use case, the requirements I have are as such.

    I would like to know if there are more ways that using .Net framework to implement this scenario using other services that Azure provides. Maybe if there is scope to automate this whole process using resources like Azure Data Factory or using Service Bus as a middleware. 

    Friday, March 9, 2018 12:17 AM

All replies

  • Hi,

    Thanks for your interest in Azure Data Lake Store (ADLS). 

    For your scenario you could implement solutions which could poll the ADLS account, check for new file arrivals and create the Event Hubs events. However, these solutions will have limitations such as latency is detection of file arrival, having to track the files that are new, scalability issues when there are lots of files at rest and coming in. We don't have templates for doing these, but you could do them using Azure Functions, Logic Apps and such. The correct solution would be to have events being pushed from ADLS itself . We have been working on providing an implementation based on Event Grid to provide ADLS events. It will be similar to what Blob storage has in preview (https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-event-overview) and should preview in a couple of months. If you would like to participate in the preview please do let us know.

    Thanks,
    Sachin Sheth
    Program Manager, Azure Data Lake.

    Friday, March 9, 2018 6:55 PM
  • Thanks for that info and the offer. I would want to participate in a preview, sure. :)
    Saturday, March 10, 2018 8:48 AM