What are standard Design patterns while using Azure Data Lake Store, Data Lake Analytics (U-Sql) and Azure Data Factory?


  • I am learning Azure Data Lake Store, Data Lake Analytics (U-Sql) and Azure Data Factory. I want to know weather there are any standard design patterns which we should follow?  specially I am interested in while creating complex data work flow using U-Sql, Data Lake Store and data lake factory.  I want to ingest around 100 TB of data per day into data lake store, then I want to do some  transformation and aggregations on that data and finally I want to show that data into reports using Power BI.  It will be good if you provide some tools available for ingesting data into Data Lake Store and showing reports from data lake store.

    Sunday, July 17, 2016 6:15 AM

All replies

  • Here are some suggestions (note these are my suggestions given the current status when I give the reply)

    1. On the data ingestion side, we recommend to either use ADF or the ADL Store's Powershell upload command.
    2. If you orchestrate it, you should use Service Principals to do the authentication.
    3. You have two ways to show PowerBI reports: You can connect PowerBI to a CSV file in your ADL Store. Or you can use another ADF data movement to take the CSV file, move it into a SQL Server/Azure SQLDB instance and connect PowerBI to it. Note that direct PowerBI integration is planned for later (and needs interactive U-SQL).

    Michael Rys

    • Proposed as answer by Michael Amadi Friday, August 19, 2016 4:23 PM
    Tuesday, July 19, 2016 9:31 PM