none
Recommended Azure Data Lake Analytics Solution/Project Directory Structure

    Question

  • Do we have any recommended solution/project directory structure?

    For Example, 

    1.src

    2.config

    3.usql

    4.adf

    5. powerbi

    something like this

    Friday, February 2, 2018 5:52 PM

All replies

  • Hi Sathiq,

    Are you talking about a recommended data pipeline structure using Azure Data Lake? Or a directory structure for storing your data in Azure Data Lake Store? 

    Wednesday, February 7, 2018 6:10 PM
  • That heavily depends on the project and some personal preferences.

    I personally follow the following structures inside an ADL solution:

    1. Have one solution for shareable code that you register as U-SQL Assembly. That should contain the U-SQL for C# (or other language) project, a project containing some sample and test U-SQL scripts and some unit tests for the language code as well as installation instruction.
    2. Have a solution for your database objects (TVF definitions, table definitions etc). Since you will be installing them once for sharing and then rebuilding them infrequently, keeping them separate makes sense. Again, this should contain a U-SQL project for the object creation, maybe one for re deployment and one for tests and samples. (please note that we are looking into adding a special database project type)
    3. Have a solution for your project specific scripts.

    You can combine some of them depending on the size and specificity (e.g., if your user-code and database objects are only making sense together, combine them in the same solution in different projects).

    If you have ADF, PowerBI etc aspects, you can either move them into the same solution or keep them in a related solution, depending on how tightly coupled they are, depending on whether you have the same programmers/teams working on them or not.

    In the end, a lot of this structure needs to make sense to you and your team.


    Michael Rys

    Wednesday, February 7, 2018 7:20 PM
    Moderator