How to further process and transform data in Azure Data Lake Store? RRS feed

  • Question

  • Hi all,

    I'm new to Azure. I realize there's this thing called Azure Data Lake store, I would like to use it to store my data and further process it.

    Here is my use case:

    I have some sales information in dynamics CRM, which needs a monthly snapshot, and I need to further analyze the snapshot data, then finally present the result in POWER BI for visualization.

    Now using data factory I'm able to connect to dynamics CRM and copy data to Azure Data Lake. However the data is in raw format, I need to process, transform the data, how do I do it with data lake store?

    Though power BI could do data transformation, but I would like to clean the data before I load it into power BI. e.g. take out blank columns, merge snapshot tables etc. How can I achieve this?

    Thank you.

    Thursday, June 21, 2018 5:40 AM

All replies

  • Hi Wendy,

    Which kind of transformation do you need make on ADLS? ADFv2 provides multiple transformation activity and control flow activity for you to use. Please reference https://docs.microsoft.com/en-us/azure/data-factory/control-flow-filter-activity to see if there is one meeting your requirement.

    Also, if you can provide a sample of the raw data in ADLS and specify your requirement, we'll get a chance to see how to further help you.

    Thursday, June 21, 2018 9:06 AM
  • Please take a look of Data Lake U SQL activity. https://docs.microsoft.com/en-us/azure/data-factory/transform-data-using-data-lake-analytics
    You could write your transform logic with Data Lake Analytics U-SQL script, and U SQL activity will help you execute that script.
    Thursday, June 21, 2018 3:36 PM