Data Lake to Azure Data Ware House RRS feed

  • Question

  • I've been looking for best practices to load incremental data from the Data Lake into the Azure Data warehouse. So far I've tried external tables which lets me query for only deltas. Unfortunately I also the the file header also, and am missing a lot of the features that normal sql has to offer. This is probably the route I'll end up going but it seems clunky.

    I've also tried using the datafactory to load the DW, unfortunately it appears that I can't load just deltas from a file. Pretty much truncate the table and reload all of it.

    Also I've used a SQL on an Azure VM, but it appears that isn't able to use external tables to connect to the data lake.

    Does anyone have suggestions for best practices? The data lake is a requirement, but the azure data warehouse is not - it could be something other than an azure data warehouse.

    Thank you.

    Monday, June 19, 2017 9:34 PM


All replies