none
Migrate azure SQL db to azure data lake

    Question

  • Hi, 

    I want to completely migrate my data from azure SQL database to azure data lake. How can I do that.

    Please give some scenario so that I can move completely to azure data lake.

    Thanks in advance.

    Friday, October 21, 2016 7:25 AM

Answers

  • You should be able to do it using one of the following approaches (assuming you can only get the delta from your SQL Server instance):

    1. if you just want the data in files in ADLS: Organize the files to identify the increments either through file or path names (e.g., /data/2016/10/{day}/data.csv) and then use U-SQL file sets to query the data.
    2. if you want the data in a non-partitioned U-SQL table: Upload the delta into a file in ADLS and then use an INSERT INTO Table EXTRACT ... FROM "/data.csv" ... ; Note that after a while you probably want to do an ALTER TABLE Table REBUILD to regain performance.
    3. if you want the data in a partitioned U-SQL table: Upload the delta into a file in ADLS and then submit a parameterized script that adds the partition (the partition value should be part of the parameter) and then insert into that partition from the file...

    You should be able to orchestrate any of these approaches with ADF.


    Michael Rys

    • Marked as answer by Syed_Aqib Wednesday, October 26, 2016 4:18 AM
    Tuesday, October 25, 2016 12:16 AM
    Moderator
  • Hi Syed,

    A good way to do that is using Azure Data Factory, take a look at the doc here


    Jason Chen, Windows Azure PM

    • Marked as answer by Syed_Aqib Monday, October 24, 2016 4:21 AM
    Friday, October 21, 2016 4:08 PM
  • Also another good place to start is here: https://azure.microsoft.com/en-us/documentation/articles/data-lake-store-data-scenarios/

    • Marked as answer by Syed_Aqib Monday, October 24, 2016 4:21 AM
    Friday, October 21, 2016 4:53 PM

All replies

  • Hi Syed,

    A good way to do that is using Azure Data Factory, take a look at the doc here


    Jason Chen, Windows Azure PM

    • Marked as answer by Syed_Aqib Monday, October 24, 2016 4:21 AM
    Friday, October 21, 2016 4:08 PM
  • Also another good place to start is here: https://azure.microsoft.com/en-us/documentation/articles/data-lake-store-data-scenarios/

    • Marked as answer by Syed_Aqib Monday, October 24, 2016 4:21 AM
    Friday, October 21, 2016 4:53 PM
  • Can i perform incremental load to azure data lake?
    Monday, October 24, 2016 5:03 AM
  • You should be able to do it using one of the following approaches (assuming you can only get the delta from your SQL Server instance):

    1. if you just want the data in files in ADLS: Organize the files to identify the increments either through file or path names (e.g., /data/2016/10/{day}/data.csv) and then use U-SQL file sets to query the data.
    2. if you want the data in a non-partitioned U-SQL table: Upload the delta into a file in ADLS and then use an INSERT INTO Table EXTRACT ... FROM "/data.csv" ... ; Note that after a while you probably want to do an ALTER TABLE Table REBUILD to regain performance.
    3. if you want the data in a partitioned U-SQL table: Upload the delta into a file in ADLS and then submit a parameterized script that adds the partition (the partition value should be part of the parameter) and then insert into that partition from the file...

    You should be able to orchestrate any of these approaches with ADF.


    Michael Rys

    • Marked as answer by Syed_Aqib Wednesday, October 26, 2016 4:18 AM
    Tuesday, October 25, 2016 12:16 AM
    Moderator