locked
Stream Data from SQL Server into Azure Data Warehouse RRS feed

  • Question

  • I want to copy our data from Azure SQL (OLTP) into Azure Datawarehouse. I want the path to be close to real time.

    The Microsoft process below looks it takes two steps copying data from SQL Server into Azure Data Warehouse.

    - First into Blob Storage with Azure Data Factory

    - and then with Polybase into Azure Datawarehouse.

    It looks like a two step ETL and not close to real time.

    (1) Does Microsoft have any simpler way of copying data from Azure SQL into DW for analytics??

    (2) Can someone give me a link to Microsoft case study to do conduct this? I see so many resources out there,

    enter image description here




    • Edited by CarApple52 Thursday, October 4, 2018 2:57 AM
    Thursday, October 4, 2018 2:51 AM

Answers

  • Hi CarApple52,

    There is no 2-step process to migrate data from Azure SQL Database to Azure SQL Data Warehouse. The Migrate Your Data documentation offers multiple paths for a 3-step process, and the simplest is the AZCopy Utility. Choose this method if you data does not need any scrubbing.

    There is the Azure Data Factory (ADF) copy option, which will allow you to automate the process, down to 1 minute intervals.

    There are additional options outlined but, there is no direct migration available. It is near real-time at best. 

    Please let us know if you have additional questions.

    Regards,

    Mike

     

    • Marked as answer by CarApple52 Friday, October 5, 2018 9:19 PM
    Thursday, October 4, 2018 9:13 PM

All replies

  • Hi CarApple52,

    There is no 2-step process to migrate data from Azure SQL Database to Azure SQL Data Warehouse. The Migrate Your Data documentation offers multiple paths for a 3-step process, and the simplest is the AZCopy Utility. Choose this method if you data does not need any scrubbing.

    There is the Azure Data Factory (ADF) copy option, which will allow you to automate the process, down to 1 minute intervals.

    There are additional options outlined but, there is no direct migration available. It is near real-time at best. 

    Please let us know if you have additional questions.

    Regards,

    Mike

     

    • Marked as answer by CarApple52 Friday, October 5, 2018 9:19 PM
    Thursday, October 4, 2018 9:13 PM
  • cool thank you, there are so many options available, ADF Copy, SSIS, Bcp, maybe Microsoft can have a white paper to discuss each option, very hard for new users, with all the new technology, not a lot of blobs or forum posts about this yet,

    Friday, October 5, 2018 2:23 AM
  • just fyi, seems like AzCopy will not copy from and to relational database, see the chart here,

    Thanks

    https://docs.microsoft.com/en-us/azure/architecture/data-guide/scenarios/data-transfer

    Friday, October 5, 2018 8:50 PM
  • AzCopy will copy data to/from an Azure Storage Account. You can use bcp Utility to load the data. 
    Tuesday, October 9, 2018 4:29 PM
  • What excatly are you aiming for? Is the product of based on your data stream an aggreate result? How big is the dataset?
    Tuesday, October 16, 2018 5:23 AM