none
How do I Stream Data from SQL Server into Azure Data Warehouse? RRS feed

  • Question

  • The Microsoft process looks like a batch import method of copying data from SQL Server into Azure Data Warehouse. Is there a simpler method, conducting close to real time of streaming data from MS SQL Server directly into Datawarehouse. This seems overly complicated with two excessive ETL steps, (Azure Data Factory, and then Polybase) . Can we simplify and continually stream data from SQL Server into Data Warehouse? (maybe map tables 1 to 1 close as possible, we do Not plan on creating Dim or facts)

    We know AWS allows streaming of data from SQL server into Redshift DW

    https://aws.amazon.com/about-aws/whats-new/2017/05/aws-schema-conversion-tool-exports-from-sql-server-to-amazon-redshift/




    • Edited by BlueCar5 Monday, September 10, 2018 1:12 AM few more words
    Sunday, September 9, 2018 5:58 PM

All replies

  • Hi there,

    The graph above is high level and does not convey some of the ingestion details.  When using ADF to load data into Azure SQL Data Warehouse, it will use Polybase.  Please refer to this article on the two use cases (direct copy using Polybase and staged copy using Polybase): https://docs.microsoft.com/en-us/azure/data-factory/connector-azure-sql-data-warehouse#use-polybase-to-load-data-into-azure-sql-data-warehouse

    You also have a second requirement which is continuously streaming - can you please be more specific what you mean by that?  Are you looking for low schedule interval (5min?  1min?  Even lower?) or a push mode instead of scheduled pull mode?

    Monday, September 10, 2018 2:53 PM