Does Incremental Sync with Azure Data Factory V2 support only Sql Server to Azure SQL RRS feed

  • Question

  • Hi,

    I was trying incremental sync to  design incremental sync data between MySQL and Azure Sql referring article and while designing pipeline for new watermark  found that lookup component only support SQL SERVER.


    Is there a way to sync incrementally on cloud from hosted Mysql to Azure SQl  using ADF v1/v2?

    What other component we can use to select data from Mysql i tried Store Procedure but it support Sql server only.


    Tuesday, March 6, 2018 2:18 PM

All replies

  • I had the same dilemma so I'll give you my two cents.  You can use a tumbling window trigger to create time slices (which is what ADFv1 uses) and within your source queries use windowStart and windowEnd to incrementally pull data instead of using a watermark.  

    I am still storing a low and high watermark value in a table for auditing purposes, but the watermark represents the window of time the tumbling window ran instead of the MAX modified date of current and previous load.  I think you could do the same behavior with a regular trigger than the tumbling window but that decision is up to you.  Unfortunately, because of the lookup activity's limited functionality and lack of support for on-prem I haven't found a different solution.

    Tuesday, March 6, 2018 2:29 PM
  • Thank you for your interest in trying out ADF.  You are correct - today the lookup activity only supports a subset of the data stores and cannot be used to look up watermark values in on-prem MySQL.  We are enhancing the lookup activity to support broader set of data stores over the next months and will post an announcement when ready.  Please stay tuned!

    Shirley Wang

    Wednesday, March 7, 2018 1:09 AM