locked
Incremental Data Load in Azure Data Factory RRS feed

  • Question

  • I am loading data from tab formatted text files to azure SQL server using Data Factory. currently i am dumping all the data into SQL. I am looking for incremental data load by comparing Last-updated column in table and Last-updated column in text file. 


    Harpal Singh


    • Edited by Harpalsgh Thursday, August 31, 2017 6:38 PM
    Thursday, August 31, 2017 6:25 PM

All replies

  • In order to get the Last-updated column from the text file you need to process it at some point. This can be costly if you want to do the processing and the comparison in one step.

    I would load the full text file into a separate SQL table and then use MERGE to update the records in your real SQL table based on the new data from the text-file and the Last-updated column

    would this work for you?

    -gerhard


    Gerhard Brueckl
    blogging @ http://blog.gbrueckl.at
    working @ http://www.pmOne.com

    Friday, September 1, 2017 7:31 AM