locked
Azure ADF pipeline workaround needed ! RRS feed

  • Question

  • Hello all,

    Currently we have a table with more than 2 lakh records so when we move the data from source azure sql database to another sql database it takes a lot of time with more than 3 hours resulting in timeout error, initially we set timeout as 1 hour however because of timeout error we have to increase the timeout interval to 3 hours but still its not working.

    This is how we have defined the process.

    1. Two datasets -> input and output
    2. One pipeline
    Inside the pipeline we have a query like select * from table;
    and we have stored procedure and its script is like
    1. Delete from table all records.
    2. Insert statement to insert all records.

    This is time consuming so we have decided to do update and insert whatever data is modified or inserted based on date column in last 24 hours.

    So is there any functionality in azure pipeline which checks the records which are inserted or updated in source azure sql db in last 24 hours or do we need to do in destination sql stored procedure.

    If its azure pipeline or stored procedure please provide some code for clear understanding, thank you.

    Wednesday, September 21, 2016 9:54 AM

All replies

  • This forum is for questions about Azure Search. I'll move this thread to the ADF forum.
    Wednesday, September 21, 2016 3:12 PM