none
Copy multiple files with different schema from Data Lake into Azure SQL Database - Rerun the failed entity not the entire CopyPipeline

    Question

  • Hi Guys,

    I have a copyPipeline that copies multiple files with different schema from Data Lake into Azure SQL Database and it runs perfectly as i wanted but the main issue is that when one of the file copy to Azure SQL Database fails i dont have an option to rerun it, i'm forced to rerun the entire copyPipeline after resolving the problem for the failed file. Please note that the copyPipeline does an  incremental copy daily  and re-running copyPipeline will result in duplicates into Azure SQL Database for successfully files.

    Is there a way i can easily manages the fails better without rerunning the entire copyPipeline?

    Regards,

    Thabo

    Monday, August 27, 2018 8:27 AM

All replies

  • Hi Thabo,

    Running a pipeline from a failure point is in the plan, but I don't have an ETA here.

    One workaround I can come up is:

    each activity has a "retry" configuration at policy field. You could find it at general tab of the copy activity. Increase it could avoid some transient system or network issue.

    Otherwise, you may need create another pipeline to just include the failed files.

    Monday, August 27, 2018 11:30 AM