locked
Designing Azure Data Factory Pipelines RRS feed

  • Question

  • Hi All, 

    While building pipeline in Azure Data factory, I have 2 options 

    1. Use ADF activity (For-each, Lookup, If etc.) to have my data loading part done.

    2. Create a custom batch activity to do looping and if/else logic and call it from ADF. 

    Only advantage of option #2 is schema can be generated at run time, which in some case will give no visibility if something changes. 

    Please suggest which one is the most elegant way to do this?

    Thanks, Amit 

    Wednesday, March 6, 2019 8:54 AM

Answers

  • Hi Amit,

    When you use a custom activity, you can use Azure SDKs to get almost everything done from operations perspective. If you are comfortable with custom activities, you can do what you want very easily within one single activity. However if you use the other activities, it makes it more modular and easy to debug.

    Hence both of them have their pros and cons and it completely depends on your use case and your comfort with .NET (programming).

    Hope this helps.


    MSDN

    Tuesday, March 26, 2019 9:50 AM