Parallel instances of data factory pipeline RRS feed

  • Question

  • We are planning to run multiple instances of Azure data factory pipeline from azure functions which are triggered by blob creation event on a set of folders (1 set of folder per customer). The idea is to run the same pipeline parallely for different customers at the same time. Is it  allowed to run the same pipeline parallely at  the same time if called from azure functions?

    Manish Pradhan

    Sunday, March 29, 2020 3:00 PM


  • Hi Manish,

    There's no harm in running multiple instances of a pipeline concurrently. Based on this azure-data-factory-limits, azure data factory V2 has certain limitations.

    Such as Concurrent pipeline runs per pipeline is 100 and Write API calls is 2500/hr. You need to optimize your behaviors against the limitations.In addition, you could contact support about your custom requirements.

    Monday, March 30, 2020 4:41 AM