locked
ADF deployment by pipe line - methods to migrate code across data factories RRS feed

  • Question

  • This is regarding the deployment of ADF components - pipelines and any associated components. Trying to understand the methods to migrate the code across data factories and environments.
    
    (We are not yet using Azure DevOps and hence the question is not in that context.)
    
    One way we are trying to do is via an ARM template deployment. From what we understand, ARM template downloads the whole codebase, which we dont want to. We have a development environment that has a lot more pipelines that what would actually get migrated to production and we want only to do a selective deployment of selected components.
    
    Could you please suggest the most optimal and efficient way to do this ? Your suggestions are appreciated.
    
    Tentative thought process : ARM template -- GitHub Enterprise -- Jenkins -- ADF
    
    However, we are also open to command line based deployment ex.using AZCLI which can be used in conjunction with terraform as a local exec.
    Monday, May 11, 2020 6:47 PM

All replies

  • Hello azdevad1 and thank you for your question.

    There are ways other than ARM templates to deploy Factories or their assets.  Please be aware the options I provide below make changes to the Factory directly, and do not include updates to Git repositories.

    • Powershell Az module
    • REST API For Data Factory v2
    • AZCLI does not have a Data Factory extension, but it may be possible to deploy using the generic resource manager.  You could also use az_rest to call the aforementioned REST API for Data Factory.
    • For your situation, the .NET SDK and Python SDK are not necessary.  This is because you already have your definitions.  You just want to deploy them.

    Both Powershell and REST API can both read and write resources.  You can fetch the resource definition (JSON) by name from the development factory.  Once you have the definition local, you can edit if needed, then push it to the next factory.

    Alternatively, you can use Git to pull the definitions (JSON) from your repository, and then push to a factory.  Fetching from Git is done by a separate git command, not by Powershell.

    To help avoid issues, remember to deploy resources in the following order:

    1. Factory (if not exist)
    2. Linked Services
    3. Datasets
    4. Data Flows
    5. Pipelines
    6. Triggers
    Tuesday, May 12, 2020 8:36 PM
  • Hi, I'm just following up to see if this resolved your issue.  If it did, please mark as answered.
    Thursday, May 28, 2020 10:20 PM
  • Since we have still not heard back from you, we will assume you found your own resolution.  If you found a solution, would you please share it here with the community?
    Tuesday, June 2, 2020 1:43 AM