locked
Is it possible to automate "Publish"? (for CI/CD) RRS feed

  • Question

  • Hi,

    The current description of CI/CD (https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment) there's still a mandatory manual step "Publish" to the development environment.

    * Is there any way to automate this step (i.e. trigger via API), so that the CI process can be completed when the Pull Request is completed?

    On a related note:

    * Is there a way to automatically generate the ARM templates to file system (e.g. to use as a pipeline artifacts), instead of having "Publish"?

    Thanks,

    Thursday, July 4, 2019 3:15 AM

Answers

  • At this time there does not appear to be any public API call for publishing.
    • Marked as answer by avihaid Thursday, July 11, 2019 3:04 PM
    Thursday, July 11, 2019 12:29 AM

All replies

  • As my understanding, if you are using azure devops, the pipeline can be triggered automatically when the pull request is completed. Or you can also configure schedule triggers. 

    Read azure devops docs here:

    https://docs.microsoft.com/en-us/azure/devops/pipelines/release/define-multistage-release-process?view=azure-devops#configure-the-triggers-in-your-release-pipeline

    https://docs.microsoft.com/en-us/azure/devops/pipelines/release/triggers?view=azure-devops

    Friday, July 5, 2019 2:16 AM
  • Thanks for your reply.

    What I'm trying to do is integrate Data Factory specific flow with Azure Devops pipeline.

    I want an Azure DevOps pipeline trigger to initiate when pull request completes, to generate ARM templates and publish. In the documentation provided in my question, this step is manual, and cannot be linked to a trigger.

    My question is hence -

    1) can Data Factory Publish action (on the Dev instance) be automated, so that it can be linked to a trigger (e.g. via an API call).

    2) if the ARM template generation can be automated and separated to generate a pipeline artifact, rather than publish into a dedicated branch.

    Monday, July 8, 2019 2:58 AM
  • Hello avihaid and thank you for your response.  The act of publishing is the same as the act of creating a new resource.  I am not yet familiar with Azure DevOps pipelines, so please forgive me.

    There are ways to create new resources (including Linked Services, Datasets, and Pipelines) programmatically.  This includes REST and powershell and several SDK's. 

    If you can identify the changed resources, you can then create / update / delete the corresponding resource in whichever Data Factory instance you like using any of the aforementioned mechanisms.

    Tuesday, July 9, 2019 2:12 AM
  • When using git integration (as described in the CI/CD documentation link provided in my original question), "Publish" seem to have at least 2 effects:

    1. Deploy the contents of the collaboration branch.

    2. Create ARM Templates (parameterized, with unified and linked variations), and commit them to the publishing branch. 

    The act of "Publishing" as documented, is a manual intervention required in the middle of an otherwise automated flow.

    I'm trying to understand if there's a way to automate this part as well (e.g. by an API call?)

    Tuesday, July 9, 2019 2:48 PM
  • At this time there does not appear to be any public API call for publishing.
    • Marked as answer by avihaid Thursday, July 11, 2019 3:04 PM
    Thursday, July 11, 2019 12:29 AM