locked
DevOps pipelines and ARM templates RRS feed

  • Question

  • I am building a backend cloud using ADLS Gen2 Storage, Data Factory, and Azure Functions. I'd like to deploy the infrastructure via an Azure DevOps Pipeline reading from a Git repo.

    I have working ARM templates for the various components, but I am having trouble stitching them together. Can anyone point to a working example I can copy from?

    In a perfect world, I'd like to use some kind of script to generate my templates, perhaps something that prompts for the storage, function and data factory options. But anything I can use as a starting point will help a lot.

    Tuesday, September 17, 2019 2:58 PM

Answers

  • steps:
    - task: AzureResourceGroupDeployment@2
      name: storageSetup
      inputs:
        azureSubscription: $(azureSubscription)
        action: 'Create Or Update Resource Group'
        resourceGroupName: $(resourceGroupName)
        location: 'Central US'
        templateLocation: 'Linked artifact'
        csmFile: 'templates/adlsg2_storage_account.json'
        overrideParameters: -storageAccount_name "$(storageAccountName)"
        deploymentOutputs: 'storageOut'
        deploymentMode: 'Incremental'
    
    - pwsh: |
        $outputs = ConvertFrom-Json '$(storageOut)'
        foreach ($output in $outputs.PSObject.Properties) {
          Write-Host "##vso[task.setvariable variable=SOUT_$($output.Name)]$($output.Value.value)"
        }
    
    - bash: |
        echo dfsEndpoint $(SOUT_dfsEndpoint)
        echo accountKey $(SOUT_accountKey)
    
    - task: AzureResourceGroupDeployment@2
      inputs:
        azureSubscription: $(azureSubscription)
        action: 'Create Or Update Resource Group'
        resourceGroupName: $(resourceGroupName)
        location: 'Central US'
        templateLocation: 'Linked artifact'
        csmFile: 'templates/data_factory.json'
        deploymentOutputs: 'dfOut'
        overrideParameters: -dataFactoryName "$(dataFactoryName)"
        deploymentMode: 'Incremental'
    
    
    - task: AzureResourceGroupDeployment@2
      inputs:
        azureSubscription: $(azureSubscription)
        action: 'Create Or Update Resource Group'
        resourceGroupName: $(resourceGroupName)
        location: 'Central US'
        templateLocation: 'Linked artifact'
        csmFile: 'templates/data_factory_internals.json'
        overrideParameters: -factoryName "$(DFOUT_dataFactoryName)" -adlsg2_dfs_url "$(SOUT_dfsEndpoint)" -adlsg2_accountKey "$(SOUT_accountKey)"
        deploymentMode: 'Incremental'
    
    This snippet might be useful for someone coming along later. It demonstrates how to thread output from one ARM template to parameters on the next. You need to set the outputs map on your ARM templates.
    Thursday, September 19, 2019 7:48 PM

All replies

  • It looks like the sort of example I'm looking for is a pipeline.yml file that uses deploymentOutputs and overrideParameters to send data from one step to the next.
    Tuesday, September 17, 2019 8:47 PM
  • Hi Chris,

    Have you gone through the CI/CD documentation for Azure Data Factory ? It has step by step explanations as well as some commonly used examples.

    Let us know if it helps. Else, we can gladly continue to dive deeper.

    Wednesday, September 18, 2019 10:18 AM
  • steps:
    - task: AzureResourceGroupDeployment@2
      name: storageSetup
      inputs:
        azureSubscription: $(azureSubscription)
        action: 'Create Or Update Resource Group'
        resourceGroupName: $(resourceGroupName)
        location: 'Central US'
        templateLocation: 'Linked artifact'
        csmFile: 'templates/adlsg2_storage_account.json'
        overrideParameters: -storageAccount_name "$(storageAccountName)"
        deploymentOutputs: 'storageOut'
        deploymentMode: 'Incremental'
    
    - pwsh: |
        $outputs = ConvertFrom-Json '$(storageOut)'
        foreach ($output in $outputs.PSObject.Properties) {
          Write-Host "##vso[task.setvariable variable=SOUT_$($output.Name)]$($output.Value.value)"
        }
    
    - bash: |
        echo dfsEndpoint $(SOUT_dfsEndpoint)
        echo accountKey $(SOUT_accountKey)
    
    - task: AzureResourceGroupDeployment@2
      inputs:
        azureSubscription: $(azureSubscription)
        action: 'Create Or Update Resource Group'
        resourceGroupName: $(resourceGroupName)
        location: 'Central US'
        templateLocation: 'Linked artifact'
        csmFile: 'templates/data_factory.json'
        deploymentOutputs: 'dfOut'
        overrideParameters: -dataFactoryName "$(dataFactoryName)"
        deploymentMode: 'Incremental'
    
    
    - task: AzureResourceGroupDeployment@2
      inputs:
        azureSubscription: $(azureSubscription)
        action: 'Create Or Update Resource Group'
        resourceGroupName: $(resourceGroupName)
        location: 'Central US'
        templateLocation: 'Linked artifact'
        csmFile: 'templates/data_factory_internals.json'
        overrideParameters: -factoryName "$(DFOUT_dataFactoryName)" -adlsg2_dfs_url "$(SOUT_dfsEndpoint)" -adlsg2_accountKey "$(SOUT_accountKey)"
        deploymentMode: 'Incremental'
    
    This snippet might be useful for someone coming along later. It demonstrates how to thread output from one ARM template to parameters on the next. You need to set the outputs map on your ARM templates.
    Thursday, September 19, 2019 7:48 PM
  • Thanks for sharing your findings Chris :)

    Friday, September 20, 2019 5:21 AM