locked
REST API - Use "Azure Key Vault" to populate the header "x-api-Key" field RRS feed

  • Question

  • Hi,

    I created a pipeline to read Application Insights custom events and entered the key manually for the "x-api-Key" header, does anyone know how to pass the key using Key Vault?

    Best regards,
    Cristina

    Wednesday, January 8, 2020 11:56 PM

Answers

  • Hello,

    I found a solution, I don't know if it's the best, but it was the only one at the moment. Below is the walkthrough:

    1) Create "arm-template-parameters-definition.json" file in Data Factory directory:

    In "Microsoft.DataFactory/factories/pipelines" the parameters "x-api-Key" and "url" were added. The "url" parameter has been set to test, but since "x-api-Key" worked it will not be used. The rest of the code is standard and was provided in the Microsoft documentation.

    {
        "Microsoft.DataFactory/factories/pipelines": {
            "properties": {
                "activities": [{
                    "typeProperties": {
                        "source": {
                           "additionalHeaders":{
                              "x-api-Key": "-::string"
                           }
                        },
                        "url": "-::string"
                    }
                }]
            }
        },
        "Microsoft.DataFactory/factories/integrationRuntimes":{
            "properties": {
                "typeProperties": {
                    "ssisProperties": {
                        "catalogInfo": {
                            "catalogServerEndpoint": "=",
                            "catalogAdminUserName": "=",
                            "catalogAdminPassword": {
                                "value": "-::secureString"
                            }
                        },
                        "customSetupScriptProperties": {
                            "sasToken": {
                                "value": "-::secureString"
                            }
                        }
                    },
                    "linkedInfo": {
                        "key": {
                            "value": "-::secureString"
                        },
                        "resourceId": "="
                    }
                }
            }
        },
        "Microsoft.DataFactory/factories/triggers": {
            "properties": {
                "pipelines": [{
                        "parameters": {
                            "*": "="
                        }
                    },  
                    "pipelineReference.referenceName"
                ],
                "pipeline": {
                    "parameters": {
                        "*": "="
                    }
                },
                "typeProperties": {
                    "scope": "="
                }
    
            }
        },
        "Microsoft.DataFactory/factories/linkedServices": {
            "*": {
                "properties": {
                    "typeProperties": {
                        "accountName": "=",
                        "username": "=",
                        "userName": "=",
                        "accessKeyId": "=",
                        "servicePrincipalId": "=",
                        "userId": "=",
                        "clientId": "=",
                        "clusterUserName": "=",
                        "clusterSshUserName": "=",
                        "hostSubscriptionId": "=",
                        "clusterResourceGroup": "=",
                        "subscriptionId": "=",
                        "resourceGroupName": "=",
                        "tenant": "=",
                        "dataLakeStoreUri": "=",
                        "baseUrl": "=",
                        "database": "=",
                        "serviceEndpoint": "=",
                        "batchUri": "=",
                        "databaseName": "=",
                        "systemNumber": "=",
                        "server": "=",
                        "url":"=",
                        "aadResourceId": "=",
                        "connectionString": "|:-connectionString:secureString"
                    }
                }
            },
            "Odbc": {
                "properties": {
                    "typeProperties": {
                        "userName": "=",
                        "connectionString": {
                            "secretName": "="
                        }
                    }
                }
            }
        },
        "Microsoft.DataFactory/factories/datasets": {
            "*": {
                "properties": {
                    "typeProperties": {
                        "folderPath": "=",
                        "fileName": "="
                    }
                }
            }}
    }


    2) Open "Data Factory", change some information and click "Publish all"

    By doing this the file "ARMTemplateParametersForFactory.json" will be changed and the new parameters will be displayed.

    3) In the Azure DevOps pipeline I put the returned value from task "Azure Key Vault: Pull Secrets"

    4) After deployment the value was successfully replaced in the testing environment

    The problem with this solution is:

    * It was necessary to create a new parameter file

    * The parameter 'x-api-Key' will not use Key Vault in development, only in other environments will the value stored in Key Vault be used. If the option to use a "Web Activity" is used then you must create a Key Vault secret to store the "Secret Identifier" value. I don't know which option is more acceptable.

    References:

    https://stackoverflow.com/questions/53659395/how-to-get-the-azure-data-factory-parameters-into-the-arm-template-parameters-fi
    https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment#use-custom-parameters-with-the-resource-manager-template

    Best regards,
    Cristina


    Friday, January 10, 2020 8:44 PM

All replies

  • Hi Cristina,

    You can use a web activity to hit the Azure Key Vault REST API and retrieve the key.

    You can then chain a set variable activity to store the output (key retrieved from the REST API) in a pipeline variable.

    Finally you would chain the copy activity and in the headers just pass the variable (eg- @variables('x')).

    Hope this helps.

    Thursday, January 9, 2020 9:10 AM
  • Hello @ChiragMishra-MSFT,

    Thanks for the reply, I saw this option on the link below, but I have a question. I'm using Azure DevOps to replicate the code to other environments. Do you know how to rename the Key Vault in code to use the name of the testing / production environments?

    https://docs.microsoft.com/en-us/azure/data-factory/how-to-use-azure-key-vault-secrets-pipeline-activities

    Best regards,
    Cristina
    Thursday, January 9, 2020 1:25 PM
  • Hi Cristina,

    I found an interesting blog that talks about the same. Please have a look at Step 2f in the below blog :

    http://datanrg.blogspot.com/2019/02/continuous-integration-and-delivery.html

    Please not that this is not an official Microsoft doc.

    In addition to this, you can also parameterize your AKV Linked Service. I have written an article on parameterizing Linked Services and datasets :

    https://social.technet.microsoft.com/wiki/contents/articles/53335.parameterizing-linked-services-and-datasets-in-azure-data-factory-v2-using-code-json.aspx

    Hope this helps.

    Friday, January 10, 2020 9:44 AM
  • Hi @ChiragMishra-MSFT,

    Thanks again for the help, I'm not able to create the variable using the Web Activity (https://github.com/MicrosoftDocs/azure-docs/issues/45892), but when I do I'll try to replace the Key Vault within the code.

    I already do this to replace the linked services names from the "ARMTemplateParametersForFactory.json" file, I didn't know that it was possible to change the name of the Key Vault inside the code using the "Azure Key Vault: Pull Secrets".

    Best regards,
    Cristina
    Friday, January 10, 2020 1:48 PM
  • Hi @ChiragMishra-MSFT,

    I used "Web Activity" directly on "Copy Activity" and it worked by setting "@activity('GetKey').output.value" to the "x-api-Key" parameter.

    However I couldn't rename the "Key Vault" in parameter "URL" of "Web Activity", in Azure DevOps I used "Azure Key Vault: Pull Secrets" but it didn't work, the name remained the same in the Testing Data Factory. Do you know what is missing or if there is any way to put the "URL" parameter in the "ARMTemplateParametersForFactory.json" file so that I can override the value?

    I want to change in the "URL" parameter the name of the "Key Vault" and the "Secret Identifier" as below:

    Before - Development Data Factory:

    https://kv-adventure-works-dsv.vault.azure.net/secrets/application-insights-key/c1b8xxxxxxxxxxxx5fea5b?api-version=7.0

    After - Testing Data Factory:

    https://kv-adventure-works-tst.vault.azure.net/secrets/application-insights-key/c1b8yyyyyyyyyyyy5fea5b?api-version=7.0

    Best regards,
    Cristina

    Friday, January 10, 2020 3:10 PM
  • Hello,

    I found a solution, I don't know if it's the best, but it was the only one at the moment. Below is the walkthrough:

    1) Create "arm-template-parameters-definition.json" file in Data Factory directory:

    In "Microsoft.DataFactory/factories/pipelines" the parameters "x-api-Key" and "url" were added. The "url" parameter has been set to test, but since "x-api-Key" worked it will not be used. The rest of the code is standard and was provided in the Microsoft documentation.

    {
        "Microsoft.DataFactory/factories/pipelines": {
            "properties": {
                "activities": [{
                    "typeProperties": {
                        "source": {
                           "additionalHeaders":{
                              "x-api-Key": "-::string"
                           }
                        },
                        "url": "-::string"
                    }
                }]
            }
        },
        "Microsoft.DataFactory/factories/integrationRuntimes":{
            "properties": {
                "typeProperties": {
                    "ssisProperties": {
                        "catalogInfo": {
                            "catalogServerEndpoint": "=",
                            "catalogAdminUserName": "=",
                            "catalogAdminPassword": {
                                "value": "-::secureString"
                            }
                        },
                        "customSetupScriptProperties": {
                            "sasToken": {
                                "value": "-::secureString"
                            }
                        }
                    },
                    "linkedInfo": {
                        "key": {
                            "value": "-::secureString"
                        },
                        "resourceId": "="
                    }
                }
            }
        },
        "Microsoft.DataFactory/factories/triggers": {
            "properties": {
                "pipelines": [{
                        "parameters": {
                            "*": "="
                        }
                    },  
                    "pipelineReference.referenceName"
                ],
                "pipeline": {
                    "parameters": {
                        "*": "="
                    }
                },
                "typeProperties": {
                    "scope": "="
                }
    
            }
        },
        "Microsoft.DataFactory/factories/linkedServices": {
            "*": {
                "properties": {
                    "typeProperties": {
                        "accountName": "=",
                        "username": "=",
                        "userName": "=",
                        "accessKeyId": "=",
                        "servicePrincipalId": "=",
                        "userId": "=",
                        "clientId": "=",
                        "clusterUserName": "=",
                        "clusterSshUserName": "=",
                        "hostSubscriptionId": "=",
                        "clusterResourceGroup": "=",
                        "subscriptionId": "=",
                        "resourceGroupName": "=",
                        "tenant": "=",
                        "dataLakeStoreUri": "=",
                        "baseUrl": "=",
                        "database": "=",
                        "serviceEndpoint": "=",
                        "batchUri": "=",
                        "databaseName": "=",
                        "systemNumber": "=",
                        "server": "=",
                        "url":"=",
                        "aadResourceId": "=",
                        "connectionString": "|:-connectionString:secureString"
                    }
                }
            },
            "Odbc": {
                "properties": {
                    "typeProperties": {
                        "userName": "=",
                        "connectionString": {
                            "secretName": "="
                        }
                    }
                }
            }
        },
        "Microsoft.DataFactory/factories/datasets": {
            "*": {
                "properties": {
                    "typeProperties": {
                        "folderPath": "=",
                        "fileName": "="
                    }
                }
            }}
    }


    2) Open "Data Factory", change some information and click "Publish all"

    By doing this the file "ARMTemplateParametersForFactory.json" will be changed and the new parameters will be displayed.

    3) In the Azure DevOps pipeline I put the returned value from task "Azure Key Vault: Pull Secrets"

    4) After deployment the value was successfully replaced in the testing environment

    The problem with this solution is:

    * It was necessary to create a new parameter file

    * The parameter 'x-api-Key' will not use Key Vault in development, only in other environments will the value stored in Key Vault be used. If the option to use a "Web Activity" is used then you must create a Key Vault secret to store the "Secret Identifier" value. I don't know which option is more acceptable.

    References:

    https://stackoverflow.com/questions/53659395/how-to-get-the-azure-data-factory-parameters-into-the-arm-template-parameters-fi
    https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment#use-custom-parameters-with-the-resource-manager-template

    Best regards,
    Cristina


    Friday, January 10, 2020 8:44 PM
  • Hi Cristina,

    Sorry for the delayed response. Glad to hear that your issue was resolved. Thanks for sharing your findings. It will help the community at large.

    Thursday, January 16, 2020 6:43 AM
  • Hi ChiragMishra-MSFT,

    No problem, do you think this is the best solution?

    Best regards,
    Cristina
    Thursday, January 16, 2020 1:43 PM
  • Hey Cristina,

    My recommendation would be to go the Web Activity route and maintain parameters (in the parameters jsonn file) for different environments to retrieve different secrets based on the environment.

    I would also recommend you to provide feedback at the feedback forum. All the feedback you share, is closely monitored by the Data Factory Product team and implemented in future releases.

    Friday, January 17, 2020 6:54 AM
  • Hi ChiragMishra-MSFT,

    Ok, thank you!

    Best regards,
    Cristina
    Saturday, January 18, 2020 12:43 AM