locked
Deleted Integration Run time still shows up in the ARM template and gets deployed in CI/CD pipeline RRS feed

  • Question

  • Hi,

    I have a Dev DataFactory where initially I had a self hosted integration run time defined and I was trying to use Azure Devops to then roll the changes out to a Test DataFactory. However, it caused errors and after reading the MSDN docs, I figured that I have to have a separate Data Factory with only the Self hosted Integration Runtime, which I then created and shared with all the other data factories. This made the CI/CD pipeline to work and that's great.

    However, since then, I deleted my old Integration Runtime that I created initially on my Dev DataFactory. However , it still gets pushed over to the Test DataFactory and when I tried to debug, it seems that the template generated in the adf_publish branch still has the old integration runtime , even though it was deleted and can't be seen in the UI.

    Why is this and what can be done to remove this? Please note, that every time I make a change , I obviously am publishing as well , so even the deletion should have been recorded in the adf_publish branch.

    How to remove this and prevent it from appearing in the Test environment during the release process?


    Wednesday, November 20, 2019 1:51 PM

Answers

  • Hi Again,

    Have now figured it out myself- So, just for the benefit of this thread posting the reason

    The issue was- I had a collaboration branch A, which I used for a while and then unlinked Data Factory from Git, moved back to Git and then set master as my collaboration branch and made all the necessary changes from there including publishing obviously (from master).

    But it seems that adf_publish kept also putting stuffs from my old collaboration branch A (even though it was not a collaboration branch anymore and was basically useless).

    Had to delete from that branch A (which at this point was basically useless), and then publish from master and though the message said nothing to publish (as there were practically no changes in master), but under the hood, it removed the old integration run time from the adf_publish branch.

    Wednesday, December 4, 2019 11:57 AM

All replies

  • Hi Saugat,

    You can try deleting the integration runtime reference from your ARM template under "parameters" and "resources" as shown highlighted below and then deploy the ARM template. You can then publish the changes to your branch and also merge it to master.

    {

        "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",

        "contentVersion": "1.0.0.0",

        "parameters": {

            "factoryName": {

                "type": "string",

                "metadata": "Data Factory name",

                "defaultValue": "****"

            },

            "integrationRuntime_integrationRuntimeTest": {

                "type": "string",

                "metadata": "Integration runtime name",

                "defaultValue": "integrationRuntimeTest"

            },

            "AzureBlobStorage1_connectionString": {

                "type": "secureString",

                "metadata": "Secure string for 'connectionString' of 'AzureBlobStorage1'"

            }

        },

        "variables": {

            "factoryId": "[concat('Microsoft.DataFactory/factories/', parameters('factoryName'))]"

        },

        "resources": [

            {

                "name": "[concat(parameters('factoryName'), '/Trash')]",

                "type": "Microsoft.DataFactory/factories/pipelines",

                "apiVersion": "2018-06-01",

                "properties": {

                    "description": "deleteme",

                    "activities": [

                        {

                            "name": "Set Variable1",

                            "type": "SetVariable",

                            "dependsOn": [],

                            "userProperties": [],

                            "typeProperties": {

                                "variableName": "yuck",

                                "value": "foo\\bar"

                            }

                        }

                    ],

                    "variables": {

                        "yuck": {

                            "type": "String"

                        }

                    },

                    "annotations": []

                },

                "dependsOn": []

            },

            {

                "name": "[concat(parameters('factoryName'), '/AzureBlobStorage1')]",

                "type": "Microsoft.DataFactory/factories/linkedServices",

                "apiVersion": "2018-06-01",

                "properties": {

                    "annotations": [],

                    "type": "AzureBlobStorage",

                    "typeProperties": {

                        "connectionString": "[parameters('AzureBlobStorage1_connectionString')]"

                    }

                },

                "dependsOn": []

            },

            {

                "name": "[concat(parameters('factoryName'), '/', parameters('integrationRuntime_integrationRuntimeTest'))]",

                "type": "Microsoft.DataFactory/factories/integrationRuntimes",

                "apiVersion": "2018-06-01",

                "properties": {

                    "type": "SelfHosted",

                    "typeProperties": {}

                },

                "dependsOn": []

            }

        ]

    }

    Hope this helps.

    Thursday, November 21, 2019 10:55 AM
  • Hi Chirag,

    Thanks for getting back. However, I have the following observations-

    The Data Factory is in Git mode and master is my collaboration branch (for now). The ARM templates are only available in adf_publish branch and the understanding is- that is only populated when someone publishes the changes to the Data Factory from the collaboration branch , which in this case happens to be the master branch.

    Is someone supposed to go in and manually change and check in to the adf_publish branch ? Isn't adf_publish branch supposed to be untouched and only the publish action from data factory allowed to push changes there?

    This is because no other branch has the ARM template. The master branch does not have the ARM template. Only the adf_publish has it.


    And even if the answer to the question is yes (meaning someone manually changing the adf_publish branch), don't you think it is something that should be taken care of automatically and not by someone going in and manually removing it from the UI and also from the generated template in the adf_publish branch?

    Feel free to correct if any of the understanding is incorrect.

    Also, I hope we are not talking about creating a fresh data factory from an ARM template.

    Thursday, November 21, 2019 12:59 PM
  • Hi Saugat,

    I tried reproducing the issue. Assuming you are on the master branch, once you delete an Integration Runtime  (and ofcourse all Linked Services linked to it) and you publish the changes, the ARM Template gets re-generated and it removes all references of the deleted IR.

    Please see the below screenshot for reference :

    Can you please try this out and see if it works.

    Monday, November 25, 2019 6:03 AM
  • Hi Chirag,

    Thanks again for getting back :).

    I did all that before raising this question. If you see my question above, I was expecting it to do the same but it didn't and that's why I raised this forum question.

    Master is my collaboration branch and since ADF only allows publishing from the collaboration branch, the publishing of course happened from the master branch (and yes, I did publish).

    I think this is a special case where something has gone wrong and I would probably open a Support case for this.

    Monday, November 25, 2019 8:34 AM
  • Hey Saugat,

    It looks strange and has to be a special case. 

    For a deeper investigation and immediate assistance on this issue, if you have a support plan you may file a support ticket, else could you please send an email to AzCommunity@Microsoft.com with the below details, so that we can create a one-time-free support ticket for you to work closely on this matter. 

    Thread URL: https://social.msdn.microsoft.com/Forums/en-US/ea9dc86e-595f-4c46-9a3d-5e41adcad161/deleted-integration-run-time-still-shows-up-in-the-arm-template-and-gets-deployed-in-cicd-pipeline?forum=AzureDataFactory
    Subscription ID: <your subscription id>

    Please let me know once you have done the same
    Wednesday, November 27, 2019 10:14 AM
  • Hi Again,

    Have now figured it out myself- So, just for the benefit of this thread posting the reason

    The issue was- I had a collaboration branch A, which I used for a while and then unlinked Data Factory from Git, moved back to Git and then set master as my collaboration branch and made all the necessary changes from there including publishing obviously (from master).

    But it seems that adf_publish kept also putting stuffs from my old collaboration branch A (even though it was not a collaboration branch anymore and was basically useless).

    Had to delete from that branch A (which at this point was basically useless), and then publish from master and though the message said nothing to publish (as there were practically no changes in master), but under the hood, it removed the old integration run time from the adf_publish branch.

    Wednesday, December 4, 2019 11:57 AM
  • Hi Saugat,

    Glad to know that the issue was resolved. Thanks for sharing your findings. This would help other community members reading this thread.

    Thursday, December 5, 2019 5:06 AM