locked
Re-deploy pipelines and data sets manually in azure data factory v1 RRS feed

  • Question

  • A few days ago, i was able to edit pipelines and data sets manually as per my requirement in azure data factory v1 .Now i am unable to find to a way to edit my query in pipeline and data sets due to recent changes in user interface(UI).

    I would really appreciate if you could help in finding a way to edit pipelines and data sets

    Thursday, August 29, 2019 12:09 PM

All replies

  • Hello mirzamwb10, and thank you for your inquiry.  The UI editor in the portal has been discontinued, but there are still ways to edit your pipelines and datasets.  One way is through powershell. Link to powershell DataFactory reference (Az module version).

    The first step is to install one of the powershell modules

    Install-Module -Name AzureRM.DataFactories

    or

    Install-Module -Name Az -AllowClobber -Scope CurrentUser

    Do not install both.  To see which one is right for you visit How to install the Azure powershell module.  I recommend the second one (Az, not AzureRM).

    If you do not want to use powershell, it is possible to use Visual Studio 2015 with the Azure SDK.

    Please let me know if you need further assistance or have more questions.

    Thursday, August 29, 2019 11:50 PM
  • mirzamwb10 , did my suggestion help you?  If it did please mark as answered, if it did not, please let me know what else I can do to assist.
    Friday, August 30, 2019 11:26 PM
  • can you provide complete steps with visual studio 2017 to edit pipelines and data sets  ??

    it would really help me as i am not able to do it with power shell


    • Edited by mirzamwb10 Monday, September 2, 2019 7:02 AM
    Monday, September 2, 2019 7:02 AM
    1. Get Visual Studio 2015, Version 14.0.25431.01 Update 3,  Microsoft .NET Framework Version 4.8.03752
    2. Get Azure SDK for VS 2015, Update 3, from Azure Download Page 
    3. Get the ADF plugin for VS from https://marketplace.visualstudio.com/items?itemName=AzureDataFactory.MicrosoftAzureDataFactoryToolsforVisualStudio2015
    4. Launch the server explorer and connect to your subscription and navigate to the Data Factory
    5. Right click the Data Factory and export as new project
    6. In the solution explorer you should be able to see the new data factory project.  Expanding the project should show you the resources.

    Fix: replaced DataFactoryTools link so it now points to 2015.
    Image: Where to find server explorer (top left corner, expands)

    Image: Right clicking on your factory gives option to Export

    Image: the definition files available in Solution explorer after exporting (redacted)

    Tuesday, September 3, 2019 7:25 PM
  • My apologies that took so long.  I have updated the directions post, with a fix and added images.  I left out the steps where you log in to your account, as that should be straightforward.  Let me know if this helps.
    Thursday, September 5, 2019 12:20 AM
  • Thank you so much for the help.. But still i am not able to find data factory after following your steps.

    Note: i used visual studio 2017.

    • Edited by mirzamwb10 Thursday, September 5, 2019 12:50 PM
    Thursday, September 5, 2019 12:49 PM
  • This is likely because you are using Visual Studio 2017.  For the AzureDataFactoryTools, I found versions for Visual Studio 2013 and Visual Studio 2015.  I have not found a version for Visual Studio 2017.
    Thursday, September 5, 2019 9:23 PM
  • can you please tell.. how we can edit only a single pipeline and build and deploy only that file..
    Wednesday, September 11, 2019 2:30 PM
  • If you are using the Visual Studio 2015, the way to publish is right click on the factory icon inside your solution, and select publish.  During the publish process, it is possible to choose which items to publish.

    If you want to fetch your existing pipeline definitions, not using Visual Studio, the way to do it is through a REST API call.  This can be accomplished using a tool like Postman, or by the PowerShell script attached at the bottom of this post.  In either method (Postman or PowerShell) you will need to create an Azure Active Directory App for the purposes of authentication.

    You do not need to have PowerShell installed on your computer.  In the portal, go to your Data Factory.  Look in the top right corner.  Click the terminal button (shown in picture below).  This will get you a command line.  If this is your first time using the terminal, you may get a few prompts such as choosing a storage account.

    You can choose whether to use PowerShell or Bash.  Unfortunately, the Data Factory module isn't available in Bash at this time.  Therefore switch to PowerShell if need be.

    Here is the script.  You will need to fill in the <placeholders> with info from the App, your subscription, and your Data Factory.  The output is placed in the location specified in the next-to-last line.  I reccomend changing from 

    "E:\V1ReadJsons\test.json"

    to something like

    "./$PipelineName"

    #<powershell script> 
    Connect-AzAccount 
    $tenantID = "" 
    $appId = "<AAD APP ID>" 
    $authKey = "<Secret key id>" 
    $subsId = "<your subscription id>" 
    $resourceGroup = "<your resource group name>" 
    $dataFactoryName = "<your data factory name>" 
    $pipelineName="<your pipeline name>" 
    $apiVersion = "2015-10-01" 
    $AuthContext = [Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext]"https://login.microsoftonline.com/${tenantId}" 
    $cred = New-Object -TypeName Microsoft.IdentityModel.Clients.ActiveDirectory.ClientCredential -ArgumentList ($appId, $authKey) 
    $result = $AuthContext.AcquireTokenAsync("https://management.core.windows.net/", $cred).GetAwaiter().GetResult() 
    $authHeader = @{ 
    'Content-Type'='application/json' 
    'Accept'='application/json' 
    'Authorization'=$result.CreateAuthorizationHeader() 
    } 
    $request = "https://management.azure.com/subscriptions/$subsId/resourcegroups/$resourceGroup/providers/Microsoft.DataFactory/datafactories/$dataFactoryName/pipelines/$pipelineName?api-version=$apiVersion" 
    $response = Invoke-WebRequest -Method GET -Uri $request -Header $authHeader  
    $response.Content | Out-File "E:\V1ReadJsons\test.json" 
    #</powershell script> 
    Thursday, September 12, 2019 12:04 AM