none
Standalone Trigger in Azure data factory RRS feed

  • Question

  • Hi Team,

    I have trigger created for a pipeline to be executed but I cant schedule that since it a standalone job I will run on adhoc basis on the business requirement. So is there any chance that I can just run that trigger where ever I need without going to the Pipeline activity and using the debug option with providing the parameters.

    Thanks

    Kalyan 

    Friday, September 13, 2019 11:29 AM

Answers

  • Hi ChiragMishra,

    We want to execute pipeline directly from Azure monitor not from the ADF at all.

    How about running the pipeline using Trigger now option from pipeline for the first time. 

    Later, you can run the same pipeline using rerun button from Monitor section.

    Will that work?


    Cheers
    Vaibhav
    MCSA (SQL Server 2014)

    Wednesday, September 18, 2019 10:38 AM

All replies

  • Hi Kalyan,

    You can definitely invoke the trigger on-demand, whenever you want.

    Please have a look at this doc to read about the same.

    You can manually trigger your pipeline by using one of the following methods:

    • .NET SDK
    • Azure PowerShell module
    • REST API
    • Python SDK

    REST API

    The following sample command shows you how to manually run your pipeline by using the REST API:

    POST
    https://management.azure.com/subscriptions/mySubId/resourceGroups/myResourceGroup/providers/Microsoft.DataFactory/factories/myDataFactory/pipelines/copyPipeline/createRun?api-version=2017-03-01-preview
    

    For a complete sample, see Quickstart: Create a data factory by using the REST API.

    Azure PowerShell

    The following sample command shows you how to manually run your pipeline by using Azure PowerShell:

    Invoke-AzDataFactoryV2Pipeline -DataFactory $df -PipelineName "Adfv2QuickStartPipeline" -ParameterFile .\PipelineParameters.json

    You pass parameters in the body of the request payload. In the .NET SDK, Azure PowerShell, and the Python SDK, you pass values in a dictionary that's passed as an argument to the call:

    JSON<button aria-label="Copy code" class="action" data-bi-name="copy" style="box-sizing:inherit;font-family:inherit;font-size:12.8px;margin:0px;cursor:pointer;padding:2px 10px;border-left-width:1px;display:flex;align-items:center;">Copy</button>
    {
      "sourceBlobContainer": "MySourceFolder",
      "sinkBlobContainer": "MySinkFolder"
    }
    

    The response payload is a unique ID of the pipeline run:

    JSON
    {
      "runId": "0448d45a-a0bd-23f3-90a5-bfeea9264aed"
    }
    

    For a complete sample, see Quickstart: Create a data factory by using Azure PowerShell.

    .NET SDK

    The following sample call shows you how to manually run your pipeline by using the .NET SDK:

    C#
    client.Pipelines.CreateRunWithHttpMessagesAsync(resourceGroup, dataFactoryName, pipelineName, parameters)
    

    For a complete sample, see Quickstart: Create a data factory by using the .NET SDK.

    Friday, September 13, 2019 11:50 AM
    Owner
  • Thanks Chirag Mishra.

    I have gone through the document which you have mentioned earlier.

    I want to know whether we can execute the trigger on demand from the triggers window without using the powershell or any other third party.

    Thanks

    Kalyan 

     

    Friday, September 13, 2019 1:10 PM
  • Hi Kalyan,

    You can manually trigger a pipeline run on-demand by clicking "Trigger Now" under Add Trigger in the Pipeline UI as shown in the below screenshot.

    Hope this helps.

    Monday, September 16, 2019 10:18 AM
    Owner
  • Hi ChiragMishra,

    We want to execute pipeline directly from Azure monitor not from the ADF at all.

    Tuesday, September 17, 2019 1:14 PM
  • Hi Kalyan,

    Thanks for clarifying the ask. Unfortunately as of today, there's no way to directly invoke a pipeline run from Azure Monitor directly. You can monitor pipeline runs on Azure monitor however.

    Wednesday, September 18, 2019 9:10 AM
    Owner
  • Hi ChiragMishra,

    We want to execute pipeline directly from Azure monitor not from the ADF at all.

    How about running the pipeline using Trigger now option from pipeline for the first time. 

    Later, you can run the same pipeline using rerun button from Monitor section.

    Will that work?


    Cheers
    Vaibhav
    MCSA (SQL Server 2014)

    Wednesday, September 18, 2019 10:38 AM
  • Hi Vaibhav,

    Thats works but if the logs are going beyond the time let say I have some thousands of Pipelines running on daily basis will I the pipeline name still what has been ran on last year.

    Wednesday, September 18, 2019 11:45 AM
  • Hi Vaibhav,

    Thats works but if the logs are going beyond the time let say I have some thousands of Pipelines running on daily basis will I the pipeline name still what has been ran on last year.

    If you have thousands of pipeline running daily, obviously - these will show up on top in Monitor section.

    Now if you want to run your pipeline that you want run on demand, you can use filtering option - filter pipeline by name or the filters and then rerun it. 

    You should be able to see your previous run no matter when it was run last time



    Cheers
    Vaibhav
    MCSA (SQL Server 2014)

    Wednesday, September 18, 2019 12:02 PM
  • My question is if I have used Trigger now then it is not a schedule pipeline then 1000 pipelines are running on daily basis and when I want to run the pipeline (XYZ) which has been triggered manually with (Trigger Now) will that be avaiable in the Monitor.

    Since we know that GUI will have some limit to show about few days of data in the GUI only or few pipelines data 

    that is my concern.

    Thanks

    Kalyan 

    Wednesday, September 18, 2019 4:30 PM

  • Since we know that GUI will have some limit to show about few days of data in the GUI only or few pipelines data 

    that is my concern.

    Thanks

    Kalyan 

    I don't think there is any limit as such. I couldn't find any limit to view historical runs in below:

    https://github.com/MicrosoftDocs/azure-docs/blob/master/includes/azure-data-factory-limits.md

    If you run a pipeline-XYZ today and you have scheduled 1000 pipelines running daily. After 6 Months, you want to look back, find and rerun the XYZ pipeline again, you can always Filter the pipelines by XYZ name and also by the dates. 


    Cheers
    Vaibhav
    MCSA (SQL Server 2014)

    Thursday, September 19, 2019 7:35 AM
  • Thanks Vaibhav.

    We not sure on that we are asumming there will limit generally in any tool of the GUI we can see only few days or months of statistics.Did you study any where that they is no limit if so can you please share that will be really helpful.

    Thanks

     

    Friday, September 20, 2019 10:18 AM