locked
Manually Running a scheduled DF pipeline RRS feed

  • Question

  • Hi,

    we have a pipeline that transfers data from datasource A to a reporting DB. The pipeline is scheduled to run every hour.

    Reports are generated regularly from the Reporting DB. 

    Sometimes our users call us saying they are not seeing certain records on the reports. This is because the records they are talking about are recently added entries and would only be transferred to the reporting DB in the next hourly run.

    Is there anything that we can do about this? I know i can increase the frequency of the DF run say 30 minutes or 15 minutes.

    But what i was hoping is in response to a users call like above, i would manually kick off the pipeline. Is this possible at all?

    thanks

    Wednesday, September 13, 2017 7:12 AM

Answers

All replies

  • at the moment this is not possible as the point in time where a valid slice can be generated has not been reached yet

    i assume you have set the slice to be processed at EndOfInterval

    If you only use your activity/pipeline to do some processing and you do not persist/copy any data you might simply change from EndOfInterval to StartOfInterval, this would kick off a processing at the beginning of the slice which, in your case, should not make a big difference

    once the slice is processed, you can reset it back manually to "Waiting" to make it run again

    -gerhard


    Gerhard Brueckl
    blogging @ http://blog.gbrueckl.at
    working @ http://www.pmOne.com

    Wednesday, September 13, 2017 9:52 AM
  • Hi,

    This is not possible out of the box. I have created a manual workaround to get this working though. You can find my blog post about it here: https://jorgklein.com/2016/11/14/orchestrate-azure-data-factory-pipelines-and-other-azure-data-platform-management-tasks-using-azure-automation/


    Jorg Klein's Microsoft Business Intelligence Blog

    • Marked as answer by MS_Prog Thursday, September 14, 2017 1:18 AM
    Wednesday, September 13, 2017 10:02 AM
  • Thanks Gerhard. But how to reset a processed slice back to "waiting". 

    sorry , i am quite new to this.

    regards,

    MS

    Wednesday, September 13, 2017 11:00 AM
  • the easiest would be to go to the Azure portal, select the ADF, select the output-dataset that is create by your activity, select the slice and click "Rerun"

    or you can use PowerShell to do this: https://docs.microsoft.com/en-us/powershell/module/azurerm.datafactories/Set-AzureRmDataFactorySliceStatus?view=azurermps-4.3.1

    there is also a REST API that you could use: https://docs.microsoft.com/en-us/rest/api/datafactory/

    -gerhard


    Gerhard Brueckl
    blogging @ http://blog.gbrueckl.at
    working @ http://www.pmOne.com

    • Marked as answer by MS_Prog Thursday, September 14, 2017 1:17 AM
    Wednesday, September 13, 2017 12:08 PM
  • This is now possible by selecting the Pipeline, clicking on the Trigger menu, and selecting Trigger Now.
    • Proposed as answer by ckphuaMVP Tuesday, February 19, 2019 6:51 PM
    Tuesday, February 19, 2019 6:50 PM
  • Hi ckphua,

    Thanks for adding your insights for the issue, which might be beneficial to other community members reading this thread. 

    Wednesday, February 20, 2019 6:55 AM