none
ADF: How to trigger a pipeline when the files are ready in file system?

    Question

  • Dear All,

    We have a pipeline which is taking source files from FTP server. Now, I have scheduled the pipeline to run it in a specified time. But, I would prefer to run it whenever the files are ready in the FTP server. How can I achieve this?

    Thank You

    Julie

    Tuesday, October 9, 2018 8:27 AM

Answers

  • Well you can use the Azure logic apps here to listen to your FTP source and whenever any file is added or modified - you can invoke a webhook (Http) which essentially would start your Azure data factory pipeline or you can directly trigger your pipeline from within logic App natively. 


    Bhushan Gawale | RapidCircle

    Hi Bhushan,

    That worked for me but it's very expensive.

    Thank You

    Julie

    Thursday, October 11, 2018 1:28 PM

All replies

  • Hi Julie,

    Your requirement smells like what Event-based trigger does. But currently Event-based trigger only supports Version2 Azure Storage account. You could consider using Azure Storage account. Hope this helps. Thanks.

    Tuesday, October 9, 2018 8:54 AM
  • Hi Julie,

    Your requirement smells like what Event-based trigger does. But currently Event-based trigger only supports Version2 Azure Storage account. You could consider using Azure Storage account. Hope this helps. Thanks.

    Hello Wang Zhang,

    Thank You for your email!

    I know about Event triggers. But, our current architecture is using FTP as source path. I'm thinking if we can identify the presence of files using web function or some other stuff then trigger the pipeline within that new pipeline.

    Thank You

    Julie

    Tuesday, October 9, 2018 9:04 AM
  • Yep, you could consider using Web Activity to check wether file exists and then execute copying data based on the result of the web activity.

    Thanks.

    Tuesday, October 9, 2018 9:12 AM
  • Well you can use the Azure logic apps here to listen to your FTP source and whenever any file is added or modified - you can invoke a webhook (Http) which essentially would start your Azure data factory pipeline or you can directly trigger your pipeline from within logic App natively. 


    Bhushan Gawale | RapidCircle

    Tuesday, October 9, 2018 11:21 AM
  • Well you can use the Azure logic apps here to listen to your FTP source and whenever any file is added or modified - you can invoke a webhook (Http) which essentially would start your Azure data factory pipeline or you can directly trigger your pipeline from within logic App natively. 


    Bhushan Gawale | RapidCircle

    Hi Bhushan,

    That worked for me but it's very expensive.

    Thank You

    Julie

    Thursday, October 11, 2018 1:28 PM
  • Can you please elaborate when you say it is very expensive? are you referring to Logic apps or the data factory pricing?

    Azure logic apps pricing is pay as you go basis i.e. every time your logic app runs then the triggers, connectors and actions are metered and you are charged for it, this makes it quite cost effective and you can validate it by visiting the azure cost calculator here https://azure.microsoft.com/en-in/pricing/calculator/


    Bhushan Gawale | RapidCircle


    Thursday, October 11, 2018 2:31 PM