locked
ADF Blob Triggers - Urgent RRS feed

  • Question

  • Hi!

    I am trying to configure a Blob Trigger to fire off my pipeline when the Blob File is written to the Blob Container. I have a main Folder called /azfunc-out/  with subfolders under this with numbers that represent our Client Company IDs. (Please see attachments). I ONLY want a trigger to fire if a particular subfolder has a new Blob written to it, i,e 

    /azfunc-out/8

    However my configuration is not working (see 2nd attachment). When I had blobs on the main folder I had too many triggers and pipelines running, so now I need to trigger per subfolder.

    Also, can I have X number of triggers per pipeline? We are in final QA Testing prior to going Live and this issue of the current trigger firing off too many pipelines came up.

    Thanks for your help!!
    Mike


    Mike Kiser



    • Edited by EMKISER Thursday, November 14, 2019 5:26 PM
    Thursday, November 14, 2019 5:22 PM

Answers

  • Hi Mike,

    Thanks you for reaching out. I see you have container name:  azfunc-out, Folder name: 8. Please configure the settings as below and give a try.



    For your second question: Total number of entities (all together) such as pipelines, datasets, triggers, linked services and integration runtime within a data factory = 5000 limit. In theory you could have 4999 triggers and a single pipeline. But realistically, you will have some number less than 4999, because you will need to use datasets, linked services in your pipeline to do anything. :)


    Additional information: Please refer to this GitHub issue for Event trigger configuration: https://github.com/MicrosoftDocs/azure-docs/issues/42345 

    Hope this helps your query. Let me know if you see any issues.


    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Thursday, November 14, 2019 10:28 PM
  • Hi,

    Did you tried to set value "Blob path begins with" to 8/


    Darko Milevski http://mkdot.net/blogs/darko/

    • Marked as answer by EMKISER Friday, November 15, 2019 3:08 PM
    Thursday, November 14, 2019 10:22 PM
  • Also please refer to below MS doc to know about Azure Data Factory limitations.


    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    • Marked as answer by EMKISER Friday, November 15, 2019 3:08 PM
    Thursday, November 14, 2019 10:32 PM
  • Hi Again Kranthi,

    When I use Blob Path begins with 8, it works. Then I want to create ANOTHER trigger for /azfunc-out/220 (just like the folder structure with /azfunc-out/8, but the sub-folder is 220.....it won't let me save the trigger for 220...I get the following error. I should be able to put a trigger on any folder, right? This is driving me crazy lol

    Thanks!

    Mike

    lobTrigger220
    Failed to save BlobTrigger220. Error: {"$id":"1","innerException":null,"message":"The path '/ScadaVisorADF/trigger/BlobTrigger220.json' specified in the add operation already exists. Please specify a new path.\r\nParameter name: newPush","typeName":"Microsoft.TeamFoundation.SourceControl.WebServer.InvalidArgumentValueException, Microsoft.TeamFoundation.SourceControl.WebServer, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a","typeKey":"InvalidArgumentValueException","errorCode":0,"eventId":0}
    a few seconds ago

    Please see my screenshot....


    Essentially, I want a trigger set for any JSON file that is placed in each of the following folders. Please assist. Thanks!!
    Mike

    Mike Kiser


    • Edited by EMKISER Friday, November 15, 2019 12:34 AM
    • Marked as answer by EMKISER Friday, November 15, 2019 3:08 PM
    Friday, November 15, 2019 12:24 AM
  • Hi Mike, 

    As Vibhav mentioned, looks like there is already an existing trigger with name 'BlobTrigger220'. Please try renaming your trigger, it should be created without issues.

    Since you are having multiple folders and your requirement needs multiple triggers, I did experimented a bit with event trigger to see if we can reduce the number of triggers and below are my findings.

    When you have multiple folders starting with 1 (For example : 1, 10, 11, 12, 121, 1212, 17129, ...) and if you want to trigger your pipeline, when a blob is dropped/created into any of these (folder mentioned in example) folders, then you can have one single trigger with below configuration.

    Folder begin with 1:


    Note: If you have a sub-folder named 1-1 inside folder named 1 and a file/blob is created inside the sub-folder 1-1, even in this case the event trigger will be fired.

    By above configuration, if a blob is created in any folder starting with name 1, in container named azfunc-out, then event trigger is fired. 

    By looking at the folders in the above screen print that you have shared, you can reduce the number of triggers with above configuration.

    If you want to trigger the pipeline only when a blob is created in folder named 1, then you can have configuration as below (Blob path begin should be 1/)


    Hope this helps. Let me know if you have any questions.

       

    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered"Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.


    Friday, November 15, 2019 7:32 AM
  •  When I had blobs on the main folder I had too many triggers and pipelines running, so now I need to trigger per subfolder.

    If you mean that many concurrent runs of same pipeline start running then you can set Concurrency to 1 in pipeline --> general tab


    If the response helped, do "Mark as answer" or Upvote it
    - Vaibhav

    • Marked as answer by EMKISER Friday, November 15, 2019 3:03 PM
    Friday, November 15, 2019 7:53 AM
  • Hi Vaibhav,

    Thanks for your reply; you were correct. The duplicate names were in my Git Repo and were not showing up on the UI. Now I am able to add a trigger per folder.

    I really appreciate your response!
    Mike Kiser


    Mike Kiser

    • Marked as answer by EMKISER Friday, November 15, 2019 2:59 PM
    Friday, November 15, 2019 2:59 PM

All replies

  • Hi,

    Did you tried to set value "Blob path begins with" to 8/


    Darko Milevski http://mkdot.net/blogs/darko/

    • Marked as answer by EMKISER Friday, November 15, 2019 3:08 PM
    Thursday, November 14, 2019 10:22 PM
  • Hi Mike,

    Thanks you for reaching out. I see you have container name:  azfunc-out, Folder name: 8. Please configure the settings as below and give a try.



    For your second question: Total number of entities (all together) such as pipelines, datasets, triggers, linked services and integration runtime within a data factory = 5000 limit. In theory you could have 4999 triggers and a single pipeline. But realistically, you will have some number less than 4999, because you will need to use datasets, linked services in your pipeline to do anything. :)


    Additional information: Please refer to this GitHub issue for Event trigger configuration: https://github.com/MicrosoftDocs/azure-docs/issues/42345 

    Hope this helps your query. Let me know if you see any issues.


    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Thursday, November 14, 2019 10:28 PM
  • Also please refer to below MS doc to know about Azure Data Factory limitations.


    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    • Marked as answer by EMKISER Friday, November 15, 2019 3:08 PM
    Thursday, November 14, 2019 10:32 PM
  • Hey Darko,

    Yes, but that didn't work either.

    Thanks

    Mike


    Mike Kiser

    Friday, November 15, 2019 12:17 AM
  • Hi Again Kranthi,

    When I use Blob Path begins with 8, it works. Then I want to create ANOTHER trigger for /azfunc-out/220 (just like the folder structure with /azfunc-out/8, but the sub-folder is 220.....it won't let me save the trigger for 220...I get the following error. I should be able to put a trigger on any folder, right? This is driving me crazy lol

    Thanks!

    Mike

    lobTrigger220
    Failed to save BlobTrigger220. Error: {"$id":"1","innerException":null,"message":"The path '/ScadaVisorADF/trigger/BlobTrigger220.json' specified in the add operation already exists. Please specify a new path.\r\nParameter name: newPush","typeName":"Microsoft.TeamFoundation.SourceControl.WebServer.InvalidArgumentValueException, Microsoft.TeamFoundation.SourceControl.WebServer, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a","typeKey":"InvalidArgumentValueException","errorCode":0,"eventId":0}
    a few seconds ago

    Please see my screenshot....


    Essentially, I want a trigger set for any JSON file that is placed in each of the following folders. Please assist. Thanks!!
    Mike

    Mike Kiser


    • Edited by EMKISER Friday, November 15, 2019 12:34 AM
    • Marked as answer by EMKISER Friday, November 15, 2019 3:08 PM
    Friday, November 15, 2019 12:24 AM

  • lobTrigger220
    Failed to save BlobTrigger220. Error: {"$id":"1","innerException":null,"message":"The path '/ScadaVisorADF/trigger/BlobTrigger220.json' specified in the add operation already exists. Please specify a new path.\r\nParameter name: newPush","typeName":"Microsoft.TeamFoundation.SourceControl.WebServer.InvalidArgumentValueException, Microsoft.TeamFoundation.SourceControl.WebServer, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a","typeKey":"InvalidArgumentValueException","errorCode":0,"eventId":0}
    a few seconds ago

    I was able to create 2 triggers for 2 different folders in a same storage account. 

    Based on the error, looks like you already have a trigger available named BlobTrigger220 (Check in repo as well if it is git enabled ADF)

    Try giving it a different name BlobTrigge220-1 and retry. Also add slash in begin with value - 220/


    If the response helped, do "Mark as answer" or Upvote it
    - Vaibhav

    Friday, November 15, 2019 5:09 AM
  • Hi Mike, 

    As Vibhav mentioned, looks like there is already an existing trigger with name 'BlobTrigger220'. Please try renaming your trigger, it should be created without issues.

    Since you are having multiple folders and your requirement needs multiple triggers, I did experimented a bit with event trigger to see if we can reduce the number of triggers and below are my findings.

    When you have multiple folders starting with 1 (For example : 1, 10, 11, 12, 121, 1212, 17129, ...) and if you want to trigger your pipeline, when a blob is dropped/created into any of these (folder mentioned in example) folders, then you can have one single trigger with below configuration.

    Folder begin with 1:


    Note: If you have a sub-folder named 1-1 inside folder named 1 and a file/blob is created inside the sub-folder 1-1, even in this case the event trigger will be fired.

    By above configuration, if a blob is created in any folder starting with name 1, in container named azfunc-out, then event trigger is fired. 

    By looking at the folders in the above screen print that you have shared, you can reduce the number of triggers with above configuration.

    If you want to trigger the pipeline only when a blob is created in folder named 1, then you can have configuration as below (Blob path begin should be 1/)


    Hope this helps. Let me know if you have any questions.

       

    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered"Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.


    Friday, November 15, 2019 7:32 AM
  •  When I had blobs on the main folder I had too many triggers and pipelines running, so now I need to trigger per subfolder.

    If you mean that many concurrent runs of same pipeline start running then you can set Concurrency to 1 in pipeline --> general tab


    If the response helped, do "Mark as answer" or Upvote it
    - Vaibhav

    • Marked as answer by EMKISER Friday, November 15, 2019 3:03 PM
    Friday, November 15, 2019 7:53 AM
  • Hi Vaibhav,

    Thanks for your reply; you were correct. The duplicate names were in my Git Repo and were not showing up on the UI. Now I am able to add a trigger per folder.

    I really appreciate your response!
    Mike Kiser


    Mike Kiser

    • Marked as answer by EMKISER Friday, November 15, 2019 2:59 PM
    Friday, November 15, 2019 2:59 PM
  • Thanks Kranthi!

    I appreciate the time you took in testing the above; that helped me tremendously to understand what I need to do to meet my requirements....

    Thanks again!
    Mike


    Mike Kiser

    Friday, November 15, 2019 3:03 PM
  • Glad to be of help, Mike! :)

    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Friday, November 15, 2019 8:54 PM