none
Enablement of loop within loop in ADF v2 RRS feed

  • Question

  • Hello Team,

    I am unable to add any loop component (foreach,untill) within other loop activity.

    So is this a known issue? And is there any alternative for the same?

    Also, for scheduled trigger in ADF v2,I need to pass the windows slice(ADf v1 parameter) or the runtime as a parameter to the pipeline rather than UTCNow() .

    The requirement is if a pipeline fails for the date :3/20/2019 and I need to rerun the pipeline for the next day,I need to have the run start as parameter rather than UTCnow to pass 3/20/19 as value rather than 3/21/19.

    Is there any way for the above issue?

    Thanks

    Monday, April 1, 2019 6:24 AM

Answers



  • 1) It is not possible at the present time to use a ForEach activity inside another ForEach activity.  I do not know if / when that might be implemented.

    2) Parameters and variables are available in Data Factory V2.  The main difference, is that parameters are passed into the pipeline when triggered, whereas variables are set during runtime.  Both can have default values.  Both can be referenced in any dynamic content.  Do take note that variables cannot be incremented/decremented directly (myVar = myVar + 1, fails).

    Parameters are referenced by "@pipeline().parameters.<parameterName>"
    Variables are referenced by "@variables.<variableName>"
    By slice date, do you mean something like this article ?

    3)  I haven't actually tried to do permissioning on Data Factory before.  It might be possible with app access, but I'd need to look into that more.

    Tuesday, April 2, 2019 8:28 PM
    Moderator

All replies

  • Hello Nandan Hegde and thank you for your question.

    My understanding of your scenario is like this:
    You are running some churn for a few days at a time.  Each day's churn depends on the previous day's churn

    Since you are unable to nest loops, you could instead try nesting pipeline execution.  This means you would have 2 pipelines.  Pipeline X has a loop, and in that loop calls pipeline Y.  Pipeline Y has a loops that runs your buisness logic.  When pipeline Y returns a failure response to pipeline X, pipeline X can try calling pipeline Y again.

    Monday, April 1, 2019 8:09 PM
    Moderator
  • On second thought, I must have misunderstood.  You are probably running a job every day.  Sometimes it fails.  If the previous day failed (perhaps data arrived after job start), you want to re-run the failed day before running next day.

    You need a way to tell if the previous day failed.  There are different ways to do this.

    • The easiest way to do is check whether the output dataset exists.
    • Another way is to modify your pipeline to record the success/failure in a blob or sql table.  Then you can lookup whether or not the previous day needs to be rerun.
    • Another way is to set up a function to look at Data Factory's pipeline runs programatically.  Here is document about programatically looking up pipeline runs.
    Monday, April 1, 2019 8:28 PM
    Moderator
  • Hello Martin,

    Thank you for your response. Basically I have multiple queries pertaining to ADF v2.

    1) Can you implement loop activity with loop activity in the same pipeline.

    Ans: As per your input,it is not directly possible but we can create a pipeline and then trigger the pipeline within the loop activity.   

    But is Microsoft going to implement loop within loop feature in future?

    2) Currently there is no way to trigger a pipeline on any nth business day. So I have created a table wherein i have populated the exact business days for a job to be triggered.And I am scheduling the ADF pipeline daily and based on IF condition activity comparing whether the pipeline schedule day matches to the extract date in the table

    Now consider a scenario wherein the file fails to arrive on 3rd BD then I would have to rerun the pipeline for that 3rd slice (so that the slice date of ADf and extractdate would match and ADF would rerun successfully).

    So I wanted to know how to pass slice date as parameter rather than UTCnow.Is it even possible in ADF v2? 

    Note: This is scheduled trigger


    3) Is it possible to give any user permission to rerun an activity via monitor and manage but disable his access for any json update or pipeline creation (ADF v1 and ADF v2 included)

    Tuesday, April 2, 2019 2:10 PM


  • 1) It is not possible at the present time to use a ForEach activity inside another ForEach activity.  I do not know if / when that might be implemented.

    2) Parameters and variables are available in Data Factory V2.  The main difference, is that parameters are passed into the pipeline when triggered, whereas variables are set during runtime.  Both can have default values.  Both can be referenced in any dynamic content.  Do take note that variables cannot be incremented/decremented directly (myVar = myVar + 1, fails).

    Parameters are referenced by "@pipeline().parameters.<parameterName>"
    Variables are referenced by "@variables.<variableName>"
    By slice date, do you mean something like this article ?

    3)  I haven't actually tried to do permissioning on Data Factory before.  It might be possible with app access, but I'd need to look into that more.

    Tuesday, April 2, 2019 8:28 PM
    Moderator
  • Hi Martin,

    Thank you for your inputs.

    And by slice date,I mean by the article which you have provided regarding ADF v1.

    And awaiting for your response for the 3rd point:)

    Thursday, April 4, 2019 2:58 AM
  • Hello Martin,

    Just a query on our input w.r.t 2nd point:

    I am joining 2 execute pipeline activities within a pipeline.

    So is it possible to provide the activity output of 1 pipeline as an input value to the parameter of the 2nd pipeline?

    Friday, April 5, 2019 5:02 AM
  • Hello Nandan Hegde, my apologies for slow answers.

    Getting the output of a pipeline can be tricky.

    Here you can see that the output of the "Execute Pipeline" activity is a pipelineRunId and pipelineName.

    I am sure this is not what you are looking for.  You want the details, what happened to that pipeline run.  To get this programattically, you take  the pipelineRunId, and use it in an API call, such as in REST: Get pipeline run by ID.
    However, if you are simply looking for success/fail, there is an easier way.

    The secret to this is:

    A word of caution, things can behave unexpectedly when cancelling a pipeline run.

    Saturday, April 6, 2019 12:06 AM
    Moderator
  • Saturday, April 6, 2019 12:10 AM
    Moderator
  • Hello Martin,

    My requirement is as below:

    I have created a pipeline wherein I have a lookup activity:

    I have created a pipeline wherein I am invoking 2 pipeline activities and I want to pass the output of the lookup activity of 1 pipeline as an input into another pipeline activity as shown below:

    Is it possible to achieve the above scenario directly or do I need to create parameters within each pipeline and then assign and pass the parameters.

    Monday, April 8, 2019 12:55 PM
  • To grab the output of the Lookup activity from "FileValidateionPipeline", you can use this api call, to get the results of all activities in that pipeline run.  You could do the call either immediately after the "FileValidationPipeline" , or inside the "Execute Pipeline1".  In either case , you would still need to add a parameter to the "ExecutePipeline1".  No parameter needs to be created inside "FileValidationPipeline".

    In my opinion, it would be easiest to just do the Lookup inside your main pipeline, instead of calling "FileValidationPipeline".
    Monday, April 8, 2019 9:12 PM
    Moderator
  • Hello Martin,

    Can a variable created within the FileValidationPipeline be passed to parameter of the Execute Pipeline 1.

    Since I am unable to do so ?

    Tuesday, April 9, 2019 11:09 AM