none
For-Each Loop with Sequential un-ticked RRS feed

  • Question

  • Hi,

    In my pipeline (Pipeline_A) I have a Lookup activity that populates a dataset with a list of table names, for arguments sake, let’s say there are 6 table names as follows:

    Table1
    Table2
    Table3
    Table4
    Table5
    Table6

    The next activity is a ForEach activity and loops through items  from the previous Lookup activity (using @activity('Some Lookup').output.value).

    Within the ForEach activity I have two further activities.  One first sets a variable with the table name using @item().TableName.  On success of that, the second activity executes another pipeline (Pipeline_B) passing in the table name variable.

    In the Settings of the ForEach activity, Sequential is un-ticked.

    When I set up a trigger on Pipeline_A I would expect to see 6 rows in Monitor->Pipeline Runs, with each parameter being Table1 to Table6.  However, this is not the case as even though I do see 6 rows in the monitor, the parameter preview for all 6 lines is always the same table name eg. Table2.  Running the trigger again will have the same result although it will be a different table eg. Table5.  

    The desired solution is to run Pipeline_B for all 6 tables in parallel and I thought this type of orchestration would work.  


    Does anybody know why this is happening, is it a known issue with ADF?

    Thanks



    • Edited by AvPowerBI Tuesday, December 3, 2019 4:15 PM
    Tuesday, December 3, 2019 4:07 PM

All replies

  • Hello AvPowerBI and thank you for your question.

    You mentioned that inside your For-Each activity, there is a Set Variable activity.

    When Sequential is not ticked, all or many of the iterations happen at the same time.  While these happen in parallel, there is only one instance of the variable.  This means all those instances try to write at the same time.

    The next activity inside the For-Each reads that variable.  All those parallel instances try to read from the same instance of the variable.  Whichever instance wrote to the variable last, is the value that will be read by each of the instances.

    This is called a race condition.

    To avoid this, there are two options:

    • Check the sequential box and disable parallel execution
    • Remove the SetVariable activity, and insert the expression used, into your second activity (execute pipeline).
    Thursday, December 5, 2019 4:31 AM
    Moderator