locked
WEB (Azure Logic Apps) calls not running in sequance in Azure Data Factory RRS feed

  • Question

  • Hi 

    I have a ADF (Azure Data Factory) that looks like

    ALA = Azure Logic Apps

    1. Get List of file and folder from a Blob         (Activity: Get Meta Data)
    2. Filter the above list and get files *.Zip       (Activity: Filter )
    3. Loop zip files one by one                         (Activity: ForEach)
    4. Loop Start 
    5.          Unzip file using ALA                        (Activity: WEB) 
    6.          Copy Zip file to Archive using ALA    (Activity: WEB)
    7.          Delete Zip File using ALA                 (Activity: WEB)
    8. End Loop

    ALL steps are linked together using a GREEN Path

    Assuming I have 3 Zip files and each one takes 30s to unzip 5s to copy and 1s to delete

    so I will have to have 3 loops 

    loop1 = 30s + 5s + 1s

    Loop2 starts after 36s

    loop2 = 30s + 5s + 1s

    and ......

    BUT what I am getting in the first loop all 3 WEB call happen at the same time the green PATH does not wait for the previous step to finish before going to the next WEB, so in the first loop all 3 web get called in the first 1s then it moves to the second loop and that happens in 1s (again calling all 3 web al the same time) them moving to the 3rd loop

    My problem is that in the loop all 3 activity gets called at the same time and they ignore the green path

    they are not waiting for the foist WEB (unzip) to finish before moving to the next WEB (COPY) and so on

    Thanks for the helpNik


    Sincerely Nik -- Please kindly mark the post(s) that answered your question and/or vote for the post(s). http://sqldataside.blogspot.ca/ (SQL Tabular + PowerShell)

    Thursday, January 3, 2019 8:20 PM

All replies

  • Hi Nik,

    Have you tried out chaining the activities so that they are executed sequentially? From the "green PATH" that you mentioned, I am assuming you have chained the activities based on success of the previous activity. If not, please have a look at branching and chaining activities :

    https://docs.microsoft.com/en-us/azure/data-factory/tutorial-control-flow-portal

    Also, there's a similar issue reported here :

    https://stackoverflow.com/questions/52612382/azure-data-factory-select-property-status-succeeded-from-previous-activity

    From the answers, you can get an idea of how you can check the status of the previous activity.

    Let me know if this helps, or we can continue to probe in further.


    MSDN

    Monday, January 7, 2019 6:54 AM
  • Yes i have the green Path in between each activity

    This is still an issue

    if you like to test you can

    1- make a ALA1 that moves a file from one blob to another, in the same ALA create a delay of 20seconds (like a wait) in the first step then move the file , 

    2-  make another ALA2 that deletes the same file

    3- create a ADF and a pipeline that has the ALA1 then ALA2 (using a green path)

    4- fire off the ADF

    you will see that the ALA1 and ALA2 will fire off together

    ALA2 will delete the file

    ALA1 tries to move the file but because it is deleted the ALA1 will fail

    this is an issue that we have.

    Nik


    Sincerely Nik -- Please kindly mark the post(s) that answered your question and/or vote for the post(s). http://sqldataside.blogspot.ca/ (SQL Tabular + PowerShell)

    Tuesday, January 8, 2019 12:41 AM
  • Hi Chirag 

    I just checked the 2 links they do not have my scenario, they do nothave 2 WEB calls after each other, the samples are side by side

    Nik


    Sincerely Nik -- Please kindly mark the post(s) that answered your question and/or vote for the post(s). http://sqldataside.blogspot.ca/ (SQL Tabular + PowerShell)

    Tuesday, January 8, 2019 12:45 AM
  • Hi Nik,

    The way it works is - when the web activity is invoked, a post request is invoked which triggers your logic app and instantly returns a "202 Accepted" response. The activity in the Logic App (copying files in your case) is executed in the background. Since the response instantly comes back, the next web activity is invoked which triggers the next Logic App.

    Now in your case, there are two ways to go about :

    1. Have just one logic app to do everything - this will make sure each step happens after the previous one completes.

    2. To your logic apps, at the end, add a "Response" step. This will make sure that only after the execution of the step in the logic app, a response is sent. Here's an example of response step.

    Hope this helps.


    MSDN

    Wednesday, January 9, 2019 11:41 AM
  • Hi

    Sorry for my late reply, i am going to check this tomorrow

    i feel that this (answer number 2 ) might be the answer, but for your answer number 1 

    it might work for some scenarios but if you have one WEB API within a loop you will still have the same problem

    anyways thanks let me check

    Nik


    Sincerely Nik -- Please kindly mark the post(s) that answered your question and/or vote for the post(s). http://sqldataside.blogspot.ca/ (SQL Tabular + PowerShell)

    Tuesday, January 15, 2019 5:08 AM
  • Hi Nik,

    Just wanted to check - Was the issue resolved ?


    MSDN

    Wednesday, February 13, 2019 12:10 PM
  • sorry i will be checking it this week

    my bad


    Sincerely Nik -- Please kindly mark the post(s) that answered your question and/or vote for the post(s). http://sqldataside.blogspot.ca/ (SQL Tabular + PowerShell)

    Friday, February 15, 2019 5:15 PM
  • No problems Nik. Do let us know if your problem was resolved or if you need any further assistance.

    MSDN

    Monday, February 18, 2019 10:01 AM