locked
Azure Data Factory slow between tasks RRS feed

  • Question

  • I've built a simple flow in ADF. 

    The tasks is working great and does what they should but ADF seems really really slow between tasks. After one task is finished it can pass between 5 seconds and in some cases up over a minute before the next one starts. I'm sure I've done something wrong here but I don't understad what it could be. 

    The tasks makes up for little more than one minute.. but the pipeline takes 5:27 to finish. 
    Unfortunately I'm not allowed to upload pictures but I hope my problem is clear enough :) 

    What could be the reason for this?

    EDIT:


    • Edited by Lard W Tuesday, October 1, 2019 6:20 AM
    Monday, September 30, 2019 10:53 AM

All replies

  • Did you trigger the pipeline or click debug to run it? Where did you check 5:27 duration ? How did you identify the run time gap in between tasks?

    I am not sure if you are talking about below case. When we debug the pipeline, the first tasks seem to be executed quickly and later execution doesn't move on quickly, we need to refresh it so that they reflect quickly. This is a refresh issue. 


    You can request to get your account verified by commenting on below thread and then do upload the screenshot of the issue.

    https://social.msdn.microsoft.com/Forums/en-US/dc4002e4-e3de-4b1e-9a97-3702387886cc/verify-account-42?forum=reportabug


    Cheers
    Vaibhav
    MCSA (SQL Server 2014)

    Monday, September 30, 2019 12:25 PM
  • Hi, 

    I triggered the pipeline for this. Havent checked with debug, but will do that later to find out if I will get another result.

    There is a big difference in time when looking at the total duration for the pipeline and the individual tasks added time. I have x tasks that in total adds up to about 1:15 minutes but the total pipeline duration is 5:27 minutes (I get this from the 'Pipeline runs' page).

    I've been sitting refreshing manually in the activity run and noticed that it takes a lot of time to start up the following task. 

    I'll try the verifying link asap and upload my pictures here.

    Thanks.

    Monday, September 30, 2019 1:00 PM

  • There is a big difference in time when looking at the total duration for the pipeline and the individual tasks added time. I have x tasks that in total adds up to about 1:15 minutes but the total pipeline duration is 5:27 minutes (I get this from the 'Pipeline runs' page).

    I tried to run one pipeline using trigger option. It has multiple pipelines within and could see just couple of seconds gaps which seems fine. 

    If the tasks are not dependent on each other, you can let them run in parallel by removing pipes that chain them together.


    Cheers
    Vaibhav
    MCSA (SQL Server 2014)

    Monday, September 30, 2019 1:13 PM
  • I tried to run one pipeline using trigger option. It has multiple pipelines within and could see just couple of seconds gaps which seems fine. 

    If the tasks are not dependent on each other, you can let them run in parallel by removing pipes that chain them together.

    Ok,

    Well my thing is a one minute pipeline that goes on for over 5 minutes. So it doesn't seem fine.

    I was able to upload my pictures now. 

    And thanks for your input.

    BR

    Lars


    Tuesday, October 1, 2019 6:25 AM
  • Hi Lard W,

    Just checking to see if you are still noticing the delay between chained activities? 


    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Thursday, October 10, 2019 11:50 PM
  • Hi Lard W,

    Following up to see if you are still noticing the delay between chained activities? 

    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Friday, October 11, 2019 6:26 PM
  • Hi, 

    Generally everything is faster now, så this isn't a problem for me anymore. But there's still time seemingly lost between tasks. I'm sure something happens there, but it would be fun to know what that something is and if I can do something to influence it.

    BR

    Lars

    Monday, October 14, 2019 8:20 AM
  • Hi Lard W,

    Sorry for the late response. I tried reproducing your issue but no success. If you notice a similar behavior in future, please share the pipeline run ID  and activity ID here, so that I can followup with the internal team to determine the reason behind it.


    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Wednesday, October 30, 2019 9:57 PM