locked
ADF v2 copy activity long queue time RRS feed

  • Question

  • I have create a pipeline for copy data to DB from on-premise file (Azure VM), and finally move the file to backup folder. I have create a GetMetadata activity and CopyData activity in the pipeline, sometimes the GetMetadata activity or CopyData activity will take a long time for process (About 10 minuses but file size only 150 KB). This is happen ocasionally, please help if have any solutions, many thanks.

    Friday, July 12, 2019 8:20 AM

All replies

  • Friday, July 12, 2019 8:42 AM
  • 10 minutes is the queue time, the transformation time is only take 00:00:00, how come the queue time is take a long time? thanks.
    Friday, July 12, 2019 9:21 AM
  • Hi ChrisLau3636,

    Just checking to see if you are still looking for assistance on this issue? If so, could you please provide a screen print of your ADF pipeline. 


    [If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster. ]

    Monday, July 15, 2019 6:45 PM
  • Tuesday, July 16, 2019 9:24 AM
  • Hi ChrisLau3636,

    Just checking to see if the above suggestion from Fang Liu helped to resolve your issue? If you still need assistance please let us know.

    If this helped to resolve your issue, please click 'Vote as helpful' and/or 'Mark as answer', so that it will be helpful for other members of the community.

    Thank you.


    [If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster. ]

    Wednesday, July 17, 2019 4:51 PM
  • The IR version is 3.18.7118.1, I think it is the latest one.
    Thursday, August 1, 2019 7:00 AM
  • It is the pipeline, the error sometimes occurred in the highlighted steps (Copy file to backup folder, delete file). Those files are in on-premise share folder (Azure VM)

    Thursday, August 1, 2019 7:10 AM
  • Hi ChrisLau3636,

    If there are more jobs running on self-hosted IR at the same time, it will put down the session on IR pending time.

    Things to be considered about IR performance:

    • When the available memory and CPU are not utilized well, but the execution of concurrent jobs is reaching the limit, you should scale up by increasing the number of concurrent jobs that can run on a node.
    • You might also want to scale up when activities are timing out because the self-hosted IR is overloaded. As shown in the following image, you can increase the maximum capacity for a node.


    Please try increasing the "Limit Concurrent jobs" in IR configuration and do let us know how it goes. 


    Additional info

    Also please refer to the below doc which has the info about high availability and scalability of IR which might be useful.


    [If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster. ]

    Thursday, August 1, 2019 9:23 PM
  • I have increase the concurrent job limit to 20 but it not works....

    Friday, August 2, 2019 2:52 AM
  • Hi Chris, 

    For deeper analysis on the issue, if you have a support plan you may file a support ticket, else could you please send an email to AzCommunity@Microsoft.com with the below details, so that we can enable a one-time free technical support for your subscription to work closely on this matter.  

    Thread URL: <https://social.msdn.microsoft.com/Forums/en-US/3ab944b5-94ee-4949-ad1f-e1f16377c75f/adf-v2-copy-activity-long-queue-time?forum=AzureDataFactory>
    Subscription ID:  <your subscription id>

    Please use the title of this thread as subject in the email.

    Let us know once you have sent the email.

    Thank you.


    [If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster. ]

    Friday, August 2, 2019 6:34 PM
  • Hi ChrisLau3636,

    Just wanted to check if you have got a chance to send an email as requested in above reply?

    Also before sending the email could you please have another try installing the latest IR (Version: 3.19.7144.1) which released on 8/1/2019.

    ADF IR Download link: https://www.microsoft.com/en-us/download/details.aspx?id=39717


    If you still face the same issue, please do send an email as requested in the previous reply.


    [If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster. ]

    Monday, August 5, 2019 6:38 PM
  • Hi ChrisLau3636,

    Just checking to see if your issue was resolved? If you still need assistance do let us know.


    [If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster. ]

    Tuesday, August 6, 2019 10:43 PM
  • HI Kranthi,

    Yes same exact on my end. The Get Metadata activity running just says queued for over 12 minutes. There is just one file in the data lake directory. Is there a resolution to this ?

    Thanks Ash


    • Edited by Ash2319 Tuesday, June 9, 2020 8:56 AM
    Tuesday, June 9, 2020 8:55 AM