none
ADF V2 - Copy does not work for containers with a large number of blobs

    Question

  • I'm having a problem with Azure Data Factory V2, where containers with a large number of blobs are not copied. One pipeline run went almost 75 hours before it finally failed with an error. 

    Activity Copy1 failed: ErrorCode=FailedStorageOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A storage operation failed with the following error 'Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.'.,Source=,''Type=Microsoft.WindowsAzure.Storage.StorageException,Message=Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.,Source=Microsoft.WindowsAzure.Storage,''Type=System.IO.IOException,Message=Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.,Source=System,''Type=System.Net.Sockets.SocketException,Message=An existing connection was forcibly closed by the remote host,Source=System,'

    Additional Details

    • 5 Pipelines each with 1 copy activity (3 succeed, 2 fail). Using auto for parallel copy and DMUs, skip incompatible rows, no staging. 
    • Source data set uses a SAS key for an Azure Storage Account. The storage account is configured for RA-GRS, and I am attempting to read from the secondary (read access) location. Using the recursive and binary file options. 
    • Destination data set uses a SAS key for an Azure Storage Account. The storage account is located in the same data center as the Source data set (storage account's RA secondary). US South Central.

    I have tried 5 different source containers, 3 were successful while 2 failed. The commonality between the 2 that failed seems to be the number of blobs in the containers. One container has over 30 million blobs in the root. I don't know the number in the other container, but it's over 1 TB made up of small files (15 KB each) organized into sub folders 2 levels deep. I've tried to reproduce the folder structure the best I could. 

    • Container 1 (Success): {Guid-Folder-Name}/Blob.jpg (65GB, 63,555 files copied)
    • Container 2 (Success): Folder/guid-file-name (209GB, 2,724,023 files copied)
    • Container 3 (Success): Folder/{Folder}/blob.txt (filtered to *.txt, 606MB, 687,559 files copied)
    • Container 4 (Failed): Folder/{IntId-FolderName}/blob (filtered to *.json, > 62,500,000 total files, filtered to *.json would be 10% of that). 
    • Container 5 (failed): Folder/guid-file-name (> 30 million blobs). 

    With container 4, I tried a source with a filter, and without and neither worked. I then changed the source to be a more specific path (Folder/1234) that had ~100,000 blobs, and it copied that just fine with a filter specified. Since I have had a successful copy using both filtered and unfiltered sources, and with varying path structures (containers 1-3), it seems that the problem is the number of blobs. 


    • Edited by AdamStr Tuesday, May 29, 2018 7:32 PM
    Tuesday, May 29, 2018 7:26 PM

All replies

  • Hi AdamStr,

    Thank you for providing the concrete information.  Could you please help to provide the runID for your failed pipeline run?

    And it will be great if you can leave your email address so that we can reach out to you for further communication.

    Sunday, June 10, 2018 7:10 AM
  • f01a4881-4f3c-4215-a191-0dd14e1a83ba

    b18aab83-70b5-4c3b-a89a-faf26502a1da

    adam . salvo AT trainerroad . com
    • Edited by AdamStr Wednesday, June 13, 2018 10:38 PM
    Wednesday, June 13, 2018 10:37 PM
  • Hi Adam,

    Per your provided runIds, I found that the exception is thrown when listing the number of files. Recently we had an enhancement on listing a number of blob files. Could you re-run your copy to see whether it could succeed now?

    Thursday, June 21, 2018 12:33 AM