locked
AzureStorageOperationFailedConcurrentWrite error while iterating in For each loop in Copy Activity in Azure Data Factory RRS feed

  • Question

  • Hi All,

    While working with For Each Loop in Azure Data Factory, I have 2 copy activities (one to pull data from REST API source to BLOB and the subsequent one to pull from BLOB and store in SQL Server). When I run the task it throws the below error, is there a way i can handle this scenario, I have set the Max Concurrent Connections to 1 in both the SINK of Copy Activity task even then I get this error message. Any help is highly appreciated.

    Failure happened on 'Sink' side. ErrorCode=AzureStorageOperationFailedConcurrentWrite,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error occurred when trying to upload a file. It's possible because you have multiple concurrent copy activities runs writing to the same file 'xxxx_yyyy.csv'. Check your ADF configuration.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.WindowsAzure.Storage.StorageException,Message=The remote server returned an error: (400) Bad Request.,Source=Microsoft.WindowsAzure.Storage,StorageExtendedMessage=The specified block list is invalid.\nRequestId:e095aec8-b01e-002d-44af-fe040a000000\nTime:2020-03-20T12:05:48.0407417Z,,''Type=System.Net.WebException,Message=The remote server returned an error: (400) Bad Request.,Source=Microsoft.WindowsAzure.Storage
    Friday, March 20, 2020 12:42 PM

Answers

  • Assuming that it's a concurrent execution of copy activities that are writing to same file. To avoid this, please try ticking Sequential checkbox property of ForEach activity.

    If the response helped, do "Mark as answer" and upvote it
    - Vaibhav

    • Marked as answer by CoolDbGuy Friday, March 20, 2020 4:27 PM
    Friday, March 20, 2020 2:03 PM