While working with For Each Loop in Azure Data Factory, I have 2 copy activities (one to pull data from REST API source to BLOB and the subsequent one to pull from BLOB and store in SQL Server). When I run the task it throws the below error, is there a way
i can handle this scenario, I have set the Max Concurrent Connections to 1 in both the SINK of Copy Activity task even then I get this error message. Any help is highly appreciated.
Failure happened on 'Sink' side. ErrorCode=AzureStorageOperationFailedConcurrentWrite,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error occurred when trying to upload a file. It's possible because you have multiple concurrent
copy activities runs writing to the same file 'xxxx_yyyy.csv'. Check your ADF configuration.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.WindowsAzure.Storage.StorageException,Message=The remote server returned an error: (400) Bad Request.,Source=Microsoft.WindowsAzure.Storage,StorageExtendedMessage=The
specified block list is invalid.\nRequestId:e095aec8-b01e-002d-44af-fe040a000000\nTime:2020-03-20T12:05:48.0407417Z,,''Type=System.Net.WebException,Message=The remote server returned an error: (400) Bad Request.,Source=Microsoft.WindowsAzure.Storage
Assuming that it's a concurrent execution of copy activities that are writing to same file. To avoid this, please try ticking Sequential checkbox property of ForEach activity.
If the response helped, do "Mark as answer" and upvote it
- Vaibhav
Marked as answer byCoolDbGuyFriday, March 20, 2020 4:27 PM