locked
Can we copy Azure blob data from one storage account to other storage accounts in parallel from the same machine? RRS feed

  • Question

  • In Microsoft Azure, I have a source storage account in one region and 3 destination storage accounts in 3 different regions. I want to copy blob data from source storage account to all 3 destination storage accounts. Currently I am using the azcopy (version 6) command in a bash script to do it. First it completes for one region then starts for another. It takes almost an hour everyday due to the geographical distance between the regions. I wanted to know if azcopy has any option to copy blobs from source to multiple destinations in a parallel manner. Any other suggestions to reduce the time are also invited :)

    Generalization of azcopy command being used in my bash script:

    /usr/bin/azcopy --source https://[srcaccount].blob.core.windows.net/[container]/[path/to/blob] --source-key $SOURCE_KEY --destination https://[destaccount].blob.core.windows.net/[container]/[path/to/blob] --dest-key $DEST_KEY --recursive --quiet --exclude-older

    Monday, July 29, 2019 12:35 PM

All replies

  • I'd recommend using Azcopy V10 since  it should have better performance than v10, but it doesn't support the possibility of running multiple copies to multiple destinations. 

    You could however use multiple CMD windows to perform the task mentioned, this should allow the upload/data transfer at the same time. Let me know if you've tried this already.

    Thanks,

    Adam


    Monday, July 29, 2019 4:46 PM
  • @inderpal2406 Just checking in to see if the above answer helped. If this answers your query, do click “Mark as Answer” and Up-Vote for the same, which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.
    Wednesday, July 31, 2019 7:55 AM
  • Is there any update on the issue?

    If the suggested answer helped for your issue, do click on "Mark as Answer" and “Vote as Helpful” on the post that helps you, this can be beneficial to other community members.

    Tuesday, August 6, 2019 10:39 AM