none
Fastest way to copy large amounts of data to an azure VM (in a script)

    Question

  • Hi,

    As part of a script to provision my azure VMs i need to transfer a large amount of data to my Azure VM (lots of files which add up to a couple of GB)

    I am currently using an azure file storage container and mapping it as a network mapped drive in the VM. This is working but it can be fairly slow, especially because i have to copy them twice (once to get it into the storage account and then again to get it onto the VM - in my case i need it on the VM itself not just in network storage).

    I am wondering if there is a better/faster way?

    I have a couple ideas (but havn't tried any of them yet):

    - setting up a VPN between the original location and the VM?

    - making a shared folder on the VM and copy directly to that?

    the advantage to what i'm doing at the moment is that i can copy it to the storage and then if i make multiple VMs they can all get it from the one storage share. however if there is a way i can get it to the VM which is dramatically faster than that is definitely preferred.

    Edit: I should mention that it is currently around 40min to do the full (double) copy operation
    • Edited by MattFenner Thursday, June 23, 2016 4:50 AM
    Thursday, June 23, 2016 4:46 AM

Answers

All replies

  • Hi,

    Thank you for posting here!

    AzCopy is a Windows command-line utility designed for high-performance copying of data to and from Azure Storage. See Transfer data with the AzCopy Command-Line Utility to learn more.

    Regards,

    Vikranth S.

     

    Please remember to click "Mark as Answer" on the post that helps you, this can be beneficial to other community members reading the thread. And Vote as Helpful 

    Thursday, June 23, 2016 1:51 PM
    Moderator
  • I have played with that, and it is definitely an option. I'm not sure how i would call it from a powershell script, since it seems to have a specialized environment.

    Also it still means i would have the double copy problem.

    Would this be significantly faster than doing it using mapped drives?

    • Proposed as answer by Adeel-A Thursday, January 3, 2019 6:37 PM
    • Unproposed as answer by Adeel-A Thursday, January 3, 2019 6:37 PM
    Thursday, June 23, 2016 11:44 PM
  • I went looking for a fast copy solution and came across this post.  I've tried both Azure Storage Explorer and AzCopy.

    We have a VPN tunnel to Azure and I was trying to download a 1.6TB (allocated, not used) VHD file from a standard storage account.  With Azure Storage Explorer I was getting an average speed 240 Mbps.  Copying the same file via AzCopy was averaging 6.4 MB/s, which is about 51 Mbps.

    Thursday, January 3, 2019 6:44 PM
  • Hi Matt,

    you might be able to improve the speed of your current process by creating a service endpoint between the Azure Storage account and the subnet where you VM resides (for the second part of the copy).

    Alternatively, rather than copying data, you might want to consider uploading a disk containing it as a page blob and attaching it to the VM. Details would depend on whether you are using managed or unmanaged disks.

    hth
    Marcin

    Thursday, January 3, 2019 7:16 PM
  • Download and install latest version of AzCopy and check for the status.

    For more details please take a look at "Microsoft Azure Storage Performance and Scalability Checklist"

    Additional information: Refer here if there is any additional parameter issue.  

    Kindly let us know if the above helps or you need further assistance on this issue.

    Friday, January 4, 2019 5:57 AM
    Moderator