none
Performance of Start-AzureStorageFileCopy

    Question

  • I'm using Start-AzureStorageFileCopy to copy all of the files and folders from one Azure Storage File Service to another.  

    In order to do so, we are issuing Start-AzureStorageFileCopy from PowerShell for each file.  With more than 250K files (122 GB), this is taking a significant amount of time.  We let it run for three hours the other day without much of the data moved.  Since the cmdlet takes about 1 second to start each file move, we are estimating it would take about 70 hours to move the files.

    We've determined this isn't a data volume issue.  With about 122 GB in data across a couple dozen files, using the same PowerShell commands took less than 10 seconds to complete.

    When we used AZCopy to move the same set of files up to Azure, it only took an hour to complete the move.

    Is there another way to copy all of the files in a Azure File Share to another Azure File Share that can complete in a reasonable timeframe?  We just need a full copy of the share to lock in a current state of the files.


    Jason Strate | http://jasonstrate.com | http://www.twitter.com/stratesql

    Tuesday, November 1, 2016 7:40 PM

All replies

  • Hi Jason,

    Have you tried using AZcopy to copy the files across the 2 file shares? If so, do you also see slow performance?

    AzCopy /Source:https://myaccount1.file.core.windows.net/myfileshare1/ /Dest:https://myaccount2.file.core.windows.net/myfileshare2/ /SourceKey:key1 /DestKey:key2 /S

    Wissam Elriachy

    Tuesday, November 1, 2016 8:18 PM
  • I'm running a test with this now to get exact timing.  Based on the first 5K files, this will be faster than using PowerShell but appears it will still take about 20 hours to complete.

    I'm really surprised there isn't a method available to create a snapshot of a file storage location.


    Jason Strate | http://jasonstrate.com | http://www.twitter.com/stratesql

    Tuesday, November 1, 2016 8:45 PM
  • Hi Wissam-

    I finished a copy of the files from Azure File Share to Azure File Share using AZCopy, this method took 9 hours.  This is about 7 hours longer than it takes to push the data from our local environment up to Azure.

    Is there really no method to clone or snapshot a folder in Azure File Share?


    Jason Strate | http://jasonstrate.com | http://www.twitter.com/stratesql

    Wednesday, November 2, 2016 2:20 PM
  • Hi Jason,

    If you have a VM in your account, you can mount both file shares and use something like 

    robocopy z: y: /E /R:1 /W:1

    In order to mount a file storage to a windows machine, use:

    net use y: \\storageaccountname.file.core.windows.net\share /u:storageaccountname storageaccesskey

    To be honest, I don't know if this is faster, but it's worth a try.

    Wednesday, November 2, 2016 3:47 PM
  • Hi,

    Thank you for posting here!

    Apart from AZCopy you can also leverage Azure CLI to copy contents between two Azure file shares. Although I am not certain if it would perform faster than AZCopy but it's worth a shot. More on Azure CLI can be found in the link given below.

    https://azure.microsoft.com/en-in/documentation/articles/storage-azure-cli/

     

    Regards.

    Md. Shihab

     

    Please remember to click "Mark as Answer" on the post that helps you as this can be beneficial to other community members reading the thread. And vote as helpful.

    Wednesday, November 2, 2016 6:23 PM
  • Hi Jason,

    You are correct. Snapshot support is not there yet. It is however one of the top features in our backlog.

    Wissam


    Wissam Elriachy

    Friday, November 4, 2016 1:16 AM