none
Timeouts when Uploading with Azcopy

    Question

  • From William Hilsum @wilhil via Twitter

    @AzureSupport Hi, trying to upload ~40GB via Azcopy on a crap machine with 5Mb upload. It times out EVERY 20 mins, is there an easier way?

    @AzureSupport Per my last tweet... I've read this - http://social.technet.microsoft.com/wiki/contents/articles/26528.resuming-timed-out-uploadsdownloads-tofrom-azure-blob-storage-with-azcopy.aspx- but couldn't get it to work :( 

    Asked if customer was seeing error codes: 

    @AzureSupport no error codes. I'll double check. Rerunning resumes fine, but it is annoying every 20 mins! Read others saying it is by design.

    Provided this documentation: aka.ms/d922609

    @AzureSupport Well, may work... but, requires a restart of the process as it is already a few days in :( Don't know if I should or not! 

    @AzureSupport with the /NC:2, it is going at a fraction of the speed -if it auto resumes,I don't mind as I lose time at night, but annoying. 

    @AzureSupport Damn. looks like it still fails - but, after an hour instead of 20 mins, so, at half the speed, it is pretty much just as bad 

    https://twitter.com/wilhil/status/692081659341053952

    Thanks,
    @AzureSupport

    Wednesday, January 27, 2016 10:22 PM

All replies

  • Hi,

    As far as I know if you are copying a file, the largest you can copy is the maximum BLOCK BLOB size of 195GB. I suggest you export your database dumps to multiple files so no single file is over 195GB.

    If the timeout issue keeps occurring in your machine, your network connection quality with global Azure might be awful. Please consider checking your proxy settings, creating the storage account in a region near China such as East Asia, or specifying a smaller /NC value in AzCopy command line (the default value is CPU core number * 8).

    Time-Out issue  might occur when you are copying large files. You could resume the transfer by running the same .\azcopy.exe command or you could use a script like the one below:

    [int]$Failed = $Result[5].Split(":")[1].Trim() $Journal = "$env:LocalAppData\Microsoft\Azure\AzCopy" $i=1 while ($Failed -gt 0) {     $i++     [int]$Failed = $Result[5].Split(":")[1].Trim()     $Result = .\AzCopy.exe /Z:$Journal     $Result     $i }

    Ref: http://blogs.msdn.com/b/windowsazurestorage/archive/2013/09/07/azcopy-transfer-data-with-re-startable-mode-and-sas-token.aspx

    For further information about AzCopy, please refer to http://aka.ms/azcopy .

    Hope this helps.

    Girish Prajwal

    Thursday, January 28, 2016 12:06 PM
    Moderator
  • Hi,

    I have written a very simple batch file to resume AzCopy when there are file transfer failed.

    You can save it as a bat file, and change the AzCopy Command as you like.

    Please let me know if it works for you.

    Thanks!

    :START
    AzCopy /source:[source] /dest:[dest] /pattern:[pattern] /DestKey[key] /y /s
    if %errorlevel% LSS 0 goto START

    Friday, February 5, 2016 2:24 AM
  • The batch script to automatically resume worked for me. Thanks!
    Monday, June 11, 2018 3:56 AM
  • @Trufus,

    Thanks for the confirmation. 

    Tuesday, June 12, 2018 6:24 PM
    Moderator