locked
Backup SQL Server 2012 database to Azure Storage: Exception Message: ParallelUpload Failed with timeout = 00:03:00 RRS feed

  • Question

  • I am trying to backup a SQL Server 2012 (SP1 CU4) database to an Azure Storage container but am not having much luck.

    I continue to receive the error below

    Exception Message: ParallelUpload Failed with timeout = 00:03:00

    I read somewhere that maybe there was an open lease on the blob but I tried with a new name and still receive the same error.

    Friday, July 12, 2013 4:59 AM

Answers

  • The purpose to use external interface is testing whether your internet bandwidth can handle big upload or not.

    And I also testing using my replication database (1.04GB, compressed=53MB), upload to Azure with and without COMPRESSION option. And it works flawlessly.

    According to this article http://technet.microsoft.com/en-us/library/jj919149.aspx,

    Backup and restore performance may vary depending on the network bandwidth, size of the database, and the location of the Windows Azure storage service relative to the on-premises location. To evaluate the backup and restore performance for your environment, we recommend that you test backup and restores and measure throughput and performance. The following items should be considered when backing up to Windows Azure Blob storage service:

    • Backup and Restore times are proportional to the network bandwidth.

    • The minimum network bandwidth for backup is 1 MB per second. Backup may fail with time-out errors, if the bandwidth is lower than 1 MB per second.

    • Restore time is also proportional to network latency, especially if you are trying to restore a file from a Windows Azure region geographically different from the instance of SQL Server that you are trying to restore to. In such cases, testing becomes very important to make sure that you can meet your RTO requirements.


    As alternative, you can also use .NET or Powershell as shown in this article http://msdn.microsoft.com/en-us/library/jj919148.aspx

    Regards,
    Christian HL
    Microsoft Online Community Support


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.


    • Edited by Christian HL - MSFT Friday, July 19, 2013 3:01 AM add information
    • Marked as answer by Dino He Friday, July 19, 2013 6:21 AM
    Friday, July 19, 2013 3:00 AM

All replies

  • Hi,

    I guess the file is big, so you need to split it into chunks:

    http://wely-lau.net/2012/02/26/uploading-big-files-in-windows-azure-blob-storage-with-putlistblock/

    Thanks,


    QinDian Tang
    MSDN Community Support | Feedback to us
    Develop and promote your apps in Windows Store
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.

    • Marked as answer by Dino He Friday, July 19, 2013 6:21 AM
    • Unmarked as answer by CI-2013 Friday, July 19, 2013 6:33 AM
    Friday, July 12, 2013 7:26 AM
  • Maybe, but I really do not have any control over how the file is uploaded. A local backup of my database only consumes 75MB while the maximum size of an Azure blob is listed as 1TB.

    I am following this article msdn dot microsoft dot com /en-us/library/454c8296-64e9-46ed-b141-5ebfbc8a4fe2. Per the article's example here is the exact form of T-SQL command being issued, using my own information for the database name, URL and credential of course:

    BACKUP DATABASE [AdventureWorks2012]

    TO URL = 'https://mystorageaccount.blob.core.windows.net/privatecontainertest/AdventureWorks2012.bak'

    WITH CREDENTIAL = 'mycredential';


    Friday, July 12, 2013 11:42 AM
  • Hi,

    Try to connect to Azure storage from SSMS directly:

    http://social.msdn.microsoft.com/Forums/windowsazure/en-US/5bbf55ee-7db2-4566-b611-3f8179460837/connecting-to-azure-storage-from-ssms

    http://msdn.microsoft.com/en-us/library/jj919148.aspx

    Thanks,


    QinDian Tang
    MSDN Community Support | Feedback to us
    Develop and promote your apps in Windows Store
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.

    Monday, July 15, 2013 5:18 AM
  • Thanks for the reply, but what will exporting a Data-tier Application to Azure Storage prove? I am trying to backup a database and the issue is not connecting, it's with the backup actually completing. As the backup is taking place I can see that a new backup is being created in my storage container via the Azure web interface. Something is happening during the backup process that is causing the failure and there does not seem to be much I can do about it.
    Monday, July 15, 2013 12:08 PM
  • Hi,

    I think the error is still caused by the size of the file which is larger than 64mb. But you say the backup is done. It is really strange, but lucky enough. You can try to use some storage tool to upload the file to storage directly.

    Thanks,


    QinDian Tang
    MSDN Community Support | Feedback to us
    Develop and promote your apps in Windows Store
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.

    Tuesday, July 16, 2013 2:23 AM
  • The backup completes locally, not to Azure Storage. I am saying that during the backup if I refresh the Azure Storage web interface I can see a new blob was created in my container. However as soon as the backup fails the blob disappears, i.e. the backup to Azure is not successfully completed.

    If the max size of a blob is 64 MB then Azure Storage would be of very little use for backing up SQL Server databases. I seriously doubt that is a real limitation. 



    • Edited by CI-2013 Tuesday, July 16, 2013 3:15 AM ps 64MB
    Tuesday, July 16, 2013 3:10 AM
  • Hi,

    I am not sure if it works, but you can try to add the timeout to sql server:http://msdn.microsoft.com/en-us/library/ms189040.aspx

    http://blogs.msdn.com/b/windowsazure/archive/2013/01/24/sql-server-backup-and-restore-to-cloud-simplified.aspx

    Thanks,


    QinDian Tang
    MSDN Community Support | Feedback to us
    Develop and promote your apps in Windows Store
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.


    Tuesday, July 16, 2013 6:45 AM
  • I do not think remote query timeout is in scope for this action. The setting supports queries using Linked Servers. Further, the setting is currently 10 minutes and my backups timeout at 3 minutes.

    I looked into the Application Log and found entries from source SQLBackupToUrl but they didn't lead me directly to a solution. When I run a backup to Azure I see these three entries in sequence:

    1. BackupToUrl initiated
    2. BackupToUrl Client is getting configuration from SqlServr
    3. ParallelUpload Failed with timeout = 00:03:00

    I looked for configuration items mentioned in item 2 above in sys.configurations but there were no good candidates.

    Could you possibly get this issue in front of a SQL Server Backup Team member or an Azure Storage Team member?

    Tuesday, July 16, 2013 5:17 PM
  • Hi,
     
    I am trying to involve someone familiar with this topic to further look at this issue. There might be some time delay. Appreciate your patience.
     
    Thanks,

    QinDian Tang
    MSDN Community Support | Feedback to us
    Develop and promote your apps in Windows Store
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.

    Wednesday, July 17, 2013 2:31 AM
  • Hi,

    Try to upgrade to CU5: http://support.microsoft.com/kb/2861107.

    And use WITH COMPRESSION option.

    - http://technet.microsoft.com/en-us/library/jj919149.aspx

    - http://msdn.microsoft.com/en-us/library/jj919148.aspx


    Regards,
    Christian HL
    Microsoft Online Community Support


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

    Wednesday, July 17, 2013 8:05 AM
  • Thanks for the reply Christian HL.

    I upgraded the instance with CU5 (Build 11.0.3373) yet we receive the same error:

    Msg 3202, Level 16, State 1, Line 1
    Write on "http://x.blob.core.windows.net/sqlbackups/db.bak" failed: Backup to URL received an exception from the remote endpoint. Exception Message: ParallelUpload Failed with timeout = 00:03:00
    Msg 3271, Level 16, State 1, Line 1
    A nonrecoverable I/O error occurred on file "http://x.blob.core.windows.net/sqlbackups/db.bak:" Error could not be gathered from Remote Endpoint.
    Msg 3013, Level 16, State 1, Line 1
    BACKUP DATABASE is terminating abnormally.

    Wednesday, July 17, 2013 1:58 PM
  • Its quite strange.

    I try with AdventureWorks 2012 DB (189MB) and upload to Azure successfully.

    Can you try with AdventureWorks 2012 DB too (http://msftdbprodsamples.codeplex.com/releases/view/55330). Also try using WITH COMPRESSION option.

    Other thing that you can try is upload manually the db backup to Azure as PageBlob to see if you get same problem or not. (you can use any free tool, this is one of it: http://www.cerebrata.com/labs/azure-explorer


    Regards,
    Christian HL
    Microsoft Online Community Support


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

    Thursday, July 18, 2013 2:13 AM
  • I tried backing up a different database and it worked. However the backup file was < 1 MB.

    The database that fails to backup is 1.2GB on disk (one 600MB data and log file each). When backed up with compression the file is only 75 MB.

    The backup operation gets to 30% within a few seconds and then stalls, and eventually it times out before making it to 40%:

    10 percent processed.
    20 percent processed.
    30 percent processed.
    Msg 3202, Level 16, State 1, Line 1
    Write on "http://x.blob.core.windows.net/sqlbackups/db.bak" failed: Backup to URL received an exception from the remote endpoint. Exception Message: ParallelUpload Failed with timeout = 00:03:00
    Msg 3271, Level 16, State 1, Line 1
    A nonrecoverable I/O error occurred on file "http://x.blob.core.windows.net/sqlbackups/db.bak:" Error could not be gathered from Remote Endpoint.
    Msg 3013, Level 16, State 1, Line 1
    BACKUP DATABASE is terminating abnormally.

    Ay help you can provide would be great. I would like to use the native backup commands to get data to Azure, as opposed to using a external interface like cerebrata, and I should be able to. Please advise if there is anything we can do to troubleshoot the issue, else we're going to have to look for another solution.

    Thursday, July 18, 2013 6:20 PM
  • The purpose to use external interface is testing whether your internet bandwidth can handle big upload or not.

    And I also testing using my replication database (1.04GB, compressed=53MB), upload to Azure with and without COMPRESSION option. And it works flawlessly.

    According to this article http://technet.microsoft.com/en-us/library/jj919149.aspx,

    Backup and restore performance may vary depending on the network bandwidth, size of the database, and the location of the Windows Azure storage service relative to the on-premises location. To evaluate the backup and restore performance for your environment, we recommend that you test backup and restores and measure throughput and performance. The following items should be considered when backing up to Windows Azure Blob storage service:

    • Backup and Restore times are proportional to the network bandwidth.

    • The minimum network bandwidth for backup is 1 MB per second. Backup may fail with time-out errors, if the bandwidth is lower than 1 MB per second.

    • Restore time is also proportional to network latency, especially if you are trying to restore a file from a Windows Azure region geographically different from the instance of SQL Server that you are trying to restore to. In such cases, testing becomes very important to make sure that you can meet your RTO requirements.


    As alternative, you can also use .NET or Powershell as shown in this article http://msdn.microsoft.com/en-us/library/jj919148.aspx

    Regards,
    Christian HL
    Microsoft Online Community Support


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.


    • Edited by Christian HL - MSFT Friday, July 19, 2013 3:01 AM add information
    • Marked as answer by Dino He Friday, July 19, 2013 6:21 AM
    Friday, July 19, 2013 3:00 AM
  • I think you may be correct about the lack of line speed being the issue. I tested the line speed using speedtest.net at the location in question and found the upload speed topped out at only ~400 Kbps.

    From another network location with ~5 Mbps upload capabilities I tested backing up AdventureWorks2012 to the same Azure Storage Container and it worked fine to backup a 45MB backup file in 1m 30s, however that was with compression on. With compression off the backup would be 194 MB and that fails to backup to Azure Storage.

    The production database backup in question, with compression on, requires 75 MB and will continue growing. I think I am going to have to go in another direction. I understand the need for line speed, but 1 MB per second equates to an actual 8 Mbps line and that's not always going to be available in this environment.

    Thanks for your assistance.

    Friday, July 19, 2013 2:14 PM
  • Hi,

    I'm from Brazil and I have the same problem that you once had. I would like to know which direction did you take to resolve the problem in question. 

    Thank you,

    Felipe Santiago

    Monday, September 2, 2013 6:21 PM
  • We are not using Azure Storage due to the high bandwidth requirements. It simply proved to be too high of a bar for us to meet.

    We are actually using SkyDrive at the moment as our space requirements are low.

    Tuesday, September 3, 2013 3:00 AM