locked
AzCopy Data Interity Check RRS feed

  • Question

  • Background:

    We’re planning to use AzCopy to copy all the on-prem files to Azure Storage (BLOB container and File Share). Through this process, we want to check the data integrity to show case to EPA that the data sent to Azure is an exact copy of the files on local server and that no corruption has occurred.

    Problem:

    However, from my research, it seems like AzCopy only provides MD5-hash when uploading to Azure. And the actual check does not happen until the uploaded blob is used (i.e. downloaded) by AzCopy.

    Questions:

    • In this case, how can we confirm the data integrity when uploading from On-prem to Azure storage account?
    • Is there any check we can perform locally before uploading and compare with the results after uploading?
    • Will files only be saved to the storage account if they pass some verification (MD5?) after file uploads are complete?

    Wednesday, April 15, 2020 4:31 AM

All replies

  • Hi Olivia111,

    Firstly, apologies for the delay in responding here and any inconvenience this issue may have caused. 

    • However, from my research, it seems like AzCopy only provides MD5-hash when uploading to Azure. And the actual check does not happen until the uploaded blob is used (i.e. downloaded) by AzCopy.

    Yes, to check the MD5 with azcopy, you'd need to download the file. If you don't want to store it, you could download to /dev/null, or the equivalent for Windows.

     

    • In this case, how can we confirm the data integrity when uploading from On-prem to Azure storage account?

    The MD5 is calculated as we upload, (and sent off to Azure, which trusts the md5 we send in). The above trick can be used to check, if concerned about data integrity.

     

    • Is there any check we can perform locally before uploading and compare with the results after uploading?

    No real checks you can do prior to uploading.

     

    • Will files only be saved to the storage account if they pass some verification (MD5?) after file uploads are complete?

    Files will be saved regardless of md5 validity. However, we do a lot of work and testing to ensure data integrity in AzCopy.

    Hope this helps! 

    Kindly let us know if the above helps or you need further assistance on this issue.
    ------------------------------------------------------------------------------------------

    Do click on "Mark as Answer" and Upvote on the post that helps you, this can be beneficial to other community members.
    Thursday, April 16, 2020 7:03 AM
  • @Olivia111 Just checking in to see if the above answer helped. If this answers your query, do click “Mark as Answer” and Up-Vote for the same, which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.
    Wednesday, April 22, 2020 5:11 AM
  • @Olivia111 Just checking in to see if the above answer helped. If this answers your query, do click “Mark as Answer” and Up-Vote for the same, which might be beneficial to other community members reading this thread. And, if you have any further query do let us know.
    Monday, April 27, 2020 3:46 PM
  • Is there any update on the issue?

    If the suggested answer helped for your issue, do click on "Mark as Answer" and “Vote as Helpful” on the post that helps you, this can be beneficial to other community members.

    Wednesday, May 6, 2020 3:46 PM