none
How to download all the blobs in a Azure storage container(block blob) from Azure Linux virtual machine ?

    Question

  •  I have around 400000 blob files (Block) in a azure storage container. Size approx (500 GB ).

    how to copy all these files into Azure Linux virtual machine ?

    I have tried the below options :

    1. Azure-cli : i'm able to download single blob at a time, but not all the blobs in the container.

    2. Blobxfer : getting the attribute error , not able to resolve it.

    Could you please suggest the possible solution.


    Thanks & Regards, Manjunath Chekuri

    Friday, September 18, 2015 4:10 AM

Answers

  • Hi Manjunath,

    Thanks for posting here.

    Method 1:

    You may use the AzCopy tool to copy data from the blob storage wherein your files have been uploaded to that VHD(second hard drive) which you have connected to VM.

    Else, if the imported contents is in VHD(blob) then you can attach the same disk as Data disk to your existing VM.

    Please refer following link for more info regarding Copy files to/from Azure Storage:

    http://blogs.msdn.com/b/mast/archive/2014/06/28/how-to-copy-files-to-from-azure-storage.aspx

    Method 2:

    you can do that using Powershell to list all containers and AzCopy to copy all blobs in a container.

    Follow these steps:

    1) Install AzCopy (http://aka.ms/downloadazcopypr)

    2) Open Powershell and login in your Azure subscription

    3) Run this script putting the right storage account names and making sure that the AzCopy path is C:\Program Files (x86)\Microsoft SDKs\Azure\AzCopy\AzCopy.exe (otherwise change it in the script)

    $SourceStorageAccountName = "<SourceStorageAccountName>"
    $DestStorageAccountName = "<DestStorageAccountName>"
    $SourceStorageKey = Get-AzureStorageKey -StorageAccountName $SourceStorageAccountName
    $DestStorageKey = Get-AzureStorageKey -StorageAccountName $DestStorageAccountName
    $StorageContext = New-AzureStorageContext -StorageAccountName $SourceStorageKey.StorageAccountName -StorageAccountKey $SourceStorageKey.Primary
    $containers = Get-AzureStorageContainer -Context $StorageContext
    
    foreach ($c in $containers) {
    	"Transfer container " + $c.Name
    	$cmd = "C:\'Program Files (x86)'\'Microsoft SDKs'\Azure\AzCopy\AzCopy.exe /Source:" + $c.CloudBlobContainer.Uri.AbsoluteUri + " /Dest:https://" + $DestStorageAccountName + ".blob.core.windows.net/" + $c.Name + " /SourceKey:" + $SourceStorageKey.Primary + " /DestKey:" + $DestStorageKey.Primary + " /S"
    
    	Invoke-Expression $cmd
    }

    Hope this helps

    Girish Prajwal

    Friday, September 18, 2015 12:49 PM
    Moderator

All replies

  • Hi Manjunath,

    Thanks for posting here.

    Method 1:

    You may use the AzCopy tool to copy data from the blob storage wherein your files have been uploaded to that VHD(second hard drive) which you have connected to VM.

    Else, if the imported contents is in VHD(blob) then you can attach the same disk as Data disk to your existing VM.

    Please refer following link for more info regarding Copy files to/from Azure Storage:

    http://blogs.msdn.com/b/mast/archive/2014/06/28/how-to-copy-files-to-from-azure-storage.aspx

    Method 2:

    you can do that using Powershell to list all containers and AzCopy to copy all blobs in a container.

    Follow these steps:

    1) Install AzCopy (http://aka.ms/downloadazcopypr)

    2) Open Powershell and login in your Azure subscription

    3) Run this script putting the right storage account names and making sure that the AzCopy path is C:\Program Files (x86)\Microsoft SDKs\Azure\AzCopy\AzCopy.exe (otherwise change it in the script)

    $SourceStorageAccountName = "<SourceStorageAccountName>"
    $DestStorageAccountName = "<DestStorageAccountName>"
    $SourceStorageKey = Get-AzureStorageKey -StorageAccountName $SourceStorageAccountName
    $DestStorageKey = Get-AzureStorageKey -StorageAccountName $DestStorageAccountName
    $StorageContext = New-AzureStorageContext -StorageAccountName $SourceStorageKey.StorageAccountName -StorageAccountKey $SourceStorageKey.Primary
    $containers = Get-AzureStorageContainer -Context $StorageContext
    
    foreach ($c in $containers) {
    	"Transfer container " + $c.Name
    	$cmd = "C:\'Program Files (x86)'\'Microsoft SDKs'\Azure\AzCopy\AzCopy.exe /Source:" + $c.CloudBlobContainer.Uri.AbsoluteUri + " /Dest:https://" + $DestStorageAccountName + ".blob.core.windows.net/" + $c.Name + " /SourceKey:" + $SourceStorageKey.Primary + " /DestKey:" + $DestStorageKey.Primary + " /S"
    
    	Invoke-Expression $cmd
    }

    Hope this helps

    Girish Prajwal

    Friday, September 18, 2015 12:49 PM
    Moderator
  • You can use

    $ wget -I url_list.txt

    where the url_list.txt is the list of files you want to download.  You can get the url of each of the files from the azure portal or create the list using powershell.

    see http://xmodulo.com/how-to-download-multiple-files-with-wget.html for more information and more options using wget.


    Dan Stolts (ITProGuru)

    Monday, September 21, 2015 7:46 PM
  • Here is the step by step guide to download Azure blob storage content in Azure VM - http://sanganakauthority.blogspot.in/2017/03/how-to-download-azure-blob-storage.html

    Hope this helps.


    http://sanganakauthority.blogspot.com/

    Tuesday, March 7, 2017 12:20 PM