locked
azcopy destination parameter RRS feed

  • Question

  • Hi I am new to hdinsight; I created storage myhdinsightstore and a cluster  stackoverflowanalysis   ; then tried to run command

    from azcopy

    azcopy     c:\data.csv       https:/............   I am confused about the destination parameter this is the url when I open my cluster

    https://manage.windowsazure.com/@kokoboshhotmail.onmicrosoft.com

    #Workspaces/HDInsightExtension/HDInsightExtension/StackOverflowAnalysis/quickstart  

    I do not know which part is  blob /container or whatever......

    also what is  the key part which is supplied at the end of destination parameter  it must be something i get from azure and save in a text file......... we are slaves of microsoft believe me nothing is intuitive or logic

    Saturday, September 6, 2014 11:36 PM

Answers

  • Lookup your Access key. In manage.windowsazure.com, click Storage on the left, select the storage account you chose when you created the Hadoop cluster, then click Manage Access Keys at the bottom:

    Either key is fine. Copy the storage account name and one of the 2 keys:

    Next, click on the storage account, click the Containers tab, and confirm the name of the Hadoop store container (myhdinsightstore by default).

    In Powershell ISE, run:

    $azPath = "C:\Program Files (x86)\Microsoft SDKs\Azure\AzCopy"
    Set-Location $azPath
    
    $StorageAccountName = "samscorner1" 
    $StorageAccountKey = "Nk48Iljv0AHBUIqaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaDjgBDKWyEkRwbd+BTqew=="
    $ContainerName = "myhdinsightstore"
    
    $SourceFolder = "D:\Sandbox\test"
    $DestURL = "https://$StorageAccountName.blob.core.windows.net/$ContainerName"
    $Result = .\AzCopy.exe $SourceFolder $DestURL /BlobType:block /destkey:$StorageAccountKey /Y
    $Result

    All files under $SourceFolder will be copied over to the 'myhdinsightstore' container. 

    Output may look like:

    Transfer summary:
    -----------------
    Total files transferred: 165
    Transfer successfully:   165
    Transfer failed:         0
    Elapsed time:            00.00:02:03

    To view the files run:

    $Context = New-AzureStorageContext –StorageAccountName $StorageAccountName -StorageAccountKey $StorageAccountKey 
    $Files = Get-AzureStorageBlob -Container $ContainerName -Context $Context 
    $Files

    based on the same variables defined above.

    Note:

    • This will time-out every 15 minutes. The good news is that azCopy can resume the upload. 
    • Calculate the expected trasfer time based on the files' size and your egress bandwidth. For example, if you're copying files that total 5GB, and you have 10 Mbps Internet pipe, expect it to take a little over 1 hour. Use some calculator like this one.
    • Run this script to continue the upload:
    [int]$Failed = $Result[5].Split(":")[1].Trim()
    $Journal  = "$env:LocalAppData\Microsoft\Azure\AzCopy"
    $i=1
    while ($Failed -gt 0) {
        $i++
        [int]$Failed = $Result[5].Split(":")[1].Trim()
        $Result = .\AzCopy.exe /Z:$Journal
        $Result
        $i # 15 minutes each
    }



    Sam Boutros, Senior Consultant, Software Logic, KOP, PA http://superwidgets.wordpress.com (Please take a moment to Vote as Helpful and/or Mark as Answer, where applicable)


    Sunday, September 7, 2014 2:04 AM
  • Yep, that won't work. The azcopy.exe tool syntax is:

    AzCopy <source> <destination> [filepattern] [options]

    Look at the examples I posted above, or try:

    azcopy  c:\azure\queryresults.csv   https://kobosh.blob.core.windows.net/stockoverflowanalysis /detskey:<copy-and-paste-your-key-here-as-shown-above>

    See this link for more details.


    Sam Boutros, Senior Consultant, Software Logic, KOP, PA http://superwidgets.wordpress.com (Please take a moment to Vote as Helpful and/or Mark as Answer, where applicable)

    Monday, September 8, 2014 8:59 AM
  • Hi Kobosh,

    About this issue, firstly we can see this command:

    AzCopy C:\blob-data https://myaccount.blob.core.windows.net/mycontainer/ /destkey:key /S

    1."C:\blob-data": this is your file url which will be uploaded.

    2.https://myaccount.blob.core.windows.net/mycontainer/: this is your storage account url.

    3."destkey:key": this is your account key. You could try code like this:

    AzCopy C:\blob-data https://mystorage.blob.core.windows.net/vhds /destkey:***+**== /S

    Also, you could try this docs:http://blogs.msdn.com/b/windowsazurestorage/archive/2012/12/03/azcopy-uploading-downloading-files-for-windows-azure-blobs.aspx

    Please try it.

    Regards,

    Will


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Thursday, September 11, 2014 12:28 PM

All replies

  • Lookup your Access key. In manage.windowsazure.com, click Storage on the left, select the storage account you chose when you created the Hadoop cluster, then click Manage Access Keys at the bottom:

    Either key is fine. Copy the storage account name and one of the 2 keys:

    Next, click on the storage account, click the Containers tab, and confirm the name of the Hadoop store container (myhdinsightstore by default).

    In Powershell ISE, run:

    $azPath = "C:\Program Files (x86)\Microsoft SDKs\Azure\AzCopy"
    Set-Location $azPath
    
    $StorageAccountName = "samscorner1" 
    $StorageAccountKey = "Nk48Iljv0AHBUIqaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaDjgBDKWyEkRwbd+BTqew=="
    $ContainerName = "myhdinsightstore"
    
    $SourceFolder = "D:\Sandbox\test"
    $DestURL = "https://$StorageAccountName.blob.core.windows.net/$ContainerName"
    $Result = .\AzCopy.exe $SourceFolder $DestURL /BlobType:block /destkey:$StorageAccountKey /Y
    $Result

    All files under $SourceFolder will be copied over to the 'myhdinsightstore' container. 

    Output may look like:

    Transfer summary:
    -----------------
    Total files transferred: 165
    Transfer successfully:   165
    Transfer failed:         0
    Elapsed time:            00.00:02:03

    To view the files run:

    $Context = New-AzureStorageContext –StorageAccountName $StorageAccountName -StorageAccountKey $StorageAccountKey 
    $Files = Get-AzureStorageBlob -Container $ContainerName -Context $Context 
    $Files

    based on the same variables defined above.

    Note:

    • This will time-out every 15 minutes. The good news is that azCopy can resume the upload. 
    • Calculate the expected trasfer time based on the files' size and your egress bandwidth. For example, if you're copying files that total 5GB, and you have 10 Mbps Internet pipe, expect it to take a little over 1 hour. Use some calculator like this one.
    • Run this script to continue the upload:
    [int]$Failed = $Result[5].Split(":")[1].Trim()
    $Journal  = "$env:LocalAppData\Microsoft\Azure\AzCopy"
    $i=1
    while ($Failed -gt 0) {
        $i++
        [int]$Failed = $Result[5].Split(":")[1].Trim()
        $Result = .\AzCopy.exe /Z:$Journal
        $Result
        $i # 15 minutes each
    }



    Sam Boutros, Senior Consultant, Software Logic, KOP, PA http://superwidgets.wordpress.com (Please take a moment to Vote as Helpful and/or Mark as Answer, where applicable)


    Sunday, September 7, 2014 2:04 AM
  • thanks Sam: I found key and saved it as azkey.txt on my pc.

    my storage name is the default   myhdinsightstore    and my cluster name is stackoverflowanalysis  so in azcopy how am i going to write the destination parameter? https://myhdinsightstore.blob.core.window.net/stackoverflowanalysis/DestKey:key    azkey.txt

    OR

    https://stackoverflowanalysis.blob.core.window.net/myhdinsightstore/DestKey:key    azkey.txt

    Sunday, September 7, 2014 5:46 PM
  • You need a working knowledge of Powershell. Here's a place to start.


    Sam Boutros, Senior Consultant, Software Logic, KOP, PA http://superwidgets.wordpress.com (Please take a moment to Vote as Helpful and/or Mark as Answer, where applicable)

    Sunday, September 7, 2014 5:49 PM
  • ok now I created new storage account 'kobosh' and  hdinsight hadoop cluster 'stockoverflownalysis'

    and ran   azcopy  c:\azure\queryresults.csv   http://kobosh.blob.core.windows.net/stockoverflowanalysis/detskey: key  azkey.txt

    I get error  command syntax is incorrect  specifying multiple files is not supported

    Monday, September 8, 2014 5:14 AM
  • Yep, that won't work. The azcopy.exe tool syntax is:

    AzCopy <source> <destination> [filepattern] [options]

    Look at the examples I posted above, or try:

    azcopy  c:\azure\queryresults.csv   https://kobosh.blob.core.windows.net/stockoverflowanalysis /detskey:<copy-and-paste-your-key-here-as-shown-above>

    See this link for more details.


    Sam Boutros, Senior Consultant, Software Logic, KOP, PA http://superwidgets.wordpress.com (Please take a moment to Vote as Helpful and/or Mark as Answer, where applicable)

    Monday, September 8, 2014 8:59 AM
  • I have tried copy paste key    with/without quotes

    azcopy c:\azure\queryresults.csv https://kobosh.blob.core.windows.net/stockoverflowanalysis

    /detskey: vhas123=^........

    and in double quotes /destkey:"vhas123=^........"


    I get error    command syntax incorrect specify one of the following for the account to which the  destination blob to be written

    Monday, September 8, 2014 2:40 PM
  • Hi Sam

    is there a more humane way of downloading and installing  azure powershell ise?

    I tried to download from technet but it is so complicated I could not understand anything of what they are trying to say? pleas please can you show me click click way to doing it.

    Monday, September 8, 2014 6:38 PM
  • You can try a GUI tool to copy files to Azure Blob storage like CloudBerry

    Sam Boutros, Senior Consultant, Software Logic, KOP, PA http://superwidgets.wordpress.com (Please take a moment to Vote as Helpful and/or Mark as Answer, where applicable) _________________________________________________________________________________ Powershell: Learn it before it's an emergency http://technet.microsoft.com/en-us/scriptcenter/powershell.aspx http://technet.microsoft.com/en-us/scriptcenter/dd793612.aspx

    Monday, September 8, 2014 6:42 PM
  • I tried  powershell

    still I get the same error

    Windows PowerShell
    Copyright (C) 2013 Microsoft Corporation. All rights reserved.

    For a list of all Azure cmdlets type 'help azure'.
    For a list of Windows Azure Pack cmdlets type 'Get-Command *wapack*'.
    For Node.js cmdlets type 'help node-dev'.
    For PHP cmdlets type 'help php-dev'.
    For Python cmdlets type 'help python-dev'.
    PS C:\> $azPath="c:\Program Files (x86)\Microsoft SDKs\Azure\AzCopy"
    PS C:\> set-location $azPath
    set-location : Cannot find path 'C:\Program Files (x86)\Microsoft SDKs\Azure\AzCopy' because it does not exist.
    At line:1 char:1
    + set-location $azPath
    + ~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : ObjectNotFound: (C:\Program File...Ks\Azure\AzCopy:String) [Set-Location], ItemNotFoundE
       xception
        + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.SetLocationCommand

    PS C:\> $azPath="C:\Program Files\Microsoft SDKs\Azure\AzCopy"
    PS C:\> set-location $azPath
    PS C:\Program Files\Microsoft SDKs\Azure\AzCopy> $strgaccnt="kobosh"
    PS C:\Program Files\Microsoft SDKs\Azure\AzCopy> $strgkey="qn2Uoy2XigtnYgZm7yKwXf+degCjRnZr9DHokDBx3NMYuuULU8nS7KY2rmbgr
    bUA4W+Df2MIoLeR/x6Is4SpqQ=="
    PS C:\Program Files\Microsoft SDKs\Azure\AzCopy> $contname="StackOverflowAnalysis"
    PS C:\Program Files\Microsoft SDKs\Azure\AzCopy> $srcfldr="Downloads\Queryresults.csv"
    PS C:\Program Files\Microsoft SDKs\Azure\AzCopy> $dest="https://$strgaccnt.blob.core.windows.net/$contname"
    PS C:\Program Files\Microsoft SDKs\Azure\AzCopy> $Result=.\AzCopy.exe $srcfldr $dest/BlobType:block/destkey:$strgkey/Y
    The syntax of the command is incorrect. Specify one of the following parameters for the account to which the destination
     blob is to be written: "/DestKey" or "/DestSAS". Alternately, you can include the SAS token as a query string in the de
    stination URI.
    PS C:\Program Files\Microsoft SDKs\Azure\AzCopy> 

    Monday, September 8, 2014 9:19 PM
  • Hi Kobosh,

    About this issue, firstly we can see this command:

    AzCopy C:\blob-data https://myaccount.blob.core.windows.net/mycontainer/ /destkey:key /S

    1."C:\blob-data": this is your file url which will be uploaded.

    2.https://myaccount.blob.core.windows.net/mycontainer/: this is your storage account url.

    3."destkey:key": this is your account key. You could try code like this:

    AzCopy C:\blob-data https://mystorage.blob.core.windows.net/vhds /destkey:***+**== /S

    Also, you could try this docs:http://blogs.msdn.com/b/windowsazurestorage/archive/2012/12/03/azcopy-uploading-downloading-files-for-windows-azure-blobs.aspx

    Please try it.

    Regards,

    Will


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Thursday, September 11, 2014 12:28 PM