none
How to access rest azure blob using cURL

    Question

  • Hi,

    The below script doesn't work

    #!/bin/bash
    
    # List the blobs in an Azure storage container.
    
    echo "usage: ${0##*/} <storage-account-name> <container-name> <access-key>"
    
    storage_account="$1"
    container_name="$2"
    access_key="$3"
    
    blob_store_url="blob.core.windows.net"
    authorization="SharedKey"
    
    request_method="GET"
    request_date=$(TZ=GMT date "+%a, %d %h %Y %H:%M:%S %Z")
    storage_service_version="2015-04-05"
    
    # HTTP Request headers
    x_ms_date_h="x-ms-date:$request_date"
    x_ms_version_h="x-ms-version:$storage_service_version"
    
    # Build the signature string
    canonicalized_headers="${x_ms_date_h}\n${x_ms_version_h}"
    canonicalized_resource="/${storage_account}/${container_name}"
    
    string_to_sign="${request_method}\n\n\n\n\n\n\n\n\n\n\n\n${canonicalized_headers}\n${canonicalized_resource}\ncomp:list\nrestype:container"
    
    # Decode the Base64 encoded access key, convert to Hex.
    decoded_hex_key="$(echo -n $access_key | base64 -d -w0 | xxd -p -c256)"
    
    # Create the HMAC signature for the Authorization header
    signature=$(printf "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" -binary |  base64 -w0)
    
    authorization_header="Authorization: $authorization $storage_account:$signature"
    
    curl \
      -H "$x_ms_date_h" \
      -H "$x_ms_version_h" \
      -H "$authorization_header" \
      "https://${storage_account}.${blob_store_url}/${container_name}?restype=container&comp=list"

    response:

    <?xml version="1.0" encoding="utf-8"?>
    <Error>
        <Code>AuthenticationFailed</Code>
        <Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
    RequestId:94b9c231-0001-001f-7a0c-9715fe000000
    Time:2016-04-15T11:49:27.8560792Z</Message>
        <AuthenticationErrorDetail>The Date header in the request is incorrect.</AuthenticationErrorDetail>
    </Error>

    Friday, April 15, 2016 11:53 AM

Answers

  • @Ashok Thank you for quick answer. I read documentation again and discovered that above script doesn't work because of locale of my OS. In order to fix it is enough to set locale to en_US. Below is complete working script:

    #!/bin/bash
    
    # List the blobs in an Azure storage container.
    
    echo "usage: ${0##*/} <storage-account-name> <container-name> <access-key>"
    
    storage_account="$1"
    container_name="$2"
    access_key="$3"
    
    blob_store_url="blob.core.windows.net"
    authorization="SharedKey"
    
    request_method="GET"
    request_date=$(TZ=GMT LC_ALL=en_US.utf8 date "+%a, %d %h %Y %H:%M:%S %Z")
    #request_date="Mon, 18 Apr 2016 05:16:09 GMT"
    storage_service_version="2015-04-05"
    
    # HTTP Request headers
    x_ms_date_h="x-ms-date:$request_date"
    x_ms_version_h="x-ms-version:$storage_service_version"
    
    # Build the signature string
    canonicalized_headers="${x_ms_date_h}\n${x_ms_version_h}"
    canonicalized_resource="/${storage_account}/${container_name}"
    
    string_to_sign="${request_method}\n\n\n\n\n\n\n\n\n\n\n\n${canonicalized_headers}\n${canonicalized_resource}\ncomp:list\nrestype:container"
    
    
    # Decode the Base64 encoded access key, convert to Hex.
    decoded_hex_key="$(echo -n $access_key | base64 -d -w0 | xxd -p -c256)"
    
    
    # Create the HMAC signature for the Authorization header
    signature=$(printf "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" -binary |  base64 -w0)
    
    authorization_header="Authorization: $authorization $storage_account:$signature"
    
    curl \
      -H "$x_ms_date_h" \
      -H "$x_ms_version_h" \
      -H "$authorization_header" \
      -H "Content-Length: 0"\
      "https://${storage_account}.${blob_store_url}/${container_name}?comp=list&restype=container"

    • Marked as answer by PavtJ Monday, April 18, 2016 5:39 AM
    Monday, April 18, 2016 5:38 AM

All replies

  • Hello,

    Thank you for posting here!

    Please check this thread which is having similar issue might help you to solve your error.

    Also this link, please check and let us know if you have any other queries.

    Regards,

    Ashok

    Saturday, April 16, 2016 11:49 AM
    Moderator
  • @Ashok Thank you for quick answer. I read documentation again and discovered that above script doesn't work because of locale of my OS. In order to fix it is enough to set locale to en_US. Below is complete working script:

    #!/bin/bash
    
    # List the blobs in an Azure storage container.
    
    echo "usage: ${0##*/} <storage-account-name> <container-name> <access-key>"
    
    storage_account="$1"
    container_name="$2"
    access_key="$3"
    
    blob_store_url="blob.core.windows.net"
    authorization="SharedKey"
    
    request_method="GET"
    request_date=$(TZ=GMT LC_ALL=en_US.utf8 date "+%a, %d %h %Y %H:%M:%S %Z")
    #request_date="Mon, 18 Apr 2016 05:16:09 GMT"
    storage_service_version="2015-04-05"
    
    # HTTP Request headers
    x_ms_date_h="x-ms-date:$request_date"
    x_ms_version_h="x-ms-version:$storage_service_version"
    
    # Build the signature string
    canonicalized_headers="${x_ms_date_h}\n${x_ms_version_h}"
    canonicalized_resource="/${storage_account}/${container_name}"
    
    string_to_sign="${request_method}\n\n\n\n\n\n\n\n\n\n\n\n${canonicalized_headers}\n${canonicalized_resource}\ncomp:list\nrestype:container"
    
    
    # Decode the Base64 encoded access key, convert to Hex.
    decoded_hex_key="$(echo -n $access_key | base64 -d -w0 | xxd -p -c256)"
    
    
    # Create the HMAC signature for the Authorization header
    signature=$(printf "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" -binary |  base64 -w0)
    
    authorization_header="Authorization: $authorization $storage_account:$signature"
    
    curl \
      -H "$x_ms_date_h" \
      -H "$x_ms_version_h" \
      -H "$authorization_header" \
      -H "Content-Length: 0"\
      "https://${storage_account}.${blob_store_url}/${container_name}?comp=list&restype=container"

    • Marked as answer by PavtJ Monday, April 18, 2016 5:39 AM
    Monday, April 18, 2016 5:38 AM
  • Thanks for posting the answer after you figured it out. I'm sure that will help others.

    Robin


    Click here to visit my ClickOnce blog!
    Microsoft MVP, Client App Dev

    Monday, April 18, 2016 10:00 PM
  • Modified it to query blob property. 


    #!/bin/bash

    # List the blobs in an Azure storage container.

    echo "usage: ${0##*/} <storage-account-name> <container-name> <access-key> <blob>"

    storage_account="$1"
    container_name="$2"
    access_key="$3"
    blob_name="$4"

    blob_store_url="blob.core.windows.net"
    authorization="SharedKey"

    request_method="HEAD"
    request_date=$(TZ=GMT LC_ALL=en_US.utf8 date "+%a, %d %h %Y %H:%M:%S %Z")
    #request_date="Mon, 18 Apr 2016 05:16:09 GMT"
    storage_service_version="2015-04-05"

    # HTTP Request headers
    x_ms_date_h="x-ms-date:$request_date"
    x_ms_version_h="x-ms-version:$storage_service_version"

    # Build the signature string
    canonicalized_headers="${x_ms_date_h}\n${x_ms_version_h}"
    canonicalized_resource="/${storage_account}/${container_name}/${blob_name}"

    string_to_sign="${request_method}\n\n\n\n\n\n\n\n\n\n\n\n${canonicalized_headers}\n${canonicalized_resource}"


    # Decode the Base64 encoded access key, convert to Hex.
    decoded_hex_key="$(echo -n $access_key | base64 -d -w0 | xxd -p -c256)"


    # Create the HMAC signature for the Authorization header
    signature=$(printf "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" -binary |  base64 -w0)

    authorization_header="Authorization: $authorization $storage_account:$signature"

    curl -I \
      -H "$x_ms_date_h" \
      -H "$x_ms_version_h" \
      -H "$authorization_header" \
      -H "Content-Length: 0"\
      "http://${storage_account}.${blob_store_url}/${container_name}/${blob_name}"

    Friday, November 9, 2018 4:30 AM
  • I need similar script to work from Windows command prompt and CURL. 

    Can someone please share the script ? 

    thanks 

    Amol


    thanks -amol

    Friday, January 4, 2019 3:16 AM
  • @Amol_paranjpe: Can you provide us more information what exactly are you trying to perform?
    Friday, January 4, 2019 4:45 AM
    Moderator
  • @SumanthMarigowda : 

    I need to upload a file using CURL and REST API to the blob storage. The script mentioned above is for Linux shell and what I am looking for is windows command prompt based script to achieve same. 

    The information such as Service Endpoint, Container, Storage Account,Key  is already with me. I don't know how to use this and CURL to access the storage and perform actions such as GET/PUT etc. using curl. 

    Hope I am clear on the requirement. 

    Thanks 

    Amol 


    thanks -amol

    Friday, January 4, 2019 1:21 PM
  • Sorry for the delay response

    The best approach would be to use the native PowerShell Cmdlets.   If you have a good case for using Invoke-WebRequest instead a few suggestions:

    Generating the derived authorization header (SharedKey) from the storage account key may be bit difficult in this environment.   It’s much easier to use SAS tokens for this and append it to the query string of the REST request.  

    If you have a relative small file (that you can load into memory and do with a single PutBlob call)

    $file = Get-Content .\file.png -Raw
    $uri = "https://javaller.blob.core.windows.net/pstest/file.png?sv=2018-03-28&ss=bfqt&srt=sco&sp=rwdlacup&se=2019-01-26T04:59:20Z&st=2019-01-10T20:59:20Z&spr=https&sig=XXXXX"
    $headers = @{}
    $headers.Add("x-ms-blob-type","BlockBlob")
    Invoke-WebRequest -uri $uri -Method Put -Body $file -ContentType "image/png" -Headers $headers
    

    Note:  $URI line removed from screenshot to protect my storage account

    I wouldn’t recommend this approach for anything over ~4MiB in size.

    Additional information: REST with PowerShell.  It appears that cURL translates to Invoke-WebRequest in PowerShell.

    Kindly let us know if the above helps or you need further assistance on this issue.


    Friday, January 11, 2019 6:02 AM
    Moderator
  • Adding more info to the above query: Creating SAS tokens, then curl really is about the same functionality as Invoke‑WebRequest for doing the upload.

    e.g., to upload to ptest/file.png… If user knows ahead of time that the blob will be called file.png – generate a Blob SAS signature for that specific name.   If they need to accommodate a variety of blob names, they can generate for the container level and also edit the filename.

    You can use curl as:

    uri='https://x.blob.core.windows.net/ptest/file.png?sr=c&  . . .' 

    cat file.png | curl -X PUT -H 'x-ms-blob-type: BlockBlob' -m 300 -H 'Content-Type: image/png'-d @- $uri

    to pipe the file in and upload.   -m is a 300-second request timeout (https takes time), and Content-type is optional .

    Notes:

    1. On the efficiency side, adding the two curl options to disable the 100-Continue processing (‘-H Expect:’ and ‘--expect100-timeout 0’) would yield only a trivial benefit as Azure Storage responds with the 100 quickly.   Leaving those options out has the added gain of a fast-fail (before data payload is sent) in the case of a >256MB payload or a bad URI.
    2. Despite the PutBlob doc page not being explicit about defaults, there’s no longer a need to send the Storage API version explicitly (e.g., -H 'x-ms-version: 2016-05-31', or higher) to allow data sizes in the 64MB to 256MB range.   2016-05-31 is currently the default assumed by Storage (at least for this API use).
    3. If you try to send a >256MB (well, really MiB – 1024-based) payload, the request fails with a 413 error.
    4. ‘—data-binary’ is nearly always what you want – curl can trip on EOF characters and stop early with just ‘-d’.
    5. curl can also read the file directly via  “—data-binary @filename” instead of “—data-binary @-“.
    6. Depending on the balance between network capacity and processing power for the local workstation, it may also be beneficial to include the curl ‘–tr-encoding’  option to have curl gzip the payload before transmission.

    Hope this helps!

    ------------------------------------------------------------------------------------------

    Do click on "Mark as Answer" on the post that helps you, this can be beneficial to other community members.


    Monday, January 14, 2019 6:39 AM
    Moderator