none
azure data lake gen2 rest api token with maxResults and continuation RRS feed

  • Question

  • I follow this doc(https://docs.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/list), try to list all the files in ADLS gen2.

    Everything works fine when I create the signature token without maxResults and continuation. But if I specify one of them or both of them in the token, an error occurs like ""AuthenticationFailed","message":"Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the
    signature.
    "

    can you please let me know how to fix this issue in python or powershell or c#? I tried both in powershell and python3.7, same issue.

    Here is the code of powershell(the code works if remove maxResults and continuation):

    $StorageAccountName = "xxx"
    $FilesystemName="dd1"
    $AccessKey="xxx"
    $file_dir="f1"
    $maxResults=5
    $continuation_token=''
    $date = [System.DateTime]::UtcNow.ToString("R")
     
    $n = "`n"
    $method = "GET"
    $stringToSign = "$method$n" #VERB
    $stringToSign += "$n" # Content-Encoding + "\n" + 
    $stringToSign += "$n" # Content-Language + "\n" + 
    $stringToSign += "$n" # Content-Length + "\n" + 
    $stringToSign += "$n" # Content-MD5 + "\n" + 
    $stringToSign += "$n" # Content-Type + "\n" + 
    $stringToSign += "$n" # Date + "\n" + 
    $stringToSign += "$n" # If-Modified-Since + "\n" + 
    $stringToSign += "$n" # If-Match + "\n" + 
    $stringToSign += "$n" # If-None-Match + "\n" + 
    $stringToSign += "$n" # If-Unmodified-Since + "\n" + 
    $stringToSign += "$n" # Range + "\n" +
    $stringToSign +=   
                        <# SECTION: CanonicalizedHeaders + "\n" #>
                        "x-ms-date:$date" + $n +
                        "x-ms-version:2018-11-09" + $n #
                        <# SECTION: CanonicalizedHeaders + "\n" #>
     
    $stringToSign +=   
                        <# SECTION: CanonicalizedResource + "\n" #>
                        "/$StorageAccountName/$FilesystemName" + $n +
                        "directory:"+$file_dir + $n +
                        "maxResults:5" + $n +
                        "recursive:true" + $n +
                        "continuation:"+$continuation_token+$n+
                       
                        "resource:filesystem"
                        #
                        <# SECTION: CanonicalizedResource + "\n" #>
     
    $sharedKey = [System.Convert]::FromBase64String($AccessKey)
    $hasher = New-Object System.Security.Cryptography.HMACSHA256
    $hasher.Key = $sharedKey
     
    $signedSignature = [System.Convert]::ToBase64String($hasher.ComputeHash([System.Text.Encoding]::UTF8.GetBytes($stringToSign)))
     
     
    $authHeader = "SharedKey ${StorageAccountName}:$signedSignature"
     
    $headers = @{"x-ms-date"=$date}
    $headers.Add("x-ms-version","2018-11-09")
    $headers.Add("Authorization",$authHeader)
     
    $URI = "https://$StorageAccountName.dfs.core.windows.net/" + $FilesystemName + "?directory="+$file_dir+"&maxResults=5&recursive=true&continuation="+$continuation_token+"&resource=filesystem"
     write-output $URI
    $result = Invoke-RestMethod -method $method -Uri $URI -Headers $headers
    foreach($r in $result.paths)
    {
    $r.name
    }


    • Edited by ime11 Thursday, May 23, 2019 10:24 AM
    Thursday, May 23, 2019 10:23 AM

All replies

  • Hello ime11 and thank you for your inquiry.  There are several possible causes for your distress.

    Let me start gathering resources.

    Authentication documentation.

    A github thread with multiple causes and solutions.

    A StackOverflow thread regarding issues in String to SignA similar MSDN thread

    Thursday, May 23, 2019 10:56 PM
    Moderator
  • Thanks for your response. I have already read these docs. And yes it works, but when I specify maxResults or continuation in the token, it will raise a 403 error.
    Friday, May 24, 2019 2:32 AM
  • I just tried using both maxResults and the continuation token together, utilizing Postman.
    That succeeded.  However, as I just had a sessions working with someone else on an ADLS gen2 issue, where we used the same tools same steps, and got different results, I do not give my success much credit.

    On Tuesday I will confer with my colleagues about this and related issues.
    Saturday, May 25, 2019 1:07 AM
    Moderator
  • Could you share with me what region your account is in, and whether it has any non-alphanumeric characters?
    Tuesday, May 28, 2019 5:15 PM
    Moderator
  • Make sure to URL encode the continuation token first, e.g.

    HttpUtility.UrlEncode(continuationToken)

    I published a helper package that does a lot of this for you: https://www.nuget.org/packages/Adlg2Helper/, e.g.

    var pathClient = Adlg2ClientFactory.BuildPathClient("account name", "shared key");

    var paths = pathClient.List("container name", recursive: true);

    Monday, August 5, 2019 11:59 PM