none
Error while calling ADLS Gen 2 Rest API to create file

    Question

  • I am trying to use the Rest API of Create path to create a file in ADLS Gen 2. The below are the parameters used for calling the API

    https://<Accountname>.dfs.core.windows.net/<FileFolder Name>/test1.zip?resource=file&<SAS Token>

    I have set the headers 

    content-length:0
    x-ms-version:2018-11-09

    Still this request continuously gives the error 

    error": {
            "code": "ContentLengthMustBeZero",
            "message": "The Content-Length request header must be zero"

    }

    I am stuck on this for 2 days. Any help is much appreciated

    I am able to call the List operations successfully

    Thursday, March 21, 2019 3:30 AM

All replies

  • Hi there,

    To use the REST APIs of ADLS Gen2, you need authorization header as well. Also the URL has SAS token directly(not as a parameter). 

    A simple way to use ADLS Gen2 APIs (Using OAuth2) to create a file would be :

    1. Acquire an Access token by making a POST request to https://login.microsoftonline.com/<tenant id>/oauth2/v2.0/token

    Headers :

    "Content-Type: application/x-www-form-urlencoded"

    Body : 

    {"client_id": <CLIENT_ID>,

    "client_secret": <CLIENT_SECRET>,

    "scope" : "https://storage.azure.com/.default",

    "grant_type" : "client_credentials"

    }

    2. Create a file system by making a PUT request to https://<storage account name>.dfs.core.windows.net/<file system name>?resource=filesystem

    Headers :

    Content-Length : 0

    "x-ms-version":"2018-11-09"

    Authorization : Bearer <access_token from step1>

    3. Set default permissions on the root directory by making a PATCH request to  https://<storage account name>.dfs.core.windows.net/<file system name>?action=setAccessControl

    Headers :

    Content-Length : 0

    "x-ms-version":"2018-11-09"

    Authorization : Bearer <access_token from step1>

    x-ms-acl : user::rwx,group::r-x,other::--x,default:user::rwx,default:group::r-x,default:other::--

    4. Create a directory by making a PUT request to https://<storage account name>.dfs.core.windows.net/<file system name>/<directory name>?resource=directory

    Headers :

    Content-Length : 0

    "x-ms-version":"2018-11-09"

    Authorization : Bearer <access_token from step1>

    5. Create a file by making a PUT request tohttps://<storage account name>.dfs.core.windows.net/<file system name>/<directory name>/<file name>?resource=file

    Headers :

    Content-Length : 0

    "x-ms-version":"2018-11-09"

    Authorization : Bearer <access_token from step1>

    Hope this helps.


    MSDN

    Thursday, March 21, 2019 12:01 PM
    Moderator
  • Hi Chirag,

    I tried the above approach( using the OAuth2 token) but i got the below error while hitting the api.

    "error": {
            "code": "InvalidAuthenticationInfo",
            "message": "Server failed to authenticate the request. Please refer to the information in the www-authenticate header.\nRequestId:b77c9de1-401f-010d-2767-e0b2e0000000\nTime:2019-03-22T04:25:19.8083293Z"
        }

    And the information included in the www-authenticate header is as below:

    Bearer authorization_uri=https://login.microsoftonline.com/<tenant-id>/oauth2/authorize resource_id=https://storage.azure.com

    The service principal used for generating access token has been added as Storage Blob Data Contributor. Also in AD App registration "Azure Storage" delegated permissions has been assigned.

    Please let me know if anything else needs to be done to make this work.

    Friday, March 22, 2019 4:42 AM
  • Hi Roshan,

    Since you have assigned the right access, can you please verify that you are generating the access token with the client_id, client_secret, scope and grant_type in the body?


    MSDN

    Friday, March 22, 2019 6:33 AM
    Moderator
  • Hi Chirag,

    Yes i am genrating the access token with the below body

    {"client_id": <CLIENT_ID>,

    "client_secret": <CLIENT_SECRET>,

    "scope" : "https://storage.azure.com/.default",

    "grant_type" : "client_credentials"

    }

    Please let me know if there any claims that i can verify with the generated access token in order to be sure that token is correct.

    Friday, March 22, 2019 7:15 AM
  • Hi Roshan,

    The error is for sure related to the access token. I tried out 3 cases :

    1. Without an authorization header.

    2. With an incorrect authorization header.

    3. With a correct authorization header.

    Please check your access token. Common mistakes could be :

    1. The access token has a leading space after that.

    2. Authorization header needs to have value : Bearer <access token> (note the space between bearer and access token).

    3. You are generating access token for a client_id that either is wrong or doesn't have right permissions.

    Please refer the below screenshots for reference :

    Right access token

    Wrong access token.


    MSDN

    Friday, March 22, 2019 11:21 AM
    Moderator
  • Here's a step by step guide to ensure you have the right permissions.

    Register with Azure Active Directory tenant

    1. Click on All services

    2. Filter on App Registration

    3. Click on New application registration

    4. Enter a name in the Name box

    5. Select Web app / API in the Application type select box

    6. Enter http://localhost in the Sign-on URL box

    7. Click the Create button to create the app registration

    Once the application is created, the portal displays the app overview. Copy the Application ID value and copy it to a text editor for later use. In an upcoming step, this value is used to define a `CLIENT_ID` environment variable.

    Generate a client secret

    The next step is to create a client secret. The client secret is used to request an access token. Once you created, you must copy the value to a text editor as the secret is hidden when you return to the list.

    1. Click on Settings

    2. Click on Keys

    3. Enter “rest demo” in the Description box

    4. Select In 1 year from the Expires box

    5. Click Save

    6. Copy the Value of the generated client secret into a text editor

    Add required permissions

    Now you need to grant permission for your application to access Azure Storage.

    1. Click on the application Settings

    2. Click on Required permissions

    3. Click on Add

    4. Click Select API

    5. Filter on Azure Storage

    6. Click on Azure Storage

    7. Click Select

    8. Click the checkbox next to Access Azure Storage

    9. Click Select

    10. Click Done

    This tutorial demonstrates how the REST API operates under different security contexts, therefore you need create a second registration by following the previous steps beginning at Register with Azure Active Directory tenant. Make sure to set aside the Application ID and client secret in a text editor and add the required permissions to Azure Storage. The Application ID and client secret are used to set up the environment variables in the next section.

    Add RBAC permissions

    Next, you set up RBAC permissions only for the first app registration*. During this section you use the Application ID that you set aside in your text editor for the first app registration. Return to your storage account in the portal.

    1. Click on Access control (IAM)

    2. Click Add

    3. Enter Storage Blob Data Contributor in the Role box

    4. Select Azure AD user, group, or application in the Assign access to box

    5. Enter the Application ID in the Select box and tab off the element

    6. Click on the name of your first app registration

    7. Click Save


    MSDN

    Friday, March 22, 2019 11:24 AM
    Moderator
  • Hi chirag,

    Thank you for the above information. I am able to generate Oauth token now successfully.

    I also used the same to create filesystem and file. But when i tried to do a update as mentioned in the below link.

    https://docs.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/update 

    I get 202 accepted as response, but the file is not getting updated.

    HTTP VERB:- PATCH

    URL:- https://<storageaccount>.dfs.core.windows.net/datalakefs2/test5.txt?action=append&position=0

    Headers:-

    Authorization:- Bearer <access_token>

    Content-Type:- text/plain

    Content-Length:-9

    Body:- test data

    Can you please let me know as to what i am doing wrong here?

    Thanks,

    Roshan

    

    Friday, March 22, 2019 12:44 PM
  • Hi Chirag,

    Can you please let me know if i am doing anything wrong with PATCH operation?

    It will be of huge help in completing the poc

    Thanks,

    Roshan

    Monday, March 25, 2019 8:27 AM
  • Hi Roshan,

    Once you do the PATCH operation, the text sits in an uncommitted buffer on the server. To endure data to the file system, flush the file using the following API :

    https://<storage account name>.dfs.core.windows.net/<filesystem name>/<folder>/<file>?action=flush&position=<no. of characters>

    Note : No. of characters is the same as the Content-Length in the previous (PATCH) step - 9 in your case.

    Headers :

    Content-Length : 0

    "x-ms-version":"2018-11-09"

    Authorization : Bearer <access_token from step1>

    Hope this helps.


    MSDN

    Tuesday, March 26, 2019 10:04 AM
    Moderator
  • Hi Roshan,

    Just wanted to check - was your issue resolved with the above suggestion?


    MSDN

    • Proposed as answer by RoshanCog Thursday, March 28, 2019 8:54 AM
    Thursday, March 28, 2019 6:31 AM
    Moderator
  • Hi Chirag,

    Yes, i was able to upload files using the above suggestion.

    Thanks for your help.

    Regards,

    Roshan

    Thursday, March 28, 2019 8:54 AM
  • Hi Roshan,

    Glad to know that the problem was solved. 

    Thanks,

    Chirag


    MSDN

    Friday, March 29, 2019 5:56 AM
    Moderator
  • Hello good afternoon.

    I'm starting with the api of azure data lake gen2 to upload files to my store, but I have a problem when doing "action=append" since my files are still empty.

    I have authenticated correctly, and I have also created the file with "resource=file", but at the time of doing PATCH I do not achieve any change.

    var client3 = new RestClient("https://xxxxxxxx.dfs.core.windows.net/xxxxxx/data/test?action=append&position=0");
    
                var request3 = new RestRequest(Method.PATCH);
                request3.AddHeader("Authorization", "Bearer " + data.access_token);
                request3.AddHeader("Content-Length", json.Length.ToString());
                request3.AddHeader("Content-Type", "application/json");
                request3.AddJsonBody(json);
    
                IRestResponse response3 = client3.Execute(request3);
                dynamic content3 = response3.Content;

    The answer does not give me any errors, but the file is not modified.

    I have also tried it with postman and the result is the same


    Thursday, May 2, 2019 4:45 PM
  • I am self-willed, since the solution was in the thread and it happens to me not to read it completely.
    Anyway, it seems a little bulky to upload a file first have to create it, then add it, and finally make flush.

    Greetings and thank you.
    Thursday, May 2, 2019 6:07 PM