none
Error using WebHDFS with Data Lake v1 storage (Gen1 storage).

    Question


  • I'm having problems using WebHDFS-calls with a "general purpose v1 storage" (Gen1 storage).

    I have successfully generated an Access-token using the following curl-call

    curl -X POST -H "Content-Type: application/x-www-form-urlencoded" --data "client_id=<CLIENT_ID>" --data-urlencode "client_secret=<CLIENT_SECRET>" --data-urlencode "scope=https://storage.azure.com/.default" --data-urlencode "grant_type=client_credentials" https://login.microsoftonline.com/<TENANT_ID>/oauth2/v2.0/token

    But when trying to create a folder using curl as below i get an error "Authentication scheme Bearer is not supported".

    curl -i -X PUT -H "x-ms-version: 2016-05-31" -H "Content-Length: 0" -H "Authorization: Bearer <ACCESS-TOKEN>" "https://nvcontentstorage.file.core.windows.net/webhdfs/v1/mytempdir/?op=MKDIRS"

    I looked into previous questions about WebHDFS but the resources found where quite old and referencing "azuredatalakestore.net" which seems to have been replaced by "file.core.windows.net".

    What does the error "Authentication scheme Bearer is not supported" mean, and what would i need to do to make my WebHDFS-calls to work?

    Thursday, May 2, 2019 8:29 AM

All replies

  • I did tested this in Postman and you need to make two changes .

    1. For the token add the below in the body 

    resource ="https://management.core.windows.net/"

    2. While calling the PUT use this https://azureaccount.azuredatalakestore.net/webhdfs/v1/mydata?op=MKDIRS


    Thanks Himanshu

    Monday, May 6, 2019 8:39 PM
    Moderator
  • Thanks for your reply. 
    I tried some testing with Postman, but could not get it working.
    After that I tried to modify my second curl-call as you described, see below;

    curl -i -X PUT -H "x-ms-version: 2016-05-31" -H "Content-Length: 0" --data-urlencode "resource=https://management.core.windows.net/" -H "Authorization: Bearer <ACCESS-TOKEN>" "https://nvcontentstorage.azuredatalakestore.net/webhdfs/v1/mytempdir/?op=MKDIRS"

    As you can see i added the "resource-parameter" and changed the address to use "azuredatalakestore.net".
    But the response i get is "
    Could not resolve host: nvcontentstorage.azuredatalakestore.net".

    Do i need to activate "azuredatalakestore.net" in the Azure-portal?
    It does not seem to be a valid server-address?
    Could you please post Curl-samples showing how to do the Rest-calls?

    Tuesday, May 7, 2019 6:34 AM
  • Hello again.
    I have found out what i did wrong. I thought that the "Storage Account Gen1" was the storage i could access by WebHDFS. But I needed to create a "Data Lake Storage Gen1", and I did not know about that.

    I have now successfully used a WebHDFS-call to create a folder in my newly created "Data Lake Storage".
    Below are the 2 curl-calls that i used:

    1:

    curl -X POST -H "Content-Type: application/x-www-form-urlencoded" --data "grant_type=client_credentials" --data "resource=https://management.core.windows.net/" --data "client_id=<CLIENT-ID>" --data-urlencode  "client_secret=<CLIENT-SECRET>" https://login.microsoftonline.com/<TENANT-ID>/oauth2/token

    2:

    curl -i -X PUT -H "x-ms-version: 2016-05-31" -H "Content-Length: 0" --data-urlencode "resource=https://management.azure.com/" -H "Authorization: Bearer <ACCESS-TOKEN>" "https://<DATALAKE-NAME>.azuredatalakestore.net/webhdfs/v1/mytempdir/?op=MKDIRS"
    

    Thanks for your help!

    Tuesday, May 7, 2019 1:31 PM