none
Azure Data Lake Gen 2 New API problems

    Question

  • Hi,
    I am trying to get the REST APIs for Azure Data Lake Store Gen 2 to work as we could like to use it in some use cases. Because the Blob API is not supported, I was looking at the new API on https://docs.microsoft.com/en-us/rest/api/storageservices/data-lake-storage-gen2.
    Even there , if we use the "try it" button and enter the details and hit run-Nothing happens. Not even an error. Just does not do anything.
    I have also tried using the Bearer token by creating a service principal and making it "Storage Blob Content Contributer" but to not avail. The request in Python goes through without errors.

    Also, tried it Shared Key and that even there, though no errors, nothing really happens.

    This is the Python code for the Bearer token

    
    
    from oauthlib.oauth2 import BackendApplicationClient
    from requests_oauthlib import OAuth2Session
    import requests
    import datetime
    
    
    
    clientID="MyClientID"
    clientSecret="MyClientSecret"
    
    
    
    client = BackendApplicationClient(client_id=clientID)
    oauth = OAuth2Session(client=client)
    
    proxies = {
    "https": "http://127.0.0.1:8888",
    }
    
    token = oauth.fetch_token(token_url='https://login.microsoftonline.com/<tenantID>/oauth2/token', client_id=clientID,client_secret=clientSecret,proxies=proxies) 
    
    auth_t=token['access_token']
    
    
    headers = {
    'Authorization': "Bearer " + auth_t,
    'content-type': "application/json"
    }
    
    try:
        requests.put('https://<mystorageaccount>.dfs.core.windows.net/myfilesystemname?resource=filesystem&timeout=5', headers=headers, proxies=proxies)
    except Exception as e:
        print(e)

    And bwlow is the code for shared Key, which I basically used from https://stackoverflow.com/questions/52839744/put-request-to-azure-storage-with-shared-key-authorization

    import requests
    import datetime
    import hmac
    import hashlib
    import base64
    
    storage_account_name = 'mystorageaccount'
    storage_account_key = 'myaccesskey'
    api_version = '2018-07-01'
    request_time = datetime.datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT')
    
    proxies = {
    "https": "http://127.0.0.1:8888",
    }
    
    
    string_params = {
        'verb': 'PUT',
        'Content-Encoding': '',
        'Content-Language': '',
        'Content-Length': '',
        'Content-MD5': '',
        'Content-Type': 'application/xml',
        'Date': '',
        'If-Modified-Since': '',
        'If-Match': '',
        'If-None-Match': '',
        'If-Unmodified-Since': '',
        'Range': '',
        'CanonicalizedHeaders': 'x-ms-date:' + request_time + '\nx-ms-version:' + api_version + '\n',
        'CanonicalizedResource': '/' + storage_account_name + '/\ncomp:properties\nrestype:service'
    }
    
    string_to_sign = (string_params['verb'] + '\n'
                      + string_params['Content-Encoding'] + '\n'
                      + string_params['Content-Language'] + '\n'
                      + string_params['Content-Length'] + '\n'
                      + string_params['Content-MD5'] + '\n'
                      + string_params['Content-Type'] + '\n'
                      + string_params['Date'] + '\n'
                      + string_params['If-Modified-Since'] + '\n'
                      + string_params['If-Match'] + '\n'
                      + string_params['If-None-Match'] + '\n'
                      + string_params['If-Unmodified-Since'] + '\n'
                      + string_params['Range'] + '\n'
                      + string_params['CanonicalizedHeaders']
                      + string_params['CanonicalizedResource'])
    
    signed_string = base64.b64encode(hmac.new(base64.b64decode(storage_account_key), msg=string_to_sign.encode('utf-8'), digestmod=hashlib.sha256).digest()).decode()
    
    headers = {
        'x-ms-date' : request_time,
        'x-ms-version' : api_version,
        'Content-Type': 'application/xml',
        'Content-Length': '',
        'Authorization' : ('SharedKey ' + storage_account_name + ':' + signed_string)
    }
    
    url = ('https://' + storage_account_name + '.dfs.core.windows.net/myfilesystemname?resource=filesystem')
    
    try:
        r = requests.put(url, headers = headers, proxies=proxies)
    except Exception as e:
        print(e)

    This too runs and I can see in Fiddler , in both cases, that I get a HTTP Status of 200.

    Also, as I mentioned also trying from the WEB UI in Microsoft docs, does not work. I just can't create a filesystem from the API - https://docs.microsoft.com/en-in/rest/api/storageservices/datalakestoragegen2/filesystem/create

    What am I missing?



    Tuesday, February 12, 2019 2:41 PM

Answers

  • So, Braja from Microsoft directed me to some really helpful resources which helped me in getting it working , finally.

    For the benefit of others, who might encounter this or are encountering this:-

    what I was missing :

    1. Access to the API : Earlier , I had just granted the Service Principal the Owner and the Blob Contributor role to the storagee account but I didn’t know that I have to give it access to the API as well from the portal.  I followed the steps in https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-app#grant-your-registered-app-permissions-to-azure-storage

    to grant permissions to the API (section : Grant your registered app access to Azure Storage)

    1. For the access token generation in Python, I was using the following:

     

    token = oauth.fetch_token(token_url='https://login.microsoftonline.com/<tenantID>/oauth2/token', client_id=clientID,client_secret=clientSecret, proxies=proxies)

     

    auth_t=token['access_token']

     

    As you can see , I was missing the resource parameter. So, I changed it to

                  

                   token = oauth.fetch_token(token_url='https://login.microsoftonline.com/<tenantID>/oauth2/token', client_id=clientID,client_secret=clientSecret, resource='https://storage.azure.com/', proxies=proxies)

                  auth_t=token['access_token']

    That's it. Finally worked !

    Wednesday, February 13, 2019 1:01 PM

All replies

  • Further update:

    Regarding the Bearer code method in Python: I see that I get a 401- InvalidAuthentication. I do not know why that is as I am including the bearer token.

    Regarding the SharedKey:

    I realized that my cannocialResource was probably not correct , so I made the following change, and also setting the content length to 0 but I still get: "InvalidHeaderValue","message":"The value for one of the HTTP headers is not in the correct format.

    Below is the updated code for the Shared Key

    import requests
    import datetime
    import hmac
    import hashlib
    import base64
    
    storage_account_name = 'myaccountname'
    storage_account_key = 'my account key'
    api_version = '2018-07-01'
    request_time = datetime.datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT')
    
    proxies = {
    "https": "http://127.0.0.1:8888",
    }
    
    
    string_params = {
        'verb': 'PUT',
        'Content-Encoding': '',
        'Content-Language': '',
        'Content-Length': '0',
        'Content-MD5': '',
        'Content-Type': 'application/json',
        'Date': '',
        'If-Modified-Since': '',
        'If-Match': '',
        'If-None-Match': '',
        'If-Unmodified-Since': '',
        'Range': '',
        'CanonicalizedHeaders': 'x-ms-date:' + request_time + '\nx-ms-version:' + api_version + '\n',
        'CanonicalizedResource': '/' + storage_account_name + '/mynewfilesystemname\nresource:filesystem'
    }
    
    string_to_sign = (string_params['verb'] + '\n'
                      + string_params['Content-Encoding'] + '\n'
                      + string_params['Content-Language'] + '\n'
                      + string_params['Content-Length'] + '\n'
                      + string_params['Content-MD5'] + '\n'
                      + string_params['Content-Type'] + '\n'
                      + string_params['Date'] + '\n'
                      + string_params['If-Modified-Since'] + '\n'
                      + string_params['If-Match'] + '\n'
                      + string_params['If-None-Match'] + '\n'
                      + string_params['If-Unmodified-Since'] + '\n'
                      + string_params['Range'] + '\n'
                      + string_params['CanonicalizedHeaders']
                      + string_params['CanonicalizedResource'])
    print(string_params['CanonicalizedHeaders'])
    print(string_params['CanonicalizedResource'])
    
    signed_string = base64.b64encode(hmac.new(base64.b64decode(storage_account_key), msg=string_to_sign.encode('utf-8'), digestmod=hashlib.sha256).digest()).decode()
    
    headers = {
        'x-ms-date' : request_time,
        'x-ms-version' : api_version,
        'Content-Length': '0',
        'Content-Type': 'application/json',
        'Authorization' : ('SharedKey ' + storage_account_name + ':' + signed_string)
    }
    
    url = ('https://' + storage_account_name + '.dfs.core.windows.net/mynewfilesystemname?resource=filesystem')
    
    try:
        r = requests.put(url, headers = headers, proxies=proxies, verify=False)
        print(r.content)
    except Exception as e:
        print(e)
    

    Wednesday, February 13, 2019 9:49 AM
  • So, Braja from Microsoft directed me to some really helpful resources which helped me in getting it working , finally.

    For the benefit of others, who might encounter this or are encountering this:-

    what I was missing :

    1. Access to the API : Earlier , I had just granted the Service Principal the Owner and the Blob Contributor role to the storagee account but I didn’t know that I have to give it access to the API as well from the portal.  I followed the steps in https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-app#grant-your-registered-app-permissions-to-azure-storage

    to grant permissions to the API (section : Grant your registered app access to Azure Storage)

    1. For the access token generation in Python, I was using the following:

     

    token = oauth.fetch_token(token_url='https://login.microsoftonline.com/<tenantID>/oauth2/token', client_id=clientID,client_secret=clientSecret, proxies=proxies)

     

    auth_t=token['access_token']

     

    As you can see , I was missing the resource parameter. So, I changed it to

                  

                   token = oauth.fetch_token(token_url='https://login.microsoftonline.com/<tenantID>/oauth2/token', client_id=clientID,client_secret=clientSecret, resource='https://storage.azure.com/', proxies=proxies)

                  auth_t=token['access_token']

    That's it. Finally worked !

    Wednesday, February 13, 2019 1:01 PM
  • Thank you for sharing your solution and experience here. It will be greatly beneficial for other community members who have similar questions.

    Wednesday, February 13, 2019 1:53 PM
    Moderator