none
Azure Data Lake Storage ACLs and Azure Active Directory Applications

    Question

  • I have an application registered in my tenant AAD instance called EmpowerWebApp.  This application is authorized to interact with Azure Data Lake and Windows Azure Active Directory.  This application is used by a web app I'm currently developing for a client to ease administration of data in an ADLS Data Lake.

    This web application already allows users to log in and authenticate via AAD using EmpowerWebApp.  Now, I'm trying to upload raw files to ADLS with this same application. 

    In my ADL Storage Account, I have created a folder /EmpowerFirst/raw.  This will be the landing area for files from our users.  I've added ACLs and Default ACLs to the /EmpowerFirst folder for AAD groups as well as for our application.  EmpowerWebApp has read, write, and execute permissions on /EmpowerFirst, and via the default ACL this goes all the way down.  So, if you open up /EmpowerFirst/raw/TestTenant, you still see that EmpowerWebApp has read, write, and execute permissions there.

    When the web app gets to the step where it tries to create a file and then <g class="gr_ gr_1274 gr-alert gr_gramm gr_inline_cards gr_run_anim Grammar multiReplace" data-gr-id="1274" id="1274">write</g> to it, I get an error.

    Error in creating file /EmpowerFirst/raw/TestTenant/Character.csv.
    Operation: CREATE failed with HttpStatus:Forbidden RemoteException: AccessControlException  [628bed3f-2fc3-4411-99e2-dfba8281be68] Access Denied : /EmpowerFirst/raw/TestTenant/Character.csv[2017-12-12T14:19:03.7446767-08:00] JavaClassName: org.apache.hadoop.security.AccessControlException.
    Last encountered exception thrown after 1 tries. [Forbidden: AccessControlException]
    [ServerRequestId:]

    Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. 

    Exception Details: Microsoft.Azure.DataLake.Store.AdlsException: Error in creating file /EmpowerFirst/raw/TestTenant/Character.csv.
    Operation: CREATE failed with HttpStatus:Forbidden RemoteException: AccessControlException  [628bed3f-2fc3-4411-99e2-dfba8281be68] Access Denied : /EmpowerFirst/raw/TestTenant/Character.csv[2017-12-12T14:19:03.7446767-08:00] JavaClassName: org.apache.hadoop.security.AccessControlException.
    Last encountered exception thrown after 1 tries. [Forbidden: AccessControlException]

    I can add files to ADLS through the data explorer, but not through this app using EmpowerWebApp.  I can confirm that is the account being used for the CREATE by checking the audit logs from ADLS.  They show an error type 403 for the GUID that represents the EmpowerWebApp.

    What's different about writing to <g class="gr_ gr_1672 gr-alert gr_spell gr_inline_cards gr_run_anim ContextualSpelling ins-del multiReplace" data-gr-id="1672" id="1672">ADLS</g> via an application compared to writing from a user?  How can I get this application to write files?

    Tuesday, December 12, 2017 10:52 PM

All replies

  • Hi Shannon, 

    Could you possibly share an example of the code that your app is running in order to upload that file? 

    -Micah

    Tuesday, December 12, 2017 11:23 PM
  • //from the class controller

    private static string ClientId = ConfigurationManager.AppSettings["ida:ClientId"];
    private static string ClientSecret = ConfigurationManager.AppSettings["ida:ClientSecret"];
    private static string Domain = ConfigurationManager.AppSettings["ida:Domain"];
    private static string ADLSAccountFQDN = ConfigurationManager.AppSettings["ida:ADLSAccountFQDN"]; 

    //from the async task

    ServiceClientCredentials clientCreds = await ApplicationTokenProvider.LoginSilentAsync(Domain, ClientId, ClientSecret); AdlsClient adlsClient = AdlsClient.CreateClient(ADLSAccountFQDN, clientCreds);

    string destFile = UploadPath + "/" + SelectedFile.FileName; using (var streamWriter = new StreamWriter(adlsClient.CreateFile(destFile, IfExists.Overwrite, null, true))) { streamWriter.WriteLine("This is test data to write"); streamWriter.WriteLine("This is line 2"); }


    Wednesday, December 13, 2017 1:42 AM
  • Thanks Shannon. So it seems that the app you are trying to use to write the file with does not have the correct permissions even though you have mentioned it does have read, write and execute. 

    Can you tell me where you are checking those access levels? 

    In my ADL I can see them by going 

    Azure Data Lake -> Data Explorer -> File Path -> Access

    I can also see access via Azure Data Lake -> Access Control (IAM)

    Could you see if your app has permissions on both levels? If not try to add it? 

    Wednesday, December 13, 2017 2:02 AM
  • Here are the permissions from the data explorer and IAM points of view.

    Data Explorer PermissionsIAM Permissions

    Wednesday, December 13, 2017 11:31 AM
  • Thanks Shannon. So it for sure seems your app does have the correct permissions. 

    Let's try this, can you give your App permission at the subscription level as well? I would say give the app the highest permission possible at first (owner). If it fixes the write error then we can lower the permission until it is just right. 

    To do this go to the Portal -> Subscriptions -> Select your Subscription -> Access Control (IAM) -> Add

    Wednesday, December 13, 2017 5:39 PM
  • I've elevated EmpowerWebApp to Owner of the subscription.

    Unfortunately I get the same error (only the timestamp and reference GUID change).

    Wednesday, December 13, 2017 10:05 PM
  • Hmm okay. Do you happen to remember what encryption option you picked for your Data Lake?

    Maybe that is causing an issue

    Wednesday, December 13, 2017 11:19 PM
  • Another thing I found that might help:

    Go to Azure Active Directory -> App Registrations -> New Application Registration 

    Enter the Name of your App and provide the Sign-on URL

    This information was based on this article: 

    https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-service-to-service-authenticate-using-active-directory

    Wednesday, December 13, 2017 11:36 PM
  • "This account is encrypted using keys managed by Azure Data Lake Store." -- per the Encryption Settings blade.
    Thursday, December 14, 2017 2:23 PM
  • Would this need to be a completely new registration?  Would I spin up a new application to replace the one I have now?

    Thursday, December 14, 2017 2:34 PM
  • You should be able to use your current application. I see nothing in the doc that suggests the app needs to be brand new. 
    Thursday, December 14, 2017 5:49 PM
  • I wanted to share all the settings for my environment for each of the steps listed in https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-service-to-service-authenticate-using-active-directory  I went through them again, but saw nothing different from last time. I'm sure I'm missing something.  Hopefully, by spelling it out, someone will spot my error.

     

    Step 1: Create an Active Directory web application

     

    I gave the app a name, and chose Web app/API.


     

    Step 2: Get application ID, authentication key, and tenant ID

     

    You can see the Application ID in the last image. At setup I also created a new client secret to be used by the app. I also have my tenant's domain name, as well as the tenant GUID.

     

     

    Step 3: Assign the Azure AD application to the Azure Data Lake Store account file or folder

    I've shown this one a few times, but here's the permission at the "root" folder that the application will be working in (/EmpowerFirst).

     

    Read, Write, and Execute are granted for the application (and it's set the same as my permission in the SG Data Management group -- which is able to write files to /Empower/Raw/TestTenant/)

     

    I would like to note the add permissions blades are a little different than the article at this time. But that happens a lot in Azure documentation.

     

    Note

    If you want to use the SDKs to create a Data Lake Store account, you must assign the Azure AD web application as a role to the Resource Group in which you create the Data Lake Store account.

     

     The Access Control (IAM) settings has come up several times throughout the debug process, so I wanted to share all those settings with you as well. 

     

    Here's the IAM at the subscription level:

     

    And at the resource group level:

     

    And finally, at the datalake level:

     

     

    Step 4: Get the OAuth 2.0 token endpoint (only for Java-based applications)

    I'm using a C#, .NET MVC web app, so this isn't applicable for our case

     

     

    Over in the app

    I borrowed from the github demo code pretty heavily.    The first part comes from https://github.com/Azure-Samples/active-directory-dotnet-webapp-openidconnect I used the structure of the Account controller and startup.Auth files to allow users to "log in" to this app.

     

    That works like a champ!

     

    I also borrowed from the two ADLS examples: https://github.com/Azure-Samples/data-lake-dotnet-client/blob/master/src/DemoAdlClient/Program.cs and https://github.com/Azure-Samples/data-lake-store-adls-dot-net-samples

     

    I don't get an error on creating the client using the client secret and key:

     

    ServiceClientCredentials clientCreds = await ApplicationTokenProvider.LoginSilentAsync(Domain, ClientId, ClientSecret);

     

    AdlsClient adlsClient = AdlsClient.CreateClient(ADLSAccountFQDN, clientCreds);

     

    The variables in those two lines are fed from ConfigurationManager and in turn that comes from web.config.

     

    In fact, in the ADLS Audit logs, I see EmpowerWebApp connecting and getting a CheckAccess request to complete successfully.  I do see it attempting GetFileStatus, but that always fails.  It also fails to create a file too.

     

    Hopefully someone spots a problem in the settings or code above.

     

     

    Thursday, December 14, 2017 7:05 PM
  • Thanks for all of that detail! It really seems like you have everything covered so its frustrating you continue to get the errors...At this point we might need to get you in contact with an engineer to help you on a 1:1 level. 

    Would you be able to email me at AzCommunity@Microsoft.com with your subscription ID? I can then enable you account for a free support request. This way you can open a ticket directly to a Support Engineer in Azure Data Lake and work 1:1 with them via email, phone and screen share to get your app working as expected. 

    -Micah

    Thursday, December 14, 2017 9:25 PM
  • Hi Shannon! 

    I was checking in on your support ticket and it looks like you were able to get your app to work properly with your ADL. 

    From reading the notes it seems you needed to Add the execute permission to the root node and that was it. 

    I also found that this is some of the information used to help figure that out: 

    https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-access-control

    Please feel free to add/ correct anything here for future users who might have this same issue :) 

    Thanks!

    -Micah

    Thursday, December 21, 2017 6:38 PM