none
Data lake File Metadata

    Question

  • This might be a dumb question, but have googled and found little no no answers.

    What I am after is a (exportable) list of the files in my data lake, preferable with some useful attributes.

    In it's simplest form : 

    • File name
    • Size
    • Date Created

    I am sure this must be simple to achieve, but for the life of me !?!?

    Any pointers appreciated.

    Thanks

    Wednesday, August 30, 2017 9:14 AM

Answers

All replies

  • You could use the PowerShell cmdlet Get-​Azure​Rm​Data​Lake​Store​Child​Item.  There is a sample example using PowerShell and U-SQL on stackoverflow @ Does U-SQL support extracting files based on date of creation in ADLS

    • Edited by CateArcher Wednesday, August 30, 2017 4:04 PM links
    • Marked as answer by Esby Tuesday, September 5, 2017 8:32 AM
    Wednesday, August 30, 2017 3:56 PM
  • Yes using PowerShell is the easiest way for this scenario. 
    Thursday, August 31, 2017 1:08 AM
  • I was hoping for a much simpler way to get this info.

    Really surprised that it can't be seen from the portal or VS.

    Yet, given the lack of any alternative answers, and I assume the PS solution works - I'm going to mark as answer.

    Thank you

    Tuesday, September 5, 2017 8:32 AM
  • You can use Azure Data Lake Explorer in Visual Studio to browse the content of ADLS, however I don't think you can export the metadata.

    The solution I would use is to use Power BI with the Data Lake store connector. Once you connect you can browse the content of Data Lake Store using Power Query and you can retrieve only the list of files with metadata if you wish (without loading the actual content of the files).

    Olivier

    Tuesday, September 5, 2017 12:26 PM
  • It is worth mentioning that Azure Data Lake is a source in PowerBI and Excel as of recently. From a collection stand point, Powershell is the way to go as it also provides functionality not currently supported in U-SQL such as Looping.

    PowerBI Displays root level folders as well as drill down into the items. Below is  of descriptive items in PowerBI:

    • Content
    • Name 
    • Extension
    • Date Accessed
    • <g class="gr_ gr_86 gr-alert gr_spell gr_inline_cards gr_run_anim ContextualSpelling ins-del multiReplace" data-gr-id="86" id="86">DateModifed</g> 
    • Date Created
    • Attributes
    • Folder Path



    Friday, September 15, 2017 1:08 AM