locked
Azure Data Factory - Get a List of Files but ignore folders RRS feed

  • Question

  • Hi,

    Using a 'Get Metadata' component I have successfully retrieve a list of "files and folders" from an on-premise folder.

    The list contains 'files' and 'folders' - the 'folders' in the list is causing an issue in later processing. I process the file/folder list in a 'ForEach' loop (@activity('Get Source File List').output.childitems)

    I simply want to ignore folders when creating the list or ignore them during processing - how do I do this in Data Factory?

    Regards

    Ian

    Thursday, August 15, 2019 8:49 AM

All replies

  • Hello Ian,

    In order to avoid folders, you can use a 'Filter Activity' between GetMetadata activity and For Each Activity then have a condition in Filter Activity to skip folders. 

    You can have a expression as below to achieve your requirement.

    Expressions for Filter Activity and condition: 

    Items: @activity('Get Metadata1').output.childItems


    Condition: @not(contains(item().type, 'Folder'))

    Hope this helps.



    Additional info:
    To explore about Filter Activity, please refer to this doc: Filter activity in ADF


    Thursday, August 15, 2019 8:25 PM
  • Hi,

    I am still having trouble with this.

    In the 'For Each Loop' I have a copy activity (copy file(s) from on premise to ADL) then a delete activity (delete the source on premise file after the copy). The below screen shot shows how I refer to the filtered (folder filtered out) in the 'For Each Loop'...

    The 'Copy Activity' in the 'For Each Loop' still refers to a 'folder' and thinks it's a 'file' - the pipeline then throws an error, message shown below... 

     

    • Edited by Caz71 Sunday, August 18, 2019 10:38 AM
    Sunday, August 18, 2019 10:32 AM
  • Hi there,

    Just checking to see if the above suggestion answered your query? 


    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Monday, August 19, 2019 4:55 PM
  • Hi Ian,

    Just checking in to see if the above suggestion was helpful. If this answers your query, please consider to click “Mark as Answer” and Up-Vote, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.


    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Tuesday, August 20, 2019 5:43 PM