locked
How to iterate through files in Blob Storage in ADF V2 RRS feed

  • Question

  • Hi All,

    Need Help, 

    I am trying to iterate through files in Blob Storage using ADF v2. 

    I am trying to do using For Each Loop Activity however I am not able configure it.

    Can someone suggest me how to do.

    Regards,

    Sam

    • Moved by YASWANTH MADI Wednesday, January 30, 2019 9:10 AM better suited here than storage
    Wednesday, January 30, 2019 8:33 AM

Answers

  • Hi Chirag,

    Thank you for the update.

    We were able to resolve it. Below is the approach we have followed.

    Requirement: Need to process Data files received in Blob Storage on daily basis. This Blob store will receive various types of data files and each type will have multiple files from various sources.

    Design: For each type of file we created a Pipeline and this pipeline has GetMetaData Activity, ForEach Activity, If Activity, and Data Copy Activity.

    GetMetaData activity has dataset which will holds list of files in the blob store and pass it to ForEachActivity.

    The ForEachActivity will process each file: First step file is passed to If Activity which check whether filename is of particular type and if true it will pass to copy activity.

    • Marked as answer by samJyothi Wednesday, February 27, 2019 6:41 AM
    Wednesday, February 27, 2019 6:41 AM

All replies

  • Hi SamJyothi,

    There's a similar issue reported here: 

    https://social.msdn.microsoft.com/Forums/Lync/en-US/f84549c8-67df-49f3-8d20-cb036ac0b7fa/iterating-through-all-the-files-in-azure-blob-storage?forum=AzureDataFactory

    The answer on the post might be useful in your case as well. 

    Let us know if this helps. Else, we can continue to probe in further.


    MSDN


    Friday, February 1, 2019 11:44 AM
  • Hi Chirag,

    Thank you for the update.

    We were able to resolve it. Below is the approach we have followed.

    Requirement: Need to process Data files received in Blob Storage on daily basis. This Blob store will receive various types of data files and each type will have multiple files from various sources.

    Design: For each type of file we created a Pipeline and this pipeline has GetMetaData Activity, ForEach Activity, If Activity, and Data Copy Activity.

    GetMetaData activity has dataset which will holds list of files in the blob store and pass it to ForEachActivity.

    The ForEachActivity will process each file: First step file is passed to If Activity which check whether filename is of particular type and if true it will pass to copy activity.

    • Marked as answer by samJyothi Wednesday, February 27, 2019 6:41 AM
    Wednesday, February 27, 2019 6:41 AM
  • Awesome. Glad to hear that your issue was resolved. Thanks for sharing your findings. It will benefit the community at large.

    Cheers!


    MSDN

    Wednesday, February 27, 2019 9:29 AM