How to iterate through files in Blob Storage in ADF V2

Question

Hi All,

Need Help, 

I am trying to iterate through files in Blob Storage using ADF v2. 

I am trying to do using For Each Loop Activity however I am not able configure it.

Can someone suggest me how to do.

Regards,

Sam

Answer 1

Hi SamJyothi,

There's a similar issue reported here: 

https://social.msdn.microsoft.com/Forums/Lync/en-US/f84549c8-67df-49f3-8d20-cb036ac0b7fa/iterating-through-all-the-files-in-azure-blob-storage?forum=AzureDataFactory

The answer on the post might be useful in your case as well. 

Let us know if this helps. Else, we can continue to probe in further.

---

MSDN

Answer 2

Hi Chirag,

Thank you for the update.

We were able to resolve it. Below is the approach we have followed.

Requirement: Need to process Data files received in Blob Storage on daily basis. This Blob store will receive various types of data files and each type will have multiple files from various sources.

Design: For each type of file we created a Pipeline and this pipeline has GetMetaData Activity, ForEach Activity, If Activity, and Data Copy Activity.

GetMetaData activity has dataset which will holds list of files in the blob store and pass it to ForEachActivity.

The ForEachActivity will process each file: First step file is passed to If Activity which check whether filename is of particular type and if true it will pass to copy activity.

Answer 3

Awesome. Glad to hear that your issue was resolved. Thanks for sharing your findings. It will benefit the community at large.

Cheers!

---

MSDN