Answered by:
How to iterate through files in Blob Storage in ADF V2

Question
-
Hi All,
Need Help,
I am trying to iterate through files in Blob Storage using ADF v2.
I am trying to do using For Each Loop Activity however I am not able configure it.
Can someone suggest me how to do.
Regards,
Sam
- Moved by YASWANTH MADI Wednesday, January 30, 2019 9:10 AM better suited here than storage
Wednesday, January 30, 2019 8:33 AM
Answers
-
Hi Chirag,
Thank you for the update.
We were able to resolve it. Below is the approach we have followed.
Requirement: Need to process Data files received in Blob Storage on daily basis. This Blob store will receive various types of data files and each type will have multiple files from various sources.
Design: For each type of file we created a Pipeline and this pipeline has GetMetaData Activity, ForEach Activity, If Activity, and Data Copy Activity.
GetMetaData activity has dataset which will holds list of files in the blob store and pass it to ForEachActivity.
The ForEachActivity will process each file: First step file is passed to If Activity which check whether filename is of particular type and if true it will pass to copy activity.
- Marked as answer by samJyothi Wednesday, February 27, 2019 6:41 AM
Wednesday, February 27, 2019 6:41 AM
All replies
-
Hi SamJyothi,
There's a similar issue reported here:
https://social.msdn.microsoft.com/Forums/Lync/en-US/f84549c8-67df-49f3-8d20-cb036ac0b7fa/iterating-through-all-the-files-in-azure-blob-storage?forum=AzureDataFactory
The answer on the post might be useful in your case as well.
Let us know if this helps. Else, we can continue to probe in further.
MSDN
- Proposed as answer by ChiragMishra-MSFTMicrosoft employee Friday, February 1, 2019 11:44 AM
- Edited by ChiragMishra-MSFTMicrosoft employee Tuesday, February 26, 2019 8:43 AM Corrected a typo
Friday, February 1, 2019 11:44 AM -
Hi Chirag,
Thank you for the update.
We were able to resolve it. Below is the approach we have followed.
Requirement: Need to process Data files received in Blob Storage on daily basis. This Blob store will receive various types of data files and each type will have multiple files from various sources.
Design: For each type of file we created a Pipeline and this pipeline has GetMetaData Activity, ForEach Activity, If Activity, and Data Copy Activity.
GetMetaData activity has dataset which will holds list of files in the blob store and pass it to ForEachActivity.
The ForEachActivity will process each file: First step file is passed to If Activity which check whether filename is of particular type and if true it will pass to copy activity.
- Marked as answer by samJyothi Wednesday, February 27, 2019 6:41 AM
Wednesday, February 27, 2019 6:41 AM -
Awesome. Glad to hear that your issue was resolved. Thanks for sharing your findings. It will benefit the community at large.
Cheers!
MSDN
Wednesday, February 27, 2019 9:29 AM