How to get the blob storage file name that is getting processed inside ADF V2 pipeline?

Question

I have a blob storage. It contains multiple csv files. I need to process each and every file in ADF V2 and need to log the file name that is processed in database. for e.g. we can determine file1.csv processed successfully or somefile.csv did not processed successfully etc.

How can I achieve this?

Any guidance will be really appreciated?

Thanks

Answer 1

1. You could use get metadata activity to get all the file names. 

2. Then use foreach activity to iterate the output of get metadata activity.

3. then inside the for each activity, it contains a copy activity.

4. the copy activity followed by a on failure activity (stored procedure activity) to log failed file name

5. the copy activity also followed by a on success activity (stored procedure activity) to log success file name

Answer 2

Hello Fang Liu

Thanks for your help. Its working now. However while processing 3 files, if one file fails the other 2 files should be processed successfully. I am facing issue like even one file fails, it logs 3 failure records in log table. How can I handle this?

Answer 3

Three same failure records? Or event the succeeded file name are been logged? Anything wrong in your stored procedure activity? 

Answer 4

Hello Fang Liu

The stored procedure appears to be correct for all 3 success file scenario, it works fine

Answer 5

Hello PankajR, Can I know what you did to catch the filename with the Store procedure?

Best regards