none
How to get the blob storage file name that is getting processed inside ADF V2 pipeline?

    Question

  • I have a blob storage. It contains multiple csv files. I need to process each and every file in ADF V2 and need to log the file name that is processed in database. for e.g. we can determine file1.csv processed successfully or somefile.csv did not processed successfully etc.

    How can I achieve this?

    Any guidance will be really appreciated?

    Thanks

    Sunday, August 26, 2018 4:53 AM

Answers

  • 1. You could use get metadata activity to get all the file names. 

    2. Then use foreach activity to iterate the output of get metadata activity.

    3. then inside the for each activity, it contains a copy activity.

    4. the copy activity followed by a on failure activity (stored procedure activity) to log failed file name

    5. the copy activity also followed by a on success activity (stored procedure activity) to log success file name

    Monday, August 27, 2018 12:22 PM

All replies

  • 1. You could use get metadata activity to get all the file names. 

    2. Then use foreach activity to iterate the output of get metadata activity.

    3. then inside the for each activity, it contains a copy activity.

    4. the copy activity followed by a on failure activity (stored procedure activity) to log failed file name

    5. the copy activity also followed by a on success activity (stored procedure activity) to log success file name

    Monday, August 27, 2018 12:22 PM
  • Hello Fang Liu

    Thanks for your help. Its working now. However while processing 3 files, if one file fails the other 2 files should be processed successfully. I am facing issue like even one file fails, it logs 3 failure records in log table. How can I handle this?

    Friday, August 31, 2018 2:25 AM
  • Three same failure records? Or event the succeeded file name are been logged? Anything wrong in your stored procedure activity? 
    Friday, August 31, 2018 7:13 AM
  • Hello Fang Liu

    The stored procedure appears to be correct for all 3 success file scenario, it works fine

    Friday, August 31, 2018 9:32 AM