locked
Until Activity - No more Files Read RRS feed

  • Question

  • Hello!

    I have a pipeline working (for Pre-Production...we are supposed to Go Live soon) which has an Event Trigger on blob files which are written to a particular folder. The Pipeline then processes (mainly a Copy Activity) that ONE file and finishes. Well, this week in performance testing, this one trigger reading and copying one file is way too slow. Therefore, I don't want to change my Pipeline SO I merged many files into a single file, and when the file containing multiple rows of data has been written to the Blob the trigger calls the same Pipeline. 

    My question is:

    How can I use a Until Activity (that seems the best) to say something like "Read the first row of data and run it through the pipeline and then read the next row, until there are no more rows". Do I just placed the Until Activity at the beginning of the Pipeline and use a Success Connector to connect my last Activity to the Until? How, in the Until Activity, can I say "if there are no more rows of data left in the merged file then terminate the Pipeline" or something like that. I have spent several hours googling Until but I am still not sure how to implement "read the rows of data in the merged file until there are no more rows left"?

    ALSO, I can't figure out what this does, although I see it when Googling: 

    pipeline().parameters.repeat

    Thank you so much!
    Mike Kiser


    Mike Kiser


    • Edited by EMKISER Friday, November 22, 2019 2:27 AM
    Friday, November 22, 2019 2:22 AM

Answers

  • I think what you need is to iterate a blob one row at a time . Since you mentioned that the initial copy was slow I am assuming that the blob is huge and so may be you cannot use lookup . What is max number of rows on the file ?At the time lookup only supports 5000 rows . It looks like if the file is not huge lookup + foreach can be useful .

    Thanks Himanshu

    • Marked as answer by EMKISER Friday, November 22, 2019 11:36 PM
    Friday, November 22, 2019 11:28 PM

All replies

  • I think what you need is to iterate a blob one row at a time . Since you mentioned that the initial copy was slow I am assuming that the blob is huge and so may be you cannot use lookup . What is max number of rows on the file ?At the time lookup only supports 5000 rows . It looks like if the file is not huge lookup + foreach can be useful .

    Thanks Himanshu

    • Marked as answer by EMKISER Friday, November 22, 2019 11:36 PM
    Friday, November 22, 2019 11:28 PM
  • Hi Himanshu,

    Thanks for your reply; I wasn't clear. When I had an event trigger on each (one data row) blob created, there were hundreds of pipelines processing these one data rows. So I am combining the blobs into ONE Blob with many rows. The rows still probably won't go over 100 or so, to give a rough idea. 

    Do you have a good example of using the lookup+foreach? That sounds like what I probably need instead of using an UNTIL activity.

    Thanks so much!
    Mike


    Mike Kiser

    Friday, November 22, 2019 11:43 PM