performance and best pattern for writing pipeline of sequnetial jobs


  • Hi,

    i am planning to split my single big job into multiple jobs, with each doing a specific step and taking the input from previous job and passing the result to next job.

    basically each step gets input, do some filtering and passed the filtered data to next job.

    with this approach

    • any recommendation/guidance to do it? i am planning to use custom table as input and output to each Table valued functions. Later I can be adding more TVF/filtering steps in between.
    • since my data passed across functions will be large, is there any performance hit, if doing everything in same job (individual steps in jobs are still sequential to be executed, as next step need filtered data from previous step).
    • For each TVF filtering step, i want to dump the records that were filtered out. I have where clause, so anything that gets filtered out in where, i need to write it to separate table.



    Friday, November 17, 2017 7:38 PM