Waiting for pipelines to complete

Question

I have a 5 stage pipe lines built using ActionBlocks and TransformBlocks.  I can't believe this is true but if I want to wait for the piple line to complete I have to write code like this

 

    stage1.DeclinePermanently();
 
                stage1.CompletionTask.Wait();
                stage2.DeclinePermanently();
                stage2.CompletionTask.Wait();
                stage3.DeclinePermanently();
                stage3.CompletionTask.Wait();
                stage4.DeclinePermanently();
                stage4.CompletionTask.Wait();
                stage5.DeclinePermanently();
                stage5.CompletionTask.Wait();

I was kind of hoping that the Decline would proper-gate through or that the interfaces used for CompletionTask and DeclinePermanently() would be non generic allowing me to write a helper method to decline and wait for all

Answer 1

Hi Andrew-

The interface on which CompletionTask lives is non-generic: IDataflowBlock.  It's true that the interface on which DeclinePermanently lives is generic: ITargetBlock<TInput>.

Typically what you'd do in a situation like this, both to help clean up the code and to avoid blocking threads, is to build yourself a small extension method, something like:

    public static void DeclineOnCompletion<TInput>(this IDataflowBlock source, ITargetBlock<TInput> target) {
        source.CompletionTask.ContinueWith(_ => target.DeclinePermanently());
    }

With that, now you can write:

    stage1.DeclineOnCompletion(stage2);
    stage2.DeclineOnCompletion(stage3);
    stage3.DeclineOnCompletion(stage4);
    stage4.DeclineOnCompletion(stage5);

and stage5 will complete as soon as everything has propagated through the system.

We didn't build this in automatically because it's not always what you want, especially in situations where you have multiple sources all feeding into the same target; in a case like that, you typically only want to decline the target when all of the feeder sources have completed.  Other times, you continually replace the sources with new sources as the sources drain, always leaving the target alive.

Thanks for the feedback, and we'll consider whether there's something we can do to improve this further.

Answer 2

Steve,

 

Thanks for the quick response, I agree the auto collapse of the pipeline is not always desirable.  I do hope you consider adding that as an option, or adding your helper method as part of the standard release.

 

Andy

 

---

Andrew Clymer www.rocksolidknowledge.com

Answer 3

Hi Andy-

We'll definitely consider it.  Thanks for the feedback!

Answer 4

For folks finding this answer via Google, this advice is out-of-date.  DeclineOnCompletion() no longer exists in modern TDF (shipped around same time as .NET 4.5, fall 2012).  

For the new way, see: http://blogs.msdn.com/b/pfxteam/archive/2010/10/28/10081950.aspx

If the link doesn't work, try "DataflowLinkOptions" and set "new DataflowLinkOptions() { PropagateCompletion = true }"