Completion of complex graph - DataflowGraph object?

Question

I am interested in a dynamically constructed scenario where I have a set of "feeders" passing messages in a many-to-many relationship to a set of "workers", something like

            ActionBlock<T>[] workers;
            BufferBlock<T>[] feeders;
            ...
            int[][] feedersForWorkers;
            for (int j1 = 0; j1 < workers.Count; ++j1) {
                foreach (int i1 in feedersForWorkers[j1]) {
                    feeders[i1].LinkTo(workers[j1])
                }
            }
            ...

Not all feeders are appropriate for all workers, but some workers may share some feeders.

My question is, what is the recommended pattern for completing the dataflow? As best I can come up with, the right way of completing a complex dataflow graph involves (1) calling complete on all initial nodes, (2) creating an auxiliary task for each non-initial node that waits on the completion of its predecessors, and finally (3) waiting globally on the auxiliaries for terminal nodes. In my bipartite case,

            ...
            foreach (BufferBlock<T> feeder in feeders) {
                feeder.Complete();
            }
            Task[] workerCompleters = new Task[workers.Count];
            for (int j1 = 0; j1 < workers.Count; ++j1) {
                Task[] feederCompletions = Array.ConvertAll<int, Task>(
                        feedersForWorkers[j1],
                        (i1) => feeders[i1].Completion);
                workerCompleters[j1] = Task.Factory.ContinueWhenAll(
                        feederCompletions,
                        (upstreamCompletionAr) => worker[j1].Complete());
            }
            Task.WaitAll(workerCompleters);
            ...

First, if there is a better pattern, I would be interested to know. Second, I wondered if you had thought of possibly adding a DataflowGraph convenience class to manage global aspects of dataflows, especially this kind of completion. The idea would be that you would construct things so that your DataflowGraph object knows about all your nodes and links. The object could then provide methods to globally start, cancel or complete dataflows. That would spare the user the effort of setting up completions as above to propagate correctly.

I haven't pondered deep ramifications of this. Just a thought.

---

Chris Stover

Answer 1

Hi Chris-

Thanks for the feedback.  We have considered adding graph management capabilities, but as you're aware thus far we haven't done so.  As part of upcoming additions to the library, we are planning to make it easier to flow completion notification from one block to another, but only in the linear case; for more complex cases, e.g. multiple sources feeding into a single target, it'll still be up to the developer to handle the completion flowing, similar to how you're doing with ContinueWhenAll.