none
How to Finish Partial batch on a BatchBlock RRS feed

  • Question

  • Hi, I'm a beginner with TPL dataflows.  I'm trying to use BatchBlock  but it seems too inflexible in non-greedy mode. A batch of size N apparently requires N contributors. Isn't there a way to make an exception to this rule if you're at the very end of your data stream? Why can't I do a partial batch at that point? I toyed around with TriggerBatch but perhaps was doing it wrong.

      BufferBlock<int> headBlock1 = new BufferBlock<int>();
                BufferBlock<int> headBlock2 = new BufferBlock<int>();
                GroupingDataflowBlockOptions options = new GroupingDataflowBlockOptions { Greedy = false, };
                BatchBlock<int> batchBlock = new BatchBlock<int>(2, options);
                ActionBlock<int[]> finalBlock = new ActionBlock<int[]>(ints =>
                {
                     Console.WriteLine(ints[0].ToString());
                    if (ints.Length == 2) Console.WriteLine(ints[1].ToString());
                });
                headBlock1.LinkTo(batchBlock);
                headBlock2.LinkTo(batchBlock);
                batchBlock.LinkTo(finalBlock);
                headBlock1.Post(7);//pends
                headBlock2.Post(8); //flushes nicely. 
                headBlock1.Post(9);//No way to flush final value?
                headBlock1.Complete();
                headBlock2.Complete();
                await Task.WhenAll(headBlock1.Completion, headBlock2.Completion).ContinueWith(x => {            
                    batchBlock.Complete();
                 });
                await Task.WhenAll(batchBlock.Completion).ContinueWith(x => finalBlock.Complete());
                await finalBlock.Completion;
                MessageBox.Show("finished"); //Never shows. 

    Wednesday, October 4, 2017 1:49 PM

Answers

  • I googled around a bit, can't seem to find a ready-made solution, so I guess the workaround is to softcode the options (either the batchsize or the greedy option). So if there is a partial batch leftover, run the pipeline again with a smaller batchsize or with greedy set to true. 
    • Marked as answer by jal2 Wednesday, October 4, 2017 2:11 PM
    Wednesday, October 4, 2017 2:11 PM