Can I create 10~100K BufferBlocks for calculation?

Question

I'm trying to create a aggregation in parallel to a data set with several level of grouping, so there would be 10~100K aggregation results and each result can be calculated by a single aggregator individually. In erlang style of data processing it's easy to create more than 10~100K actors and we just need to send messages to them for calculation, but can we use this style with TPL DataFlow?

---

Happy programming, happy life.
http://blog.zhaojie.me/

Answer 1

I don't see any fundamental reason why you couldn't do that. Dataflow blocks are just object that perform some computations using Tasks (usually on the ThreadPool) when necessary. They don't hold any limited or expensive resources, most of the time.

But I'm really not sure if doing that would be the best approach. Basically, I think that each block should have different role in your network. And I can't imagine having a program with 100 thousand different roles. Dataflow blocks are meant for processing streams of data, not just single results.

Answer 2

In my scenario, each block work as an aggregater and there're hundred and thousand of aggregators. The change of data set would be posted to the affected aggregators. So in any time there're maybe only small partial of aggregators need to work but every aggreagator has chance to update its result.

---

Happy programming, happy life.
http://blog.zhaojie.me/

Answer 3

It's fine to have thousands of blocks, especially if only a few of them need to be processing at a time.  As svick said, blocks consume very few resources when they're not actively processing; at that point they're just a data structure in memory.  The only time a block is consuming a thread is when it's actively processing, as at that point it's spun up a Task to do the work.