How To Implement a DataFlow pipeline That Outputs a Lazy IEnumerable<T> RRS feed

  • Question

  • Hi,

    I have a WebAPI scenario in which I transform a user request to multiple queries, execute them (in parallel if I can), and stream the results back to the user as a lazy IEnumerable (i.e. using 'yield return').

    The laziness of the IEnumerable is important as when results are materialized in the server and then sent to the client, they consume a lot of memory on the server. The multiple underlying data platforms allow me to enumerate results right from the network stream so I avoid materializing them on server as much as I can.

    I was wandering if I could somehow utilize DataFlow for its awesome power of processing and Producer/Consumer like features, MaxDOP control etc. in an async/await context.

    The problem is that I'm missing some mechanism that allows me to start streaming the results when the first async query completes, and not after all of them complete

    I thought of implementing a custom DF block, but the issue above still eludes me

    Thursday, February 23, 2017 4:20 PM