none
Rx and large sequences

    Question

  • I'm just beginning to look at Rx for async interaction with REST sourced sequences of data items.

    Clearly Rx represents the "push" metaphor whereas IEnumerable represents the "pull" metaphor.

    In my mind I'm wondering how to manage huge sequences from which we may only want a few elements.

    In a specific example we have a REST method that uses "paging" each call returns 100 items + a bookmark, the wrapper code uses yield etc to expose this as an IEnumerable<Item> and when the 100 have been yielded it internally does the next "paged" read to get the next 100 items and resumes yielding these on demand; to the consumer it looks like a long continuous IEnumerable.

    In the above network calls to get the next 100 items are only made as needed, so how do we handle this in an Rx world?

    From my (admittedly hurried) reading it seems the Subscribe operation (which runs asynchronously) would just run and pull every thing, perhaps 100 million items yet the consumer may only end up actually looking at say the first 100.

    I'd kind of expect the interface to provide some kind of continue bool or something allowing the consumer to indicate that its done...

    ....UPDATE....

    Studying the documentation for this suggest that - ideally - OnNext could have been defined to return a bool rather than a void. This would enable the Observer to indicate that's it's done processing and thus cause the Observable to cease further data retrieval...

    • Moved by CoolDadTx Friday, November 10, 2017 3:06 PM Rx related
    • Edited by Captain Kernel Saturday, November 11, 2017 1:43 PM
    Friday, November 10, 2017 2:37 PM