locked
Parallel Extensions and Streaming Data RRS feed

  • Question

  • Hi,

    let's assume wie have an parallel algorithm which processes __ streaming __ (!!) data on a machine with __ 4 __ (!!) cores.

    [Pseudo Code]

    function (a)
    parallel {
       x = f1(a);
       y = f2(a);
    }
    immediately after f1 and f2 have finished {
       z = f3(x,y)
       print z
    }

    Let's say we have the following time points: t0 < t1 < t2
    At t0 the value V0 receives; at t1 the value V1 and at t2 the value V2.

    Is there a way to use Parallel Extensions to display the whole result (=z) for V0 as soon as possible followed by the result for V1 and then for V2 ? That means that the execution of f3 for x and y resulting from  f1(V0) and f2(V0) should not be delayed as the available cores are busy with other tasks for V1 or V2.

    In .Net 3.5 I would use something like three threads -- one for every funktion (f1, f2, f3). I would let them communicate via queues.

    I think there is not a way to solve this type of parallel scenario using Parallel Extensions as it was primary designed for other scenarios in the field of parallel programming.

    Of course, I could use Futures, but that doesn't guarantee that f3 is instantly called after x and y resulting of f1(V0) and f2(V0) have been executed, as there could be other executing tasks at this point in time and lots of other tasks in the internal queues of Parallel Extensions.

    Am I right?

    Regards,
    Manfred

    Monday, June 1, 2009 10:06 PM

Answers

All replies

  • Hi Manfred,

    It sounds like you want every set of f1-f2-f3-print (f1 and f2 may run in parallel) to finish executing before starting the next set.  Is that correct?  If so, you could schedule a continuation Task for each incoming value that does not begin executing until the previous value's computations have completed.

    Let me know if I'm misunderstanding your scenario.

    Thanks,
    Danny
    Monday, June 1, 2009 11:53 PM
  • Hi Danny,

    no, not exactly. Let's say f1 takes 2 seconds, f2 takes 3 seconds and f3 takes 4 seconds.

    Now I want to have a behavior like this one (t0 is point in time 0):

    t0: V0 comes in and gets processed by f1 and f2
    t0+1: V1 comes in and has to wait, as f1 and f2 ist busy
    t0+2: f1 returns x for V0 and starts processing V1
    t0+3: f2 returns y for V0 and starts processing V1
    t0+3: f3 starts processing x and y calculated of V0
    t0+4: f1 returns x for V1
    t0+5: f2 returns y for V2
    t0+6: f3 returns overall result for V0 and starts processing x and y calculated out of V1

    Regards,
    Manfred
    Tuesday, June 2, 2009 5:59 AM
  • Hi Manfred,

    You might still use Danny's approach but with a custom TaskScheduler. Or you could take messaging approach - as you've mentioned (threads and queues). Thinking of this you might even consider using Axum - it sounds like a good tool for your problem.

    You hava a scaling problem though - your application won't scale with additional cores because of this sequential request.
    Miha Markic [MVP C#] http://blog.rthand.com
    Tuesday, June 2, 2009 6:40 AM
  • Hi,

    thanks for your reply.

    Regards,
    Manfred
    Tuesday, June 2, 2009 9:03 AM
  • Hi Manfred,

    So executions of f1-f2 for later values should not be blocked on f3-print for previous values.  However, f3-print for a value should take priority over later f1-f2 executions.  Sound right?

    If so, I think you can schedule each f1-f2 pair as a normal continuation Task off of the previous f1-f2 Task.  Then, you can schedule the f3-print pairs as continuation Tasks specified with the ExecuteSynchronously option.  ExecuteSynchronously allows a continuation to execute immediately, rather than waiting in the scheduler queues.  This way, f1-f2 Tasks are not held up by f3-print Tasks for previous values, but f3-print Tasks still take precedence over later f1-f2 Tasks.

    Does this sound like it might work for your scenario?

    Thanks,
    Danny

    Tuesday, June 2, 2009 7:50 PM