locked
Measuring throughput RRS feed

  • General discussion

  • After doing some work today on pushing a large chunk of data through the
    pipeline it occurred to me that it would be great to be able to measure
    throughput (rows/time). Namely, in 2 places:
    1. Throughput of a data-flow   (admittedly this is available via a
    calculation using a rowcount component and the System::ContainerStartTime
    variable that I know is coming soon)
    2. Throughput of a data-flow component

    #2 would be invaluable for finding bottlenecks in a data-flow which is
    really what I could have done with today.

    My idea for how to accomplish this would be to fire an OnInformation event
    containing the information on completion of the data-flow/data-flow
    component. That's the easy bit I suppose, actually capturing the throughput
    to make it available to the OnInformation event would I guess be the hard
    bit.

    In the case of #1, perhaps this could be done by the TaskHost container???
    Doesnt help with #2 though which is what I really want.

    Any thoughts?


    -Jamie
    Wednesday, June 8, 2005 9:32 PM

All replies

  • Good suggestions Jamie. Number 1 is an interesting problem, and a couple of approaches come to mind. The taskhost one isn't one of them though, since the task host doesn't even really know about the pipeline. As far as it's concerned, the pipeline is just another task. That's by design.
    Number 2 is an interesting approach, and it looks workable too. This is the kind of thing I hope we can take a closer look at in coming versions.
    Did you file a DCR for this? If not, please do, then we can track it and consider it when we're looking at features for V.Next.
    Thanks,
    K
    Thursday, June 9, 2005 5:07 AM
  • No problem  Track ID 764896465
    Thursday, June 9, 2005 7:37 AM