none
StreamInsight's source and sink for dummies

    Question

  • Hi everyone

    I have to write a paper about Microsoft StreamInsight 2.1 and as a non-developer I have some problems understanding sources and sink.

    I’ve read lots of stuff but it is still hard to understand for me as I’m not familiar with the .NET framework.

    Can someone explain me simply how StreamInsight “plugs” into stream of data from other sources?

    Thank you

    Anne

    Friday, December 27, 2013 2:10 PM

Answers

  • StreamInsight "plugs" into event streams from different sources through the use of sources and sinks. Prior to StreamInsight 2.1, the model was based on input and output adapters.

    Sources are "adapter code" that take the data from a given data source and convert/project it into an event that StreamInsight can work with. Sources come in two types; push and pull. A push-type source is an observable which has data pushed to it from the data source. A source that exposes a WCF endpoint that data sources send requests to is a good example. A pull-type source is an observable which has to pull data in. A source that reads a flat-file line by line would be a good example. Sources implement IObservable<T> (push) or IEnumerable<T> (pull) where T is the data type of the event.

    Sinks are adapter code that converts/projects events from StreamInsight into a form that can be used by the target data source. So if your target data source is a flat file, then the code in your sink will flatten the event into text and write it to the file. If your target data source is a SQL database, then your sink will map the event to whatever SQL command is needed. Sinks implement IObservable<T> where T is the data type of the event.

    Does that help?

    Saturday, December 28, 2013 6:15 PM
  • When you have data that can be "pushed" to the StreamInsight source, you will want to use an IObservable<T> based source. The exact workings of the source are going to vary by data source, but it will implement IObservable<T>.

    When you have data that you have to "pull" or "fetch" to the StreamInsight source, you will want to use an IEnumerable<T> based source. The exact workings of the source are going to vary by data source, but it implement IEnumerable<T>.

    To address your last question, this depends on how your other applications are written/hosted. This is the part where you have to start listing your data sources/applications you want to get data/events from and then work out how to connect them to StreamInsight sources.

    Monday, December 30, 2013 7:17 PM

All replies

  • StreamInsight "plugs" into event streams from different sources through the use of sources and sinks. Prior to StreamInsight 2.1, the model was based on input and output adapters.

    Sources are "adapter code" that take the data from a given data source and convert/project it into an event that StreamInsight can work with. Sources come in two types; push and pull. A push-type source is an observable which has data pushed to it from the data source. A source that exposes a WCF endpoint that data sources send requests to is a good example. A pull-type source is an observable which has to pull data in. A source that reads a flat-file line by line would be a good example. Sources implement IObservable<T> (push) or IEnumerable<T> (pull) where T is the data type of the event.

    Sinks are adapter code that converts/projects events from StreamInsight into a form that can be used by the target data source. So if your target data source is a flat file, then the code in your sink will flatten the event into text and write it to the file. If your target data source is a SQL database, then your sink will map the event to whatever SQL command is needed. Sinks implement IObservable<T> where T is the data type of the event.

    Does that help?

    Saturday, December 28, 2013 6:15 PM
  • Yes thank you very much.

    So if I understand right, when you have an "outside data" that is not possible to use as observable you use a pull ?

    Can you acquire data from other application that are not in c# this way ?

    Thank you

    Monday, December 30, 2013 5:54 PM
  • When you have data that can be "pushed" to the StreamInsight source, you will want to use an IObservable<T> based source. The exact workings of the source are going to vary by data source, but it will implement IObservable<T>.

    When you have data that you have to "pull" or "fetch" to the StreamInsight source, you will want to use an IEnumerable<T> based source. The exact workings of the source are going to vary by data source, but it implement IEnumerable<T>.

    To address your last question, this depends on how your other applications are written/hosted. This is the part where you have to start listing your data sources/applications you want to get data/events from and then work out how to connect them to StreamInsight sources.

    Monday, December 30, 2013 7:17 PM