Input and output adapters with Query object

Question

Can I have an application whose job is only read the data and insert events into StreamInsight process (outof proc process, for example windows service)?

And another application to connect to StreamInsight and read the data (runs complex queries on StreamInsight to get data)?

Basically I want to decopule the Input and Output adapters, but whatever the examples i saw, the Query object is having both Input and Output adapters and starting the adapters by calling Query.Start()

 

---

Venkat

Answer 1

Ah ... so this is what you are looking for in your previous message!!

So, from what I gather, you want to have some sort of UI that will display the data coming from StreamInsight, correct? In this case, you will need an output adapter that puts the data on the wire in some format (i.e. TCP/UDP/Multicast/WCF Duplex) and connect to that service from your client application.

---

DevBiker (aka J Sawyer)
My Blog
My Bike - Concours 14

---

If I answered your question, please mark as answer.
If my post was helpful, please mark as helpful.

Answer 2

Hi DevBiker,

couple of points:

I wnat to run the StreamInsight as a Windows service, becuase running as Windows service is more reliable than hosting streaminsight in my own host.

So when I installed the StreamInsight i have choosen one option, with that, StreamInsght running as Windows service.

Now, i want to write an applicaiton1(Inputadapter):

which can pull the data from sensors and connect to StreamInisght Windows service and Enqueue the data.

And, I want to have another application2 (Outputadapter), which can keep Dequeing the data from StreamInsight engine.

I don't want to crash the StreamInsght process becuase of the Input and Output adapters (for any issues).

So I want to decopule the Input and output adapters and they don't know each other and they runs and connects to StreamInsight engine independently.

But all examples you have shows that: the Query always links Input and Output adapters and we call: Query.Start() which kick starts the running of Input and Output adapters.

As MS documents shows the StreamInsight design,  i really want decouple the Input and Output adapters. Can you share a sample with me which can demonistrate this scenario?

I hope my question is clear...

---

Venkat

Answer 3

Yes, you were clear. But you still need to use input and output adapters and they need to run in process with StreamInsight. If you expose the management service, you can start, stop and configure both queries and adapters remotely via the management server - Server.Connect() vs. Server.Create(). Both methods are shown in the Simple StreamInsight Application on MSDN Samples. And you can use Server.Connect() to connect to the installed StreamInsight service.

Since the StreamInsight engine processes everything in memory (and in process), there is no other way to get the data dequeued except to have some sort of inter-process communication mechanism ... such as the output adapters (TCP/UDP/WCF) that I mentioned.

All of that said, you don't have to use the StreamInsight service from the installation; there is no reason that you can't create your own service hosting StreamInsight in process and run from there. Finally, it is highly unlikely that your adapters are going to crash StreamInsight - StreamInsight will simply abort the query rather than crashing. I tell you this from experience, my friend. ;-)

---

DevBiker (aka J Sawyer)
My Blog
My Bike - Concours 14

---

If I answered your question, please mark as answer.
If my post was helpful, please mark as helpful.

Answer 4

Hi DevBiker,

you said: " there is no reason that you can't create your own service hosting StreamInsight in process and run from there"

From the MSDN documents it is not clear to me on the relation between Hosting process and StreamInsight engine.

I thought StreamInght is a self running engine, and Input and Output adapters are clients to it. But what you are telling is:

StreamInight is an Inproc (for exmaple dll) and we need to create a host to host it. Am I correct?

So can I create as many hosts as I want with one StreamInsight license?

So can I have more than one StreamInsight engine running (in a seperate host processes) on my box?

Do you have any Deployment diagram for StreamInsight? (like when we deploy a solution built with StreamInsight, how does it look like when the applicaiton is running (or at runtime)? I mean, everything runs in a single process?)

Thanks,

 

---

Venkat

Answer 5

From StreamInsight Server Deployment Models :

Microsoft StreamInsight supports two deployment scenarios for the StreamInsight server:

• Full integration into the StreamInsight application as a hosted (embedded) DLL.
• As a stand-alone server with multiple StreamInsight applications and users sharing the server. In its stand-alone configuration, the StreamInsight server runs in a wrapper such as an executable, or the StreamInsight server could be packaged as a Windows Service.

When deploying as an embedded DLL, it's being hosted in process ... that is, in a process that you write and supply. Since a Windows service is a process that you can write and then host the DLL embedded into that process, you can create your own service hosting StreamInsight in process and run from there.

The StreamInsight engine is a DLL so yes, it's inproc. There is, however, an option to install a pre-built service hosting the StreamInsight server when installing. This service doesn't do anything that you couldn't do in your own service.

Can you have more than one StreamInsight engine running in a single process? Well, you can call Server.Create() multiple times using the same instance - and it appears that they are different instances. You can also call Server.Create() multiple times using different instances that are installed on the same machine. That said, I'm not sure of a good use case for doing that ... you have the Application object to provide isolation without the overhead associated with a full instance.

As for the licensing implications of hosting multiple instances of the same StreamInsight instance on a single machine or having multiple apps on the same machine use the same installed StreamInsight instance, I don't know. I'm pretty sure that each installed StreamInsight instance requires a separate license.

I don't know of any deployment diagrams per se but the diagram on StreamInsight Architecture shows it pretty well. Adapters run in-process with the StreamInsight engine ... always. Regardless of your deployment model. So ... when running using the StreamInsight service included with the installation, you need to make sure that the DLLs for the adapters are present and available to the service by installing them into the GAC or copying them to the service folder.

---

DevBiker (aka J Sawyer)
My Blog
My Bike - Concours 14

---

If I answered your question, please mark as answer.
If my post was helpful, please mark as helpful.

Answer 6

Hi DevBiker,

I think I got what you are saying about inproc and license etc.. If i try to map to my scenario:

I can have StreamInsight runing as windows service and it can expose a WCF endpoint.

I can write as many applications (say for example 2 console applications) as I want and connect to the WCF end point.

For example: One application can get Temperature sensor data, and another applicaiton can get the Gas Sensor data.

Both Temperature sensor Stream application and Gas sensor Stream applicaiton can read the data from sensors (Input adapters)

and Enqueue the events to StreamInsight server (exposed via WCF end point). Similaraly these console applications can (should?) have Output adapters, which can Dequeue the events from StreamInsight.

I also Understand from your disc that: I can't seperate Input and output adapters. Both has to run in the same application (using Query.Start mechanism). Query binds both input and output adapters. Am I correct?

Well, I was going through your blog and got this question:

For example, Temperatue sesnsor can give the same data most of the time. And we are intersted only event when: Temperature > 90 degree c or Temperature < 5 degree c.

So do you expect my application(Temperature sensor cosole app) to identify temperature is beyond limits and Send event to StreamInsight? Or I can simply push all sensor data to StreamInsight and can have Output adapter to detect events which is outside the Temperature limits?

Basically my question is: Do I need to process the data and send only required data to StreamInsight? (this is a big work..)

Or I can simply push all the data read from sensor to StreamInsight and run LINQ queries (using output adapater) to find the data/events I am looking for?

---

Venkat

Answer 7

I can have StreamInsight runing as windows service and it can expose a WCF endpoint.

I can write as many applications (say for example 2 console applications) as I want and connect to the WCF end point.

For example: One application can get Temperature sensor data, and another applicaiton can get the Gas Sensor data.

Both Temperature sensor Stream application and Gas sensor Stream applicaiton can read the data from sensors (Input adapters) and Enqueue the events to StreamInsight server (exposed via WCF end point). Similaraly these console applications can (should?) have Output adapters, which can Dequeue the events from StreamInsight.

That is correct - almost. First, your input adapters would be in-process with the StreamInsight instance so there wouldn't be any need to enqueue via a web service. While you can read from the sensors and post to a separate input adapter (via WCF) on the StreamInsight instance, this will introduce a great deal of latency into the process and limit your throughput. Your input adapters should be reading directly from the sensors and enqueue without anything in between. A general rule of thumb is to minimize the number of steps/processess/hops between the source of the data and StreamInsight. So ... if you are getting your sensor data via, say, OPC, you would want to have an OPC input adapter. The WCF service that you expose would be the management service that is provided by StreamInsight and allows you to manage your running queries remotely. But all of the DLLs for your adapters (input and output) need to be available to the StreamInsight instance.

And yes, with one instance of StreamInsight, you can have 2 applications. A StreamInsight application serves as a container within the single instance ... kinda like a namespace.

I don't know why you are referring to console applications. For a production app, everything should be running as a service. In the solution that my team and I have put together, we use a Windows service to host StreamInsight. We use console apps for testing and to run behind the scenes during demos. And, in fact, our console host app and our windows service app are the same exe but with different command-line switches.

I also Understand from your disc that: I can't seperate Input and output adapters. Both has to run in the same application (using Query.Start mechanism). Query binds both input and output adapters. Am I correct?

Yes, that is correct. One note: an output adapter is optional.

Well, I was going through your blog and got this question:

For example, Temperatue sesnsor can give the same data most of the time. And we are intersted only event when: Temperature > 90 degree c or Temperature < 5 degree c.

So do you expect my application(Temperature sensor cosole app) to identify temperature is beyond limits and Send event to StreamInsight? Or I can simply push all sensor data to StreamInsight and can have Output adapter to detect events which is outside the Temperature limits?

Basically my question is: Do I need to process the data and send only required data to StreamInsight? (this is a big work..)

Or I can simply push all the data read from sensor to StreamInsight and run LINQ queries (using output adapater) to find the data/events I am looking for?

Ummm ... what were you reading on my blog where you got the idea that you should only send limited data to StreamInsight? No ... all data should go to StreamInsight. StreamInsight queries should evaluate for rules and identification of interesting events. You should put as much of this kind of processing into the StreamInsight engine as possible. Input adapters should have only enough logic to get the data from the source and enqueue it into StreamInsight; they shouldn't do any filtering, threshold analysis, event identification or data cleansing - all of that belongs in your query logic. The StreamInsight engine can do this MUCH better than you (or I) can.

I do not expect a Temperature sensor console app to really exist at all except for testing. You should have a temperature sensor input adapter that enqueues directly into StreamInsight.

---

DevBiker (aka J Sawyer)
My Blog
My Bike - Concours 14

---

If I answered your question, please mark as answer.
If my post was helpful, please mark as helpful.

Answer 8

Ok, one important thing I got from your reply is: Input Adpaters should not filter any data. Its job is to read data (periodically, say for every minute) from Source (in my case Temperature Sensor) and create event and Enqueue to StreamInsight. That's the only job of InputAdapater. Whatever logc we want we should build it only on Standing Queries. it is very clear and perfect.

Same applies for Output adapter also. Outpput adapter job is only to Dequeue the event and give to other class (or WCF service) to process it.

I am not sure why you are saying that, I don't need a console app.. let me put my code here:

I have StreamInsight engine running as Windows service: MyStreamInsightEngine

Now I need to write an applicaiton, say Console app, which can connect to StreamInsight engine, and also instantiate the Input and Output adapters and Query.

Here is my applicaiton sample:

static void Main(string[] args)
{
        using (var server = Server.Connect(new EndpointAddress@"http://localhost/StreamInsight/<strong>MyStreamInsightEngine</strong>")))
         {

		var app = server.CreateApplication("TemperatureSensorApp");
		var inputConfig = new TemepratureSensorConfig
                    {
                        SleepTime = 10
                    };

		var inputStream = CepStream<EventPayload>.Create("input", typeof(TemperatureSensorInputAdapterFactory), inputConfig, EventShape.Point);

		var filtered = from i in inputStream
                 where i.Value > 40 or i.Value < 5
                 select i;

		var query =
                    filtered.ToQuery(app, "query", "description",
                        typeof(TemperatureSensorOutputAdapterFactory),
                        new TemperatureSensorOutputAdapterConfig
                        {
                            OutputMessageFormatString = "Received event: {0}"
                        }, EventShape.Point, StreamEventOrder.FullyOrdered);

		Console.WriteLine("Starting Monitoring Temperature Sensor...");
                query.Start();

 		Console.WriteLine("Monitoring Temperature Sensor Started. Press Enter to Stop.");
                Console.ReadLine();
                
                query.Stop();
                Console.WriteLine("Stopped. Press Enter to quit.");
                Console.ReadLine();


	 }

}

Without the above application, how do I use the Input and Output Adapters and Query and Start using the StreamInsightEngine? The Adpaters are just classes/types. Some process need to instantiate them and use it.. right? Or is there a UI, where I can simply give configure these types to MyStreamInsightEngine windows service?

Or you mean: The above code is fine, but run it as service (just by implementing the Servicebase interfaces) instead of a console app ?

Now with the above approach, I can have another Console app (or windows service) with different adapters and Query to connect to the same StreamInsight engine host: MyStreamInsightEngine (this is running as Windows service)

---

Venkat

Answer 9

With the above example, you are connecting to a service hosting StreamInsight - I'm assuming that it's the service that ships with StreamInsight. You don't need to keep it running for the StreamInsight engine to continue processing - just don't call Query.Stop().

What I'm trying to say, though, is that you don't need to use the service that ships with StreamInsight ... you can host the engine in process in your own service. You would use Server.Create() instead of Server.Connect().

In your example, the adapters are created in the service process and need to be available to the service process. And yes, if you have another console app with different adapters and queries, you can connect to the same service/engine and run those queries. You can do this in the context of the same application or a different one.

---

DevBiker (aka J Sawyer)
My Blog
My Bike - Concours 14

---

If I answered your question, please mark as answer.
If my post was helpful, please mark as helpful.