Using plug-in architecture with StreamInisght

Question

Hello,

I am working on an application which should be flexible enought to let developpers add rules. Meaning that in the futur, using my framework, they should easily be able to add a query on an event from an input and to an outputAdapter that is already defined or that they can define themselves. With this aim in view, i am thinking about using a plug-in architecture where the developers can add theirs own event/Adapter/query. So i would like to know if there is any constraint by using events or adapters or templateQueries as plug-ins in order to load them dynamically into the application? I think that Adapters or queries are Ok but i have some doubts abouts events...

Thank you for you answer,

Answer 1

I assume that by "events" you are referring to the actual payload that is being delivered along the stream.

I have worked on a dynamically created plugin architecture for my streamInsight app and can say that there is nothing stopping you from doing this: input and output adapters both deployed as plugin assemblies. The standing query is also deployed as a plugin. StreamInsight loads up the query and binds the adapters based on the instantiated plugins at runtime. The Event payload can also be defined dynamically at runtime. I use an xml payload and have UDF's in place to unwrap and wrap the xml content at the start and end of the query respectivelly

I hope this is along the lines of what you have in mind.

Answer 2

Yes, this is exactlly what i meant, event = payload.

Ok thank you for your answer, this is what i had in mind so i am glad to hear someone else did it before me and that it is possible.

Answer 3

Well, i guess you generate the new payload at the end of the query by using something like that( or an UDF wrapping the xml content):

var query = from e in myStream 
            select new MyNewPayload() {...}

At the begining you need a stream "myStream" with a valid payload. Because this stream is generated using the template function:

var myStream  = CepStream<MyNewPayload>.Create(inputUri,...)

, i can't figure out how you create your stream myStream using a dynamic class MyNewPayload at  runtime. How do you do that?

Answer 4

I create the stream using a payload of a fixed type (XmlPayload). The payload has one field, the xml document. The document I use has  single element with an arbitrary number of attributes, each attribute ultimately being cast t0 the "payload field" of the event.

The "casting" happens by way of the UDF and produces a projected payload with fields named according to the names of the attributes in the incoming xml payload. My app creates the actual query using a template like the one shown here, which is resolved at runtime.

@"var {0}= from e in {1} select new {2};";

where {0} is my queryname determined at runtime, {1} is the stream determined at runtime and {2} is the projection of the xml payload as a projected event and involves calling the udf (again, this happens at runtime)

I build up my Query object at runtime and then to a stream like this:

            var newStream = primaryQuery.ToStream<XmlPayload>();

Answer 5

I'm very interested also in the dynamic nature of StreamInsight, in particular the ability to provide a flexible way for users of the application to add/tweak new or existing standing queries at runtime and even control what gets reported out. The problem I have is that while dynamic scripting works well for one liners, and your example above is also a one liner, I don't know how this is meant to work with moderatly complicated queries. For instance any standing query that's even remotely sophisticated will involve a number to steps (or rather queries), possibly a join or a GroupAndApply or all of the above. Is this a step too far for StreamInsight or is there other techniques out there to address this?

Answer 6

It is possible to have dynamic queries specified at runtime. The snippet I showed above is the query I use to unwrap the xml payload and project it onto fields for use by my "user query" (the one defined by the user at runtime).

I use QueryBinder to create my query at runtime. The snippet below is an example of a query I build up to do the following:

1. project the xml payload to one that can be queried by the user query
2. the middle part is the user query that operates on the fields projected out of the xml payload above
3. Wrap the result back into an xml payload for delivery to the output adapters

// Query Prefix - projects attributes in the incoming Xml payload to payload fields for use by the user query
// In this example, there is one input adapter called MySource1, but there can be many; each would be unwrapped 
// as shown for this single adapter 
var TagsSource1 = from e in MySource1 
select new 
{ 
FQN = MyUDFs.GetAttributeValue(e.Xml, "FQN"), 
Value = MyUDFs.GetAttributeValue(e.Xml, "Value"), 
Quality = MyUDFs.GetAttributeValue(e.Xml, "Quality"), 
Timestamp = MyUDFs.GetAttributeValue(e.Xml, "Timestamp") 
}; 
// Query Template - the user can define any query here that properly leverages the 
// fields projected above and is of course well formed linq 
var q = from e in TagsSource1 select e; 
// Query Suffix - wrap it back up as Xml payload for delivery to output adapters 
var definition = from e in q 
select new 
{ 
Xml=MyUDFs.CreateXmlPayload("FQN="+e.FQN+","+"Value="+e.Value+","+"Quality="+e.Quality+","+"Timestamp="+e.Timestamp) 
};

Answer 7

Thanks, i think i understand now how you do it. So i misunderstood, and your payload is not a plug-in which has members like attributs "described" in an xml. The first trick is actually to have exactly the same payload all among the code, right?  The second trick is to have the UDF in the payload to wrap and unwrap user defined members.

Actually i am wondering the same question as CRafman01's. For example i have a more complicated query wich takes at the start 2 differents events (a point and an interval) and makes a join on a window and generates a new event. So i would have something like that:

@"var {0} = from left in {1}
from right in {2}
where {3}
select {4}; ";

and where {3} is the string representation of a static function hard-coded

Isn't too complicated?

Answer 8

The payload is not itself a plugin, but the adapters are and the payload delivered by each adapter is established by itself at runtime, depending on the specific properties setup by the user for the instance of the adapter.

Once the adapters are defined and the query starts, the snippet I showed earlier in the thread goes about unwrapping the xml payload for each adapter and projecting it forward as a payload for consumption by the user query. The user query I showed was a simple pass through query but there is nothing stopping you from putting any query there. If you had two adapters, for instance, your user query could perform unions, joins etc on the fields from both the adapters.

Answer 9

Hi,

Even I am looking for an example on the Plugin based architecture.

For example, user can define query filters through some user Interface, and say Apply, then there should be a Standing query generated, but this has to refer Input and output adapters... and assume that the adapters also pluggable, user can choose these adapters..

so, creating s tream on Input and Out adapters, and creating a Query on the Input adpter stream and running the query.. all these should happen dyanamically... I have gone through the CreateQueryTemplate concept, but still did not get how to achieve the plugin based architecture with StreamInsight.

Can some one share a sample on this? thanks.

---

Venkat