Simulating collections using multiple streams?

Question

My colleagues and I are attempting to circumvent StreamInsight's lack of support for collections within event payloads by creating several streams and using the guid of the main object to join them. Below is the Object structure we are attempting to use.

public class Event
{
	public Guid EventGuid { get; set; }
	public string EventName { get; set; }
	public List<Attribute> Attributes { get; set; }
	public List<DataSet> DataSets { get; set; }
}

public class Attribute
{
	public Guid AttributeGuid { get; set; }
	public Guid EventGuid { get; set; }
	public string Name { get; set; }
	public string Value { get; set; }
}

public class DataSet
{
	public Guid DataSetGuid { get; set; }
	public Guid EventGuid { get; set; }
	public List<DataPoint> DataPoints { get; set; }
}

public class DataPoint
{
	public Guid DataPointGuid { get; set; }
	public Guid DataSetGuid { get; set; }
	public DateTime Date { get; set; }
	public string Value { get; set; }
}

We are attempting to create a CepStream with each of these object types as the payload, and we think we should be able to join these streams using the appropriate guids. Our question is this:

Is there a way for us to insert into each of these streams using a single input adapter? That is to say, we want to pull an Asset object off an MSMQ, split it into its constituent parts, and enqueue them into the appropriate CepStream. The part we are stuck on is that we want to avoid the overhead or writing input adapters that understand each of our payload types. Is there some way to template the payload? Is this strategy for circumventing the lack of collections even feasible?

Answer 1

You could try encoding it all in xml, create the payload in the adapter as an xml document, maybe something along the lines of the one below. You would then create a UDF to parse the xml payload and project it into the payload structure you show above.

<Event Guid="theGuid" Name="theName">

     <Attributes>

          <Attribute Guid="theGuid" Name="theName" Value="theVallue"/>

          . . .

          <Attribute Guid="theGuid" Name="theName" Value="theVallue"/>

     </Attributes>

</Event>

Answer 2

Unfortunately, there isn't a way that you can have a single input adapter as the input to multiple streams. And yes, the lack of support for collections can make some things ... well ... difficult. The StreamInsight team is well aware of this.

I'm assuming that the entire collection needs to be used for in-stream analysis, correct? The reason I ask is that the DataPoint structure that you describe above has a DateTime .. it would seem, at first blush, that this is your actual data event and the rest is metadata around it, correct? If so, why can't these items be enqueued? How are they coming into the system? Can you order them across events and enqueue them separately? Remember ... StreamInsight time is application time and not necessarily clock time.

Farnham's suggestion ... depending on what you are doing ... may work for you. However, I'd look at using a UDSO rather than a UDF. UDSO's tend to have better performance, less overhead ... and also give you the ability to take your complex package and return it as an array of items. So ... your input adapter could enqueue a big chunk of Xml once and then the UDSO parses it into multiple StreamInsight events for analysis. It's not pretty and I would be concerned about the overhead associated with Xml but if you use LinqToXml, you'll be better off than if you did something heavy like XmlDomDocument.

---

DevBiker (aka J Sawyer)
My Blog
My Bike - Concours 14

---

If I answered your question, please mark as answer.
If my post was helpful, please mark as helpful.

Answer 3

After reading your response, we are revising our approach a bit. We decided that instead of dealing with the overhead of the XML solution, we're going to streamline our event structure into one that doesn't rely on collections. We're flattening the event to represent a single DataPoint, and removing the ability for arbitrarily named attributes, as it doesn't provide us value commensurate with the trouble it creates.

We really appreciate your insight (if you'll pardon the pun) however, and we're sure we'll be using the aforementioned UDSOs in the future for some circumstances where standard LINQ queries aren't sufficient.