Merging two streams and finding adjacent events

Question

Hi,

I have two streams: str1 and str2. Both of them contain similar set of data. Now with str1 as the reference stream, I would like to find events from str2 which occur just after each event in str1. For example:

str1:

2012/05/02 10:00:00

2012/05/02 10:10:00

2012/05/02 10:17:00

str2:

2012/05/02 10:00:02

2012/05/02 10:01:00

2012/05/02 10:02:00

2012/05/02 10:10:30

2012/05/02 10:18:00

2012/05/02 10:20:00

required output:

2012/05/02 10:00:02

2012/05/02 10:10:30

2012/05/02 10:18:00

I tried by altering the duration of events of str1 to TimeSpan.Max, but this returns all str2 events which occur after an event in str1. What I want is only the first event of str2 occurring after an str1 event.

Regards,

Answer 1

Here's how I got this working in LinqPad.

1. Extend and clip the reference stream to the data stream. This will convert the reference stream to a signal whose events start at the original event start time while the end time is set to just before the start time of a matching event in the data stream.
2. Now shift the event time of the events in the reference stream by 1 tick and join with the data stream. This will cause the events in the reference stream to overlap with the events in the data stream.

Here's the code:

var referenceStream = from e in str1
		.AlterEventDuration(e=> TimeSpan.MaxValue)
		.ClipEventDuration(str2, (e1, e2) => e1.ItemId == e2.ItemId)
	select e;

var resultStream = from l in referenceStream
		.ShiftEventTime(e => TimeSpan.FromTicks(1))
	join r in str2
	on l.ItemId equals r.ItemId
	select r;

I added the ItemId field in the code sample because that seems to be the pattern we have been using around here. If you don't have a field that serves as the unique identifier for an event, you just have to change your code to something like this:

var referenceStream = from e in str1
		.AlterEventDuration(e=> TimeSpan.MaxValue)
		.ClipEventDuration(str2, (e1, e2) => true)
	select e;

var resultStream = from l in referenceStream
		.ShiftEventTime(e => TimeSpan.FromTicks(1))
	from r in str2
	select r;

Answer 2

To make life a little easier, you can also use the FoldPairs macro that is in the LinqPad samples for StreamInsight. It's under "What's new in StreamInsight 1.2".

---

DevBiker (aka J Sawyer)
Microsoft MVP - Sql Server (StreamInsight)

---

---

If I answered your question, please mark as answer.
If my post was helpful, please mark as helpful.

Answer 3

The FoldPairs macro in LinqPad, as is, only works on 1 stream. If you have 2 seperate streams of data, like in the original post, you'll have to modify the FoldPairs macro to take in both streams.

Answer 4

Thanks, your solution works perfectly.