StreamInsight is good for the scenario: report me only when my temp sensor is not in control for 30 miutes ?

Question

 

Hi,

I have the following scenarios:

I would like record an Event about Temperature sensor, only if the sensor is in outside the Limits for 30 minutes.

Also, I would like to record an Event when Sensor comes back to Normal state.

How to fit these cases into StreamInsight?

As per my knowledge of StreamInsight, we can't really store the data for 30 minutes and compare the data after 30 minutes and report if there is problems with the data. SI can do the processing at real-time, on the spot. It does not care about, what is the Previous value of the sensor. It can only tell whether current sensor value is with in the Limits or not.

Can someone enlighten me whether SI can fit into these cases or not?

---

Venkat

Answer 1

Yes, it absolutely can. AlterEventDuration will keep events in memory for comparisons like you are referring to. What you do want to be careful about is putting something very large into AlterEventDuration. For example, AlterEventDuration(TimeSpan.FromDays(365)) would not be a good thing in most real-time systems. But 30 minutes ... and in many cases, even a few hours ... is perfectly reasonable. The exact timeframe depends on the amount of memory available and the size of the payload.

Take a look at my blog post on turning alarms into edge events for a start ... in there, the timeout is something like 5 minutes but there would be no problem changing that to 30 minutes. In fact, most real-world alarm scenarios would have a timeout longer than 5 minutes. This query, btw, would give you edge events representing the start and end of the alarm state.

---

DevBiker (aka J Sawyer)
My Blog
My Bike - Concours 14

---

If I answered your question, please mark as answer.
If my post was helpful, please mark as helpful.

Answer 2

Hi DeviBiker,

thanks for reply.

Let me add some more details to my scenario.

I can't fix the AlterEventDuration. Say for example:

I started Monitoring Temp sensor at 10:00 AM. It went outside the Limits at 10:30 AM, and it is in the outside limits for an X hours.

Asume that, it came back to Normal at 4:00 PM, then I should be able to reconize that, it was in outside the limits, and came back to normal at 4:00 PM

and in some cases, the sensor can come back to Normal at 11:00 AM, instead of 4:00 PM. so basically, I can't fix AlterEventDuration.

So whenever I get an event (or scan the sensor) from sensor, I should know what is its previous state and what is its current state.

I hope my scenario is clear. Can this be done with SI?

---

Venkat

Answer 3

I have encountered similar problems for similar issues I have faced in the past. There may be better (more pure-streamInsight ways) but the way I solved it was to put the state tracking into the InputAdapter. It doesn't take up much memory to keep the last state of your "inputs" and the logic is quite simple.

Answer 4

By doing it in your input adapter, however, you miss several opportunities ... for example, to use a StreamInsight reference stream for the threshold or to determine the threshold based on several different variable that may be coming from different input adapters. Take a look at the LinqPad sample below - it seems to be exactly what you are looking for. The results are dumped several times (in stages) so you can see how the queries are impacting the results. The final result, however, is a single edge event that has a start at 4:45 that lasts until 5:45. A "transient" alarm at 4:17 is dropped because the invalid status doesn't continue for more than 30 minutes, even though it is repeated - there is a "Good" reading at 4:30 that effectively cancels the alarm.

void Main()
{
	Func<int, int, DateTime> t = (h, m) => new DateTime(2011, 1, 25, 0, 0, 0).AddHours(h).AddMinutes(m);
	
	var sourceData = new []
	{
		new {Id="A", Status = 0, TimeStamp = t(4,10) },
		new {Id="A", Status = 0, TimeStamp = t(4,15) },
		//False alert @ 4:17
		new {Id="A", Status = 1, TimeStamp = t(4, 17) },
		new {Id="A", Status = 1, TimeStamp = t(4,20 ) },
		new {Id="A", Status = 0, TimeStamp = t(4,30 ) },
		//Real alert @ 4:45 lasts until 5:45
		new {Id="A", Status = 1, TimeStamp = t(4,45) },
		new {Id="A", Status = 1, TimeStamp = t(4,50) },
		new {Id="A", Status = 1, TimeStamp = t(5,00) },
		new {Id="A", Status = 1, TimeStamp = t(5,15) },
		new {Id="A", Status = 1, TimeStamp = t(5,20) },
		new {Id="A", Status = 1, TimeStamp = t(5,25) },
		new {Id="A", Status = 1, TimeStamp = t(5,30) },
		//Back in range
		new {Id="A", Status = 0, TimeStamp = t(5,45) },
		new {Id="A", Status = 0, TimeStamp = t(5,50) },
		new {Id="A", Status = 0, TimeStamp = t(6,00) },
		new {Id="A", Status = 0, TimeStamp = t(6,15) }
	};
	var source = sourceData.ToPointStream(Application, ev => 
		PointEvent.CreateInsert(ev.TimeStamp.ToLocalTime(), ev),
		AdvanceTimeSettings.StrictlyIncreasingStartTime);
	
	var timeout = TimeSpan.FromMinutes(30); 
	
	var validData = source.Where(e => e.Status == 0); 
	var alarmEvents = source.Where(e => e.Status == 1);
	
	var nonTransientAlarms = from alarm in alarmEvents
			 where (from nextevent in source
						.AlterEventLifetime(
							e => e.StartTime.Subtract(timeout),
							e => timeout)
					where nextevent.Status == 0
					select nextevent).IsEmpty()
			 select alarm;
				
	nonTransientAlarms.ToEdgeEnumerable().Dump("Non transient");
	
	
	// Expand all alarm events to the timeout and count over snapshots
	var counts = from win in nonTransientAlarms
				 .AlterEventDuration(e => TimeSpan.MaxValue)
				 .SnapshotWindow(SnapshotWindowOutputPolicy.Clip)
			 select new { count = win.Count() };

	// Those snapshots with a count of 1 belong to the initial alarms.
	// reduce to points and join with original stream.
	var initialAlarm = from c in counts
					.Where(e => e.count == 1)
				from e in nonTransientAlarms
				select e;
	
	initialAlarm.ToEdgeEnumerable().Dump("Initial Alarms"); 

	var final = initialAlarm.AlterEventDuration(ef=>TimeSpan.MaxValue)
				.ClipEventDuration(validData, (e1, e2) => e1.Id == e2.Id);
				
	final.ToEdgeEnumerable().Dump("Final alarms"); 

}

// Define other methods and classes here

The final output from LinqPad is an edge event starting at 4:45 (telling you it's out of range) and ending at 5:45 (telling you that it's back in range). Edge events are, in fact, perfect for these types of use cases because the start gets released immediately to the output adapter and then, sometime later, the end is released. With an interval, the interval isn't released to the output adapter until after its end time has passed.

---

DevBiker (aka J Sawyer)
My Blog
My Bike - Concours 14

---

If I answered your question, please mark as answer.
If my post was helpful, please mark as helpful.