by hand latency evaluation

Question

Hello,

I built a cont query processing application using the in-process StreamInsight deployment model. The application is responsible for both submitting queries to the server and evaluating query latency. I have two questions:

1. I want to evaluate latency "by hand", so right before enqueueing an event, I set its StartTime as DateTime.Now, and at dequeueing I get the DateTime.Now and subtract from it the event StartTime. Then based on the total number of processed events, I can compute average latency. One weird thing that happens is that each event's StartTime is decreased during processing with one hour (e.g., event e before enqueue point has a StartTime = 9pm, and right after dequeue its StartTime is changed to 8pm). If I'm missing why that happens, I think I might it's possible that I'm missing smth else as well. Do you see any issues with such an approach?

2. Does the in-process model affect the performance of the server? 

Answer 1

1. For determining latency, I would use the Performance Counters for StreamInsight. Have you seen Monitoring the StreamInsight Server and Queries and Monitoring StreamInsight Performance Counters and Events on MSDN? As far as your StartTimes go, they should be of type DateTimeOffset. Try DateTimeOffset.Now.

2. Not that I know of. It depends more on what your adapter/query logic does as well as what else is consuming resources on the machine you are running it on. You should also take into consideration the differences between the 2 editions of StreamInsight. More information is available here: Choosing a StreamInsight Edition.

Answer 2

1. The timestamps can be modified by your query. Things like AlterEventLifetime will change the start time. Also, if you are doing any aggregate windows, their start times won't match the start times on your events. That said, as Tony pointed out, you should use DateTimeOffset, not DateTime. It could be that what you are seeing is due to timezone settings.

2. Not at all. It's just a matter of how you want to work with the server. Using the installed StreamInsight service is actually hosting the very same runtime that you host using the in-process model. And you can, in fact, host in-process, expose the management service and then connect to your service - just as if you were running the StreamInsight service.

---

DevBiker (aka J Sawyer)
Microsoft MVP - Sql Server (StreamInsight)

---

---

If I answered your question, please mark as answer.
If my post was helpful, please mark as helpful.

Answer 3

1. I used Performance Counters as well, I was just wondering whether there are any drawbacks that I fail to see in my approach. I switched to DateTimeOffset and still get the same behavior. Will dig further.

2. I installed the premium edition, so performance should be as good as it can be.

Thanks

Answer 4

1. I tested with a simple "from event in inpustream select event", changed to DateTimeOffset and still get the same behavior.

2. Ok, if it's the same runtime for the service. It's a resource intensive application, and I wasn't sure whether this affects the in-process server.

Thanks 

Answer 5

1. That's extremely odd. I just tested the same thing using the code from my blog samples and didn't see the same behavior. Question ... what time zone are you currently in? Specifically, what's your GMT offset?

---

DevBiker (aka J Sawyer)
Microsoft MVP - Sql Server (StreamInsight)

---

---

If I answered your question, please mark as answer.
If my post was helpful, please mark as helpful.

Answer 6

I tested with +1, +2, etc. Given event e, and the code:

//in the ConsumeEvents of the input adapter, right before the call to Enqueue:

e.StartTime = DateTimeOffset.Now;

           

//in the ConsumeEvents of the output adapter, right after the call to Dequeue:

DateTimeOffset dequeueTime = DateTimeOffset.Now;

,

for a GMT+x offset, the StartTime of e is decreased with x hours.

Answer 7

Just a thought ... try using UtcNow rather than Now.

Although ... I added code into the samples from my blog that does the exact thing that you are doing here. So ... it's a bit puzzling ...

---

DevBiker (aka J Sawyer)
Microsoft MVP - Sql Server (StreamInsight)

---

---

If I answered your question, please mark as answer.
If my post was helpful, please mark as helpful.

Answer 8

Cool, it works.

Thanks.

Answer 9

I had some time to look more carefully into this. I logged and compared StartTime for events right before enqueuing and dequeueing on and from the server. The StartTime and dequeueTime are set as DateTimeOffset.Now (code snippet is in my previous comment)

For a GMT+1 setting, I got values like the following for one event:

logged from the input adapter right before enqueue

StartTime = 3/4/2013 2:10:33 PM +01:00

LocalDateTime = 3/4/2013 2:10:33 PM

Offset = 01:00:00

TimeOfDay = 14:10:33.2228047

UtcNow = 3/4/2013 1:10:33 PM

logged from the output adapter, for the same event, right after dequeue

StartTime = 3/4/2013 1:10:33 PM +00:00

LocalDateTime = 3/4/2013 2:10:33 PM

Offset - 00:00:00

TimeOfDay = 13:10:33.2228047

UtcNow = 3/4/2013 1:10:33 PM

DequeueEventTime = 3/4/2013 2:10:33 PM +01

It looks the StartTime takes into account the offset before enqueueing the event, but not after dequeueing it. LocalDateTime and UtcNow don't change (not a surprise, their names are selfexplanatory), whereas TimeOfDay changes (as expected, since it\s the time of the actual DateTimeOffset object, which has changed). Interestingly enough, the dequeue time does consider the Offset.

I checked the log and this happens for all the events that go through the server. Like is has been pointed out earlier, UtcNow does not change and it works well for me. But I'd like to understand why other fields from the StartTime change. 

It's an in-process server and a a simple select from query that doesn't alter events.

Thanks