Late-arriving data issue

Question

Our application enqueues asynchronously events from many GPS-trackers, that means we have a deal with late-arriving data (events are not ordered). Theoretically, we could have events with timestamp 2 years ago as well. Using CTIs as a delay of time will unacceptably increase latency. Also we need re-run events from a log using the original timestamp and need to enqueue CTIs with the proper (in the past) timestamps.

As we know, after a CTI event has been issued, no input event can have a timestamp earlier than the timestamp of the CTI event. So, our events violate the StreamInsight semantics.

I am fairly new to StreamInsight and maybe not understanding it.
Any advice would be greatly appreciated.

Answer 1

First, "replaying" events isn't an issue ... that's easy enough to do. Just enqueue with the original timestamp. As long as the query or queries isn't joining or unioning with any queries in a different timeline, you're fine. The timeline in StreamInsight is completely application based and scoped to a particular query or query tree, so you can have multiple, independant queries (that never join/union) with completely disjointed timelines.

As for the late-arriving events, the only way to handle that is with to introduce some amount of latency in the CTIs ... I'm assuming that you are using AdvanceTimeSettings rather than generating the CTIs from the input adapters/sources themselves. Which also means that you need to make a decision on where the drop-off point is ... that is, how far in the past do events need to be for you to just drop them as though they never happened? As you mentioned, input events can't have a timestamp earlier than the last CTI; the CTI moves the application timeline forward. It's just like the way we, as humans, experience the world's timeline ... much as we'd sometimes like to, we can't go back in time either. ;-)

Let me know if this helps or if you have more questions.

---

DevBiker (aka J Sawyer)
Microsoft MVP - Sql Server (StreamInsight)

---

---

If I answered your question, please mark as answer.
If my post was helpful, please mark as helpful.

Answer 2

Thank you for clarification.