Weird ToSignal Behaviour

Question

Let's say I have defined a point stream like this;

var ctiSettings = new AdvanceTimeSettings(new AdvanceTimeGenerationSettings(1, TimeSpan.FromTicks(-1), false), null, AdvanceTimePolicy.Drop);

var streamAllTicks = CepStream<QuoteTick>.Create("AllTicksStream", typeof(QuoteTickAdapterFactory), configBloomberg, EventShape.Point, ctiSettings);
streamAllTicks = streamAllTicks.ToSignal((s, e) => s.Isin == e.Isin && s.Exchange == e.Exchange && s.Provider == e.Provider && s.Type == e.Type);

var queryAllTicks = from t in streamAllTicks select t;
queryAllTicks = queryAllTicks.Distinct();

If I go on and use the queryAllTicks for some joins (inner and outer), the memory usage hits the top.

But if I correct this code like this;

var ctiSettings = new AdvanceTimeSettings(new AdvanceTimeGenerationSettings(1, TimeSpan.FromTicks(-1), false), null, AdvanceTimePolicy.Drop);

var streamAllTicks = CepStream<QuoteTick>.Create("AllTicksStream", typeof(QuoteTickAdapterFactory), configBloomberg, EventShape.Point, ctiSettings);
// streamAllTicks = streamAllTicks.ToSignal((s, e) => s.Isin == e.Isin && s.Exchange == e.Exchange && s.Provider == e.Provider && s.Type == e.Type);

var queryAllTicks = from t in streamAllTicks select t;
queryAllTicks = queryAllTicks.Distinct();

queryAllTicks = queryAllTicks.ToSignal((s, e) => s.Isin == e.Isin && s.Exchange == e.Exchange && s.Provider == e.Provider && s.Type == e.Type);

Then it works pretty smooth.

I don't know if it's by design or a bug. Any ideas?

Answer 1

Why do you need to use Distinct(), especially right after creating the input adapter? Are you getting duplicate events (and they need to be exact duplicates, including temporal)?

Also, once you use ToSignal(), Distinct(), most likely, will not do anything with the stream ... because ToSignal() will eliminate any exact matches as all events are extended and then clipped.

Have you recorded the events and examined how the temporal properties are changing? It may be better to use the trace.cmd rather than the event flow debugger as that will let you record the initial events going into the stream as well.

---

DevBiker (aka J Sawyer)
My Blog
My Bike - Concours 14

---

If I answered your question, please mark as answer.
If my post was helpful, please mark as helpful.

Answer 2

No I didn't record the events but it is also like this even without Distinct(). And yes I sometimes receive duplicate events but I also need those but that's not the problem.

As silly as it sounds, I think using ToSignal alone, after input adapter creation and before any LINQ queries, causes this problem.

I don't know it maybe extends also the CTIs duration or does something with metadata but it's definitely a problem.

Answer 3

ToSignal(), if done correctly, will extend the event duration and then clip to the next event based on your match expression. In your sample, they will be clipped by matching   s.Isin == e.Isin && s.Exchange == e.Exchange && s.Provider == e.Provider && s.Type == e.Type.

It works like this:

Step 1: Extend the event lifetime of the events in the source stream (StreamA). Usually this is TimeSpan.MaxValue but it doesn't have to be ... you can have a timeout for the event. So .. your stream as a lot of events extended to infinity. This gives you StreamB.

Step 2: Clip the event duration of the events in StreamB. You use StreamA for the clip and the clip is done based on your match (above). When you have 2 events that match, the earlier event will be clipped so that its end time is 1 tick before the start time of the later event - so there is no overlap. The FoldPairs macro sample does the same thing ... but then uses ShiftEventTime(TimeSpan.FromTicks(1)) to create a 1 tick overlap between previous and current values.

StreamInsight is very efficient in removing events that will be clipped since there is a match in StreamB. However, as you can imagine, you can get into a scenario where your match expression doesn't match ... and your events get extended into infinity. I have seen (and done) this ... by not getting the ClipEventDuration quite right. You can verify this by recording the events and looking at what is happening each step of the way.

What is it that you are trying to do with your query?

---

DevBiker (aka J Sawyer)
My Blog
My Bike - Concours 14

---

If I answered your question, please mark as answer.
If my post was helpful, please mark as helpful.