Console.ReadLine() in Streaminsight

Question

Hi again,

I see in all streaminsight code examoples Console.ReadLine(). I now what this funktion do. But if i remove this command line my streaminsight code execute for a short time and close again. How can i run a whole application without using Console.ReadLine()?

Thank you!

Edit: I need this to measure with Stopwatch, how long the application was executed!

Answer 1

Assuming you're referring to the typical console application sample that looks like the following, the Console.ReadLine() is there to prevent the program from moving past that line of code until the program is ready to shut down.  This could just as easily have been a while (true) { Thread.Sleep(1000); }.

Can you describe your target scenario - do you want to run in a different execution context (such as a windows service)?

With regards to using Stopwatch, what is the measurement you're looking for - how long it took StreamInsight to process a single event, a sequence of events, the end-to-end latency?  Dependending on your target scenario there are a number of diagnostic views that and approaches that can provide more insight.  Using Stopwatch may not provide an appropriate answer (as StreamInsight is performing event processing in a set of background threads, so what is observed on the foreground thread may not be representative).

Answer 2

You see this in my Simple StreamInsight App sample for he reason that Mark highlighted above ... to prevent the application from shutting down. You need to understand that StreamInsight queries do not end until they are stopped; they run continuously against the data that is passing by in the stream. They are not at all like, say, a Sql Server query that is issued and the returned with the results so using a Timer or a Stopwatch to "time" the query won't do you any good if you are trying to determine performance of your query. Instead, without Console.ReadLine(), the application would exit immediately and little, if any, of your query would actually run. Using while(true) {Thread.Sleep(1000);} would, as Mark said, work as well ... but wouldn't give you much of a way to shut down the application gracefully unless you had some way to tell the app that it is time to shut down. Hence ... Console.ReadLine().

---

DevBiker (aka J Sawyer)
My Blog

---

If I answered your question, please mark as answer.
If my post was helpful, please mark as helpful.

Answer 3

Ok i understand. Your both wright.  Streaminsight queries are standing query and will run infinitly long. I see, there is no way around to using Console.ReadLine() or using while(true){Thread.Sleep(1000)} as well. In both variants i have to stop the StreamInsight engine manually.

I will describe the second question. I have to application. The first one was implemented for a long time without StreamInsight with LINQ. I convert this application in a StreamInsight style and now i will measure both application to see, which one have the better performance. In other words: Which one is faster than the other. Is there a possibilty to do that?

Thanks in advance!

 

edit: Thanks for the samples!

Answer 4

You can do a performance comparison between the two but there are a couple of things to keep in mind.

First, your key metrics should be throughput and latency. Throughput is how many events you can pump through the system in given time period (usually events/sec) and latency is how long it takes for an event to make its way through the system. StreamInsight does provide the capability to measure this in performance counters and diagnostic views. Most applications, however, do not so you would need to figure out some way to instrument your non-SI app. Outside of that, you'll also typically want to look at CPU and memory utilization - at least. For example, if both apps have equal throughput and latency, the app that uses less CPU is actually better - it has more headroom to grow. You'll also want to look at a number of other counters while you're doing this, some of which are going to be dependant on your inputs and outputs.

Second, your key limitations in StreamInsight are most likely going to be from your adapters. So ... in doing any kind of performance metric on StreamInsight, it is pretty important to place a lot of focus on optimizing your adapters. Mark Simms has a great blog entry on tuning adapter performance.

Which brings me to number 3 ... you need to make sure that you have a methodology to determine where your bottlenecks are - is it in the input adapter? output adapter? Queries? Because your SI app could well lose IF you have written a bad adapter or some non-optimal queries. This is true of any performance comparison - you need to make sure that you can determine if some of your own code that's in the system is causing poor performance. For example - lots of string concatenation will kill your performance (regardless of the system; it's just expensive) as will neglecting to use I/O completion ports for most disk or network operations.

Beyond your adapters and queries, you also need to look at the source data and the target for your output adapter. For understanding raw StreamInsight throughput, using something that generates random data at high-speed in memory is going to give you a higher throughput than most source systems because there isn't anything else in the way. The same is true on the output side ... StreamInsight will typically flood any type of I/O based output adapter and bring it to its knees, begging and pleading for mercy. This could be network or disk - and keep in mind that an RDBMS is going to be disk-bound for performance. These kinds of external dependencies will often have a far bigger impact on throughput and latency than anything else.

In addition to all of this, make sure that you are taking advantage of everything StreamInsight provides. StreamInsight is designed to process data from high-speed real-time data sources and this can be hard to simulate accurately. Most traditional applications aren't designed to read directly from high-speed data sources but use some source of intermediary. Simply bypassing that intermediary will often result in better throughput and latency and takes full advantage of StreamInsight's full capabilities.

Performance testing and tuning, at the end of it, is more of an art than a science and is never as simple as just doing a quick measurement using a Stopwatch. Keep in mind that your testing should also include time for iterative tuning ... it is VERY rare that a first-pass of your adapters will result in optimal code. IF your test determines that your non-SI app is faster, it is most likely going to be the factors surrounding StreamInsight, not the SI engine. I've gotten over 20K events/second of raw throughput from a i7 dual core with 8 GB of RAM ... CPU turned out to be the limiting factor. On a quad-core i7 with 16GB of RAM, I got almost 45K ... and had CPU cycles to spare.

---

DevBiker (aka J Sawyer)
My Blog

---

If I answered your question, please mark as answer.
If my post was helpful, please mark as helpful.

Answer 5

DevBiker thank you alot. You took the time and wrote a very large text to explain all of this. I will look again to my adapters and after this i will try to measure the latency time and throughput.

I have a simple notebook with core2duo with 4GB of RAM.

Answer 6

Hi again, I look over my adapters and correct them as possible.

On the way i have question about messure the performance. Can i use 2 dummy events, one at the beginning and one at the end, to measure the whole time? Or can anyone help me, how i can measure the execution time of my application? Or how it possible measure the throughput and latency? I have no idea :confused: