none
Answers needed to take decision to Move to streaminsight - Pls answer

    Question

  • 1.  For 2.5M customer with 48 readings/per day, what could be an apprx sizing of the hardware?

    3. Is StreamInsight licenses cost is on top of SQL server license? If yes, how much is StremInsight cost (apprx) per processor?

    4. When StremInsight send data out thru Output Adapaters, does it store a copy of the same? If yes, then is this the SQL server 2008 platform? Then we need huge space for SQL server also.

    5. If some reference data is required for validation, then what is the mechanism for pulling such data from transactional system?

    6. Can we have an architecture to solve the following business requirement

    A) Input message received thru Input Adapter and then gets validated.

    B) Then 2 output streams to be generated.

    C) Output 1: Same message with some structural changes sent to System 1/Folder 1.

    D) Output 2: Input message will be aggregated and then aggregated result sent to System 2/Folder 2.

    Tuesday, May 11, 2010 6:00 AM

Answers

  • Hi,

    I will try to give you some of the answers (and let the MS guys complete it eventually):

     

    1.  For 2.5M customer with 48 readings/per day, what could be an apprx sizing of the hardware?

    R: You should be careful especially on the memory part. You may calculate it based on:
    - which is the size of your payload information (estimate in bytes, by cumulating the sizes of your fields)
    - add some overhead for the event itself. Cannot tell you exactly, but consider that the event itself contains few information, especially time (start/end time) + type/shape/kind info.
    - multiply it by how many events per second you should have in your process
    - multiply then by the time you need to have the events in memory : your events duration (endTime - startTime) + your time slack (the accepted delay for processing delayed events)
    No special disk size requirements as most of the processing is done in memory

    3. Is StreamInsight licenses cost is on top of SQL server license? If yes, how much is StremInsight cost (apprx) per processor?
    R: StreamInsight is delivered with SQL Server 2008 R2. You will buy SQL Server in order to have SI. There are 2 levels of licensing, axed on the SQL Server versions, which give the final performance limits (you have more info on SQL Server site).

    4. When StremInsight send data out thru Output Adapaters, does it store a copy of the same? If yes, then is this the SQL server 2008 platform? Then we need huge space for SQL server also.
    R: No SQL Server required. All processing is done in memory; it's up to you to plug in persistence layers if needed; this may affect performance (speed) eventually

    5. If some reference data is required for validation, then what is the mechanism for pulling such data from transactional system?
    R: You should have:
    - shared library calls (.NET code), or
    - input streams containing your validation info
    Not recommended to call WS or something from SI, it will induce latency

    6. Can we have an architecture to solve the following business requirement

    A) Input message received thru Input Adapter and then gets validated.

    B) Then 2 output streams to be generated.

    C) Output 1: Same message with some structural changes sent to System 1/Folder 1.

    D) Output 2: Input message will be aggregated and then aggregated result sent to System 2/Folder 2.
    R: You should publish your input stream first, and then subscribe with 2 queries which do each of them the job you want, each query having its Output Adapter. See "Composing queries at runtime" topic in the SI manual.

     

    Hope it helped,


    Marius
    Tuesday, May 11, 2010 12:46 PM

All replies

  • Hi,

    I will try to give you some of the answers (and let the MS guys complete it eventually):

     

    1.  For 2.5M customer with 48 readings/per day, what could be an apprx sizing of the hardware?

    R: You should be careful especially on the memory part. You may calculate it based on:
    - which is the size of your payload information (estimate in bytes, by cumulating the sizes of your fields)
    - add some overhead for the event itself. Cannot tell you exactly, but consider that the event itself contains few information, especially time (start/end time) + type/shape/kind info.
    - multiply it by how many events per second you should have in your process
    - multiply then by the time you need to have the events in memory : your events duration (endTime - startTime) + your time slack (the accepted delay for processing delayed events)
    No special disk size requirements as most of the processing is done in memory

    3. Is StreamInsight licenses cost is on top of SQL server license? If yes, how much is StremInsight cost (apprx) per processor?
    R: StreamInsight is delivered with SQL Server 2008 R2. You will buy SQL Server in order to have SI. There are 2 levels of licensing, axed on the SQL Server versions, which give the final performance limits (you have more info on SQL Server site).

    4. When StremInsight send data out thru Output Adapaters, does it store a copy of the same? If yes, then is this the SQL server 2008 platform? Then we need huge space for SQL server also.
    R: No SQL Server required. All processing is done in memory; it's up to you to plug in persistence layers if needed; this may affect performance (speed) eventually

    5. If some reference data is required for validation, then what is the mechanism for pulling such data from transactional system?
    R: You should have:
    - shared library calls (.NET code), or
    - input streams containing your validation info
    Not recommended to call WS or something from SI, it will induce latency

    6. Can we have an architecture to solve the following business requirement

    A) Input message received thru Input Adapter and then gets validated.

    B) Then 2 output streams to be generated.

    C) Output 1: Same message with some structural changes sent to System 1/Folder 1.

    D) Output 2: Input message will be aggregated and then aggregated result sent to System 2/Folder 2.
    R: You should publish your input stream first, and then subscribe with 2 queries which do each of them the job you want, each query having its Output Adapter. See "Composing queries at runtime" topic in the SI manual.

     

    Hope it helped,


    Marius
    Tuesday, May 11, 2010 12:46 PM
  • Some late additions:
    Hi,

    I will try to give you some of the answers (and let the MS guys complete it eventually):

    1.  For 2.5M customer with 48 readings/per day, what could be an apprx sizing of the hardware?

    R: You should be careful especially on the memory part. You may calculate it based on:
    - which is the size of your payload information (estimate in bytes, by cumulating the sizes of your fields)
    - add some overhead for the event itself. Cannot tell you exactly, but consider that the event itself contains few information, especially time (start/end time) + type/shape/kind info.
    - multiply it by how many events per second you should have in your process
    - multiply then by the time you need to have the events in memory : your events duration (endTime - startTime) + your time slack (the accepted delay for processing delayed events)
    No special disk size requirements as most of the processing is done in memory

    RS: Marius already described a number of important aspects. The performance of StreamInsight is bound by the available memory and CPU. Memory requirements depend on the amount of state you need to keep, i.e., the size of the windows in your queries. CPU requirements are determined by the complexity of the query. Your data rate translates into an order of 2000 events/sec, which sounds manageable on a single commodity box. Then again, the data may come bursty, and you may have latency requirements when receiving such bursts.

    3. Is StreamInsight licenses cost is on top of SQL server license? If yes, how much is StremInsight cost (apprx) per processor?
    R: StreamInsight is delivered with SQL Server 2008 R2. You will buy SQL Server in order to have SI. There are 2 levels of licensing, axed on the SQL Server versions, which give the final performance limits (you have more info on SQL Server site).

    4. When StremInsight send data out thru Output Adapaters, does it store a copy of the same? If yes, then is this the SQL server 2008 platform? Then we need huge space for SQL server also.
    R: No SQL Server required. All processing is done in memory; it's up to you to plug in persistence layers if needed; this may affect performance (speed) eventually

    RS: Correct. There is no interaction or dependency between StreamInsight and SQL Server, and the latter is not needed to install the former.

    5. If some reference data is required for validation, then what is the mechanism for pulling such data from transactional system?
    R: You should have:
    - shared library calls (.NET code), or
    - input streams containing your validation info
    Not recommended to call WS or something from SI, it will induce latency

    RS: Marius outlined the two possibilities: pull in the data through a user-defined function or operator, which is a reasonable solution for the occasional annotation of events with 'external' data. Pulling in such data as long-lasting events through another input adapter gives you the possibility to let StreamInsight keep it in memory when it is joined with the fast event stream. Of course, for very large amounts of relational data, this might not be viable anymore.

    6. Can we have an architecture to solve the following business requirement

    A) Input message received thru Input Adapter and then gets validated.

    B) Then 2 output streams to be generated.

    C) Output 1: Same message with some structural changes sent to System 1/Folder 1.

    D) Output 2: Input message will be aggregated and then aggregated result sent to System 2/Folder 2.
    R: You should publish your input stream first, and then subscribe with 2 queries which do each of them the job you want, each query having its Output Adapter. See "Composing queries at runtime" topic in the SI manual.

     

    Hope it helped,


    Marius

    Regards,

    Roman


    MS StreamInsight Team

    Disclaimer: This posting is provided "AS IS" with no warranties, and confers no rights.
    Tuesday, May 18, 2010 2:48 PM
  • Thanks Roman. It helped
    Monday, October 25, 2010 6:49 AM