looking for samples/real life project to update a dashboard from realtime systems

    General discussion

  • Hi,

    I'm looking for options for a new project.

    We have parking systems which logs entry and exit events, these events are logged locally on the device and produces flat files uploaded on an FTP server every 15min today. a process read the files to store the data in SQL Server, and we have a dashboard created on top of the database.

    The project is to move to a more realtime system, each device may access an SQL Server database directly. the devices are using a java application.

    so I'm looking for options to manage the data and produces alerts to update the dashboard in realtime.

    is streaminsight a technology which can be used here?

    is a tool which can be a warper between the Java application and the SQL database? or is it better to use MSMQ or a webservice to do the bridge (and the code behind the bridge will fill the SI system)?

    or is it good enough to directly "connect" SI against the SQL Server database? (so SI will monitor the data loaded in the database directly)

    I'm not sure what's the best way here.

    so if I can have real life samples because I'm not sure if its the right technology for us.

    thanks if you have any relevant links for me!!!

    Monday, July 01, 2013 8:55 PM

All replies

  • When you say "more real time", how close to real time do you need to get?

    Second, what kind of volume are you looking at here? What kind of alerts do you need for the dashboard? What kind of analytics are you looking at doing? How good is the connectivity to the devices in the field? Do you have issues with dropped connections/inconsistent connections/etc?

    Is it a tool that can do this? Absolutely. Is it the right tool? That kinda depends on the answers to the previous questions. As for how to design it, you would (ideally) expose a web service directly from the SI server; this would be your source/input adapter. From there, SI can send directly to SQL Server via a sink/output adapter.

    DevBiker (aka J Sawyer)
    Microsoft MVP - Sql Server (StreamInsight)

    Ruminations of

    If I answered your question, please mark as answer.
    If my post was helpful, please mark as helpful.

    Monday, July 01, 2013 11:50 PM
  • how close to the realtime... well... as close we can, in fact as the current solution has a delay of 15min+ going below 10sec will be very good I think, but we have to confirm this.

    I anticipate the problem of connectivity issues, the Java app (the sender of the events) will have a buffer in case of a connection issue. but the systems are connected through VPN connections.

    the volume is low, I think its only few KB per minute per sender, I don't know how many we have, maybe 200. (all of this as to be confirmed)

    for the dashboard side, we'll have non realtime analysis based on a cube we will create.

    + realtime dashboard side where the number of car in/out, delays between 2 events etc... here 1 of the challenge: our dashboard must works with all tablets, and I'm looking for a tool which can do this for us. if its not real-time updated, an auto refresh must be available. I also anticipate that we'll have to create special real-time monitoring events, its where SI become powerfull, has we can add queries to answers different needs.

    I'm also looking at using MSMQ. this may help us managing a processing queue on the server side, and still generates events to SI. Not sure how I can compare MSMQ and SI here... does SI can replace MSMQ? so the sender is not locked when a message is submitted and the process of the message is an async process...

    Tuesday, July 02, 2013 12:06 AM