locked
Question re architecture to capture and evaluate events (long post) RRS feed

  • Question

  • I have some questions about how best to architect the following scenario. This is an ASP.Net app written primarily in C#, using IIS, SQL Server and any other MS software necessary.

    Scenario:
    1. Sensors are attached to entities that are moving around the world. These entities could be cargo containers, automobiles, airplanes, ..., etc. Let's assume cargo containers. I don't think it really matters.
    2. Each sensor will have a unique value that identifies it. Call it sensor-id.
    3. These sensors send events to my application. These events are things like location, temperature, humidity, ..., etc.
    4. These events are in the form of XML messages and will arrive to my application via an MSMQ.
    6. Before a container is sent, the sender can optionally specify that he'd like to be alerted (via email) if a sensor reading matches a user defined test. For example sensor.temperature < 30 or > 100.
    7. This alert test can be any expression involving elements of the message, for example, sensor.temperature > 100 or < 30. I'll call this the alerting-expression.
    8. The alerting-expression will not be compiled into the application as the user specifies it using any set of attributes and relational operators from the message.
    9. There is a need for the alert-expression to be able to refer to previous sensor values. For example, an alert test might be alert if sensor.temperature > previous temperature + 2. I.e. the temperture has suddenly jumped 2 degrees.

    Observations:
    1) When a message arrives I have to know:
    a) is this message one whose sensor-id is to be monitored for alerting
    b) if yes, do the attributes of the message, once evaluated by the alerting-expression, require an alert to be sent

    Questions:
    1. How should I associate the sensor-id with the fact (Y|N) that it could be the subject of an alert. I.e., that it is being monitored?
    1.1. Should I create a queue for each sensor-id that has been associated with an alert-expression and route the message from the incoming queue to the sensor-specific queue using an MSMQ trigger (probably not)
    1.2. Should I keep a map in memory that maps sensor-id to a sensor-specific queue and reroute that way
    1.3. Something else

    2. What technology should I use to evaluate the contents of the XML message from the queue against the alert-expressionasociated with that sensor to determine if it requires an alert to be sent.
    2.1. Should I evaluate the message contents using LINQ?
    2.2. Should I evaluate using some other expression evaluator. Probably not Eval() but maybe something that uses a parser-generator or business rules engine.

    3. How should I represent/store the alert-expression in order to minimize the cost/pain associated with evaluating it against the message contents.
    3.1 I guess this probably depends on what will be used to evaluate it (item 2)

    4. If the message didn't arrive via MSMQ but rather via an asyc web service call, how would that influnce the architecture
    Sunday, May 25, 2008 9:18 PM

All replies

  • It sounds like you need some sort of a business rules engine to evaluation values for a particular sensor.

    I would expect that you would extract information from the message and compare it to the business rule conditions.

     

    Are the sensors running some software locally, and can then send notification when a particular condition is met?  You wouldn't want the sensor to keep pinging the system with changes in values as that wouldn't scale well, or at least you should try to avoid that situation if at all possible.

     

    How you implement such a subsystem would require a much deeper knowledge of the requirements.

     

    Technologies such as LINQ could be a possibility, as could expression evaluation technology such as XPath, or Xslt, alhtough those last two would be at the end of my list.

     

    You could possibly have a message routing channel, something to send a message to an active queue if it meets a set of conditions as a possibilty, using some sort of a content based filter.

     

    This is quite a difficult topic to discuss, since we're at such a high level, and the topic is so broad.

     

    There is a starting point, if there is anything else perhaps a little more specific, I will try to answer,

     

    Martin Platt.

    Tuesday, May 27, 2008 7:02 AM
  • The sensors are not running software locally. The exact mechanism of how the messages get into the system is not 100% defined. Most likely via an async web service call. My assumption is that they have ended up on a queue to be processed.

    Unfortunately, the sensors are not programmable to be able to detect an event such as temperature > 100. The sensors send pings with current sensor readings. I agree with your comment that this should be avoided but alas that's not the way it is. It is up to my app to determine what's an interesting message.

    My main concern is now that I've received a message, what is my best approach to efficiently determine if the elements of the message match some criteria. The requirement is somewhat similar to MSMQ's triggers. I'll have a rule that can use relational or boolean operators to determine if the message is a match. The other requirement is that the user has the ability to enter rules via a UI to specify what messages, to them, are interesting and be alerted (email, SMS)

    Here's the use case: Create Trigger (very similar to  to creating triggers via MSMQ)
    1. User goes to UI and enters a rule that uses relational and boolean operators and keywords that represent the elements of the message. For example temperature < 30 OR temperature > 60.
    2. User specifies that an action of send-email is invoked as a response to a message that matches this trigger
    3. System saves the rule definition and action into some internal format

    Now a message enters the system and is evaluated against the trigger and sends an email if there's a match.

    I think (at least for now) the two biggest questions are:
    1. How to efficiently perform the test of the message against the rule. Should I:
    1.1 Store this rule in some format that is then iterpreted against the message contents (sounds unappealing/slow)
    1.1 After the use enters the rule, generate some code for it, compile it and store the compiled code (as a DLL, COM object, ... etc) for later execution against the incoming message
    1.2 Use a business rules engine (sounds unappealing/slow)

    2. How best to associate the sensor reading with information in the database.
    2.1 The sensor is identified by a device-id. In my database i'll maintain a sensor to container and container to shipment relationship.
    2.2 A sensor reading may arrive that is not associated with any trigger. I.e. the shipper has not deemed the shipment worthy of alerting upon so I need to be able to efficiently determine, for a given message (sensor-id), is this a message that I have to test a rule against. The information that answers the "is the sensor-id a candidate to be alerted upon" question is in the database. Certainly I don't want to go into the DB for every sensor reading that enters the system. So maybe I keep an in memory map of sensor-id to yes-no-must-be-tested flag. The downside of this is that it must survive a system outage.

    Wednesday, May 28, 2008 2:04 PM