What to do???? RRS feed

  • Question

  • We have an app that accepts data from onboard computers located in vehicles.  The data arrives via cellphone technology and comes in pieces we call packets. 
    A call can consist of a few to very many packets.  The packets are stored in memory  When we receive 5 packets they are written to a SQL Server database table, a acknowledgement is sent to the onboard computer letting it know that we got the data, and it the OBC deletes those 5 packets from its memory.  We do this for every 5 packets received for the duration of the call. 
    If a call fails before we get the final packet, we retrieve the data that was previously stored in the database when that OBC makes its next call and store the data back into memory.
    Instead of reading and writing to a database all the time, we were wondering if MSMQ might be a solution we could use.  That way if the database goes down or is responding very slowly, we could still receive packets from OBCs.  I have been looking at many articles, but have not found a way where I could retrieve messages from MSMQ that were previously stored for a OBC.  Would it be bad to have many private queues (one per each onboard computer)?  There would be 100,000+ private queues if we did that.

    Any other ideas?
    Wednesday, April 1, 2009 3:46 AM

All replies

  • As I read the first two paragraphs I am thinking....use MSMQ...then I got to the final paragraph and there was the answer.  Yes...use MSMQ.  The added benefit is that you can now scale each platform separately.  The receive, the MSMQ, the queue cleaner, and the database can all be on their own platforms.  Very scalable and very reliable.
    Andrew Siemer
    Wednesday, April 1, 2009 10:56 PM
  • But is there a way to efficiently pull messages out of the queue for a specific OBC when I need them?  I could not find a way to do that without PEAKing at every message.  There could be many messages in the queue for various OBCs.
    Friday, April 3, 2009 2:02 PM
  • Management of 100K queues on MSMQ would be very cumbersome.  For performance reasons, you should not plan on deploying more than a few hundred queues on a server if you are running anything older than WS08.  I do know from personal experience that on WS08/Win7 you can run 10K queues without buying an extraordinary server.  I suspect that given adequate RAM tens of thousands of queues are probably doable, but you should do some performance tests prior to setting any plans for such a solution.

    I don't think Peeking is the right solution either.  You should use MSMQ for transport and buffering and nothing else.  As long as you can write data to the database, read messages from the queue as quickly as you can and write them to the dabase, otherwise you let them pile up in the queue.  Keep in mind that with MSMQ, message also pile up in the outgoing queue on your OBC if there is no connection with the server.  You'll have to design around that. 

    You do not describe a purely asynchronous system.  It sounds like when an OBC loses connection and then reconnects, the server has to detect this and send stored information to it.  How critical is the timeliness of that exchange?  What is the nature of the data?  MSMQ may be a solution for you, but I suspect you will have to rethink some of the processes.  It sounds like you have synchronous processes with a kind of recovery mechanism.  That could be the result of rolling your own transport and recovery, which MSMQ could easily replace, or maybe it's an inherent requirement of your system? 

    Tell us more about you application.

    Saturday, April 4, 2009 2:14 AM
  • I am looking for a solution where I can offload "packets" from the OBC when the database is down.  Packets come is with a start packet, then packet data, and then and end packet.

    If a call fails and that OBC comes back with a call before the database is back online,  I need to reassemble the packets that were stored in MSMQ (or some other location).  If on this call I get the "end" packet from the OBC, I can then move on without having to store anymore packets in MSMQ or the database. 

    Calls from an OBC can come in on any of 40 different servers.
    Saturday, April 4, 2009 1:35 PM
  • I suspect that is the wrong approach if MSMQ is going to be in the picture.  I think we need a higher level picture of what you are attempting to do here because MSMQ should completely replace a substantial portion of the low-level functionality you just described.  Forget about packets for the moment and describe the essential goal(s) and inherent properties of the intended solution.

    What is the nature of the "cell phone technology" you mentioned earlier?  Is it TCP/IP or some other scheme?  What is the capacity of the link in bytes/second?  How long is the average connection?

    If TCP/IP cannot be tunnelled through the cell phones, then MSMQ is not going to help you at all with the communictions between the OBC's and whatever is processing your data and you need not read any further.

    How much data do the OBC's generate per hour?

    Are the OBC's just data sources?  Ignoring the details of your current communications protocols, do the OBC's ever require timely responses from whatever is processing the data they send?

    I suspect that if you have TCP/IP over those cell phones then MSMQ can greatly simplify your current system.  Instead of using MSMQ (or the database) as a shunt for packets, you might consider using it for all of your communications.   MSMQ has a bit less than 4MB per message limit.  Is your data bigger than that?  If not, you can just stuff the data into a message and queue it up for sending.  If it is, you'll have to split it up across multiple messages.  It sounds like you might need to use transactional queues.  MSMQ was never intended to replace a database, you can think of it as a message transport scheme with deep buffers.

    When you queue up a persistent message on the OBC, MSMQ will see that it gets delivered to the destination queue.  You don't have to worry much about broken connections.  MSMQ will retry and it will always recover if a connection can be made.  Depending on the properties of the communications link, you may have to tweak an MSMQ setting or two, but that is highly unlikely.  MSMQ messages have a number of useful properties you can set to tune your system.  If message content is not relevent after some period of time, you will have to set the PROPID_M_TIME_TO_BE_RECEIVED property on the messages.  When such a messages times-out in a transactional queue it will always be deposited in the local transactional dead-letter queue.  You will have to process those if you use transactional queues.

    Because you already have an operational system developed, it may be too expensive to modify it in such a way as to make good use of MSMQ.  But from your description, it sounds like you have implemented a kind of queuing scheme and MSMQ should replace that completely, not get incorporated into it as a replacement for your database.

    Joseph w Donahue
    Wednesday, April 8, 2009 8:06 PM