Scaling Out MSMQ RRS feed

  • Question


    I've been thinking about re-architecting an existing, large scale MSMQ implementation to allow for easier scaling out of our middle tier.


    The basic gist of the system is that the front end web services generate lots of MSMQ messages and send them a middle tier server. This middle tier processes these messages (DB stuff, call web services, etc.), and eventually removes them from the queue when its done.


    We currently utilize MSMQ Triggers to pass these messages to a COM+ hosted component written in C#. We use the PEEK option for the triggers. In other words, the message isn't considered received until we have finished processing it.


    The big limitation with the current system is that we have no way to scale out the middle tier. Because these messages are all transactional/persistant, we can't use any load balancing strategy (NLB, F5, etc.) because MSMQ requires a persistant machine-to-machine session. Packet based load balancing strategies won't work.


    The new architecture that I'm considering is to do something similar to how the Stock Trader app (from MS) does load balancing. I'll have the DB hold a routing table that contains many valid queue paths for the target of the messages. I'll (at first at least) use a random or round robin technique to send messages to the target machines.


    I'll make sure the machines are "live" periodically (and if they're not, I'll disable that path in the routing table so that future messages won't be sent there), and utilize the dead letter functionality to ensure that messages that are sent to an unavailable machine will eventually timeout and be resent to a valid target.


    Obiviously, I'm glossing over a lot of details... but I'd love some feedback on this design so far.

    Saturday, October 20, 2007 2:25 AM

All replies

  • Hi RMD!


    I have nearly the same problem. On one hand I want to ensure reliability by using transactional/persistant queues, on the other hand I do want a better scaling for my middle tier.


    I've also thought about your solution using multiple target queues, but was not so keen on doing the re-inserting of messages into another queue if they are not received within a certain time limit.


    Maybe it would be best if microsoft would add the load balancing feature to the MSMQ product... but then I cannot wait until they do...


    A kind of hoped that you could use the multiple target queue option in a message to tell MSMQ to deliver to only one of multiple queues mentioned, but that is no option in MSMQ.


    At present I'd like to implement some sort of routing servers which are load-balanced by hardware as the first connection point for the webserver farm. From there the messages are routed to different servers offering services (not all servers offer the same services in my installation). The routing servers are checking which servers are online and are responsible for resending messages if a server goes offline. The server offering services can inform the routing servers that they will go offline (for maintenance, update, etc.).


    If you come up with any usable solution, I'd love to hear about it!!!

    Saturday, October 20, 2007 8:25 AM
  • I've solved this issue with NServiceBus. What I've implemented is a message distributor which transactionally transfers messages to "worker" servers as they notify about their ability to handle messages.


    You still have to take into account the "race condition" of a worker coming down after the message was transactionally transferred to it and thus no response returning. This is done through the use of "workflows" and "persistent reminders".


    Take a look and tell me what you think.


    Saturday, October 20, 2007 11:09 PM