MSMQ Performant Design RRS feed

  • Question

  • Hi,

    I'm designing an application in C# .NET which receives XML via web service and converts into different formats using XSL transformations. I'm really concerned about performance and so I thought to split it up into receiver (Web Service) and converter (Windows Service) modules just to move the conversion load to seperate process and MSMQ as a communication channel between them. I'm visualizing the following few variaty of designs.

    Approach #1
    The receiver web service that pushes the incoming XML into a single queue and the converter framework reads the queue and triggers an event. The converter for each format are modelled as plug-ins and registered to consume the event triggered by the converter framework. The converter plug-in, on consuming the event, transforms using XSL and write it to a file server.

    Approach #2
    The receiver web service that pushes the incoming XML into "n" queues depending on the numbers of fomats desired. The converter for each format are modelled as a seperate process which reads the XML from its dedicated queue and transforms it using XSL and writes to the file server.

    I see pros and cons in both the approaches. The former approach uses multi-threading and the greater maintainability while the latter uses multi-tasking.

    Having performace of the system as my primary concerns, what would be points to ponder here? Is there any other design variance that would be suitable for this suitation? Any ideas/suggestions would be helpful.

    Thanks in advance.


    Wednesday, October 21, 2009 11:17 AM

All replies

  • Greeting

    One Suggestion : Doing the pratical samples will help you understand the hidden issues which you are unaware while you are design the system or module.

    Kindly have a clear picture of what you want to do.

    Suggestions :

    1. Have you studied windows service architecture , before coming up with the approach.
    2. One question ? What happens if the windows service is not working at all, due to unforeseen reasons. Their is a chance of system failure in this case. Where is the backup option if the windows service fails.
    3. When you design the system , if need to have a back up plan when any of the critical modules fails. Here windows service plays a major role.
    4. Have thought of using WCF technology here. The .Net framework that you require is 3.5. using WCF technology communication performance can greatly be enhanced.
    5. Why can you club two approach(receiver and converter) into a single module and the communication between main application and single module take place using WCF Service or a web service.  WebService can decide where you want to sent the data. That mainly depends on your software design. Here we are removing the MSMQ ie communication between the receiver and converter.
    6. You can divide the single module into two sub modules ie converter and  receiver. The web service or WCF service will get the data and the receiver will receive the data and  converter sub module convert the data and sent the data to respective system based on application software architecture.
    7. The system design should  in such a way that , the single unit should be able to process only 3 to 4 XML files. After the current processing of the XML files are done(operation only 2 second - performance grading). An acknowledgment should sent to the system to sent another batch of XML files. By doing so we are not overloading the system with XML files which can lead to system crash. Our objective is to avoid this situation. In the case of errors or exceptions of processing the XML files in the module, the module should log the event of failure and the module should recover itself in the case of exceptions that might occur during the processing of the XML files.  Here we are trying to built a strong module that should be capable to handle all unforeseen problems.  The system should be  capable to handle situations.
    8. The whole module will be acting as a single unit. The sub system converter can communicate with framework (bi- directional) . After the conversion is done , their will be another communication channel which will be sending the data to the file server.
    9. In the case of software problem , the module should be intelligent to intimate the concerned person regarding the problem and the system will back online in no time when the you resolves the issue. 
    10. Proper Event logging of the activities , in the case system crash.
    11. Here the question lies how would  you design the single module with required functionality.
    12. Here three communication channel the single unit receiving the data , Second one Communication with framework (Bi- directional) , Third is communication of sending the final data to the file server.
    13. In the case of single module failure , there should be a backup module should be capable of take care of operation , in case failure. Concerned person should be informed regarding the failure(automated process done by the module - intelligent module ). You can rectify the problem. Back up module is only safety measure that the application will be  running 24/7 without any problems.
    14. In the case of multi threading , be very sure that you avoid dead lock other wise application will go hay wire.
    15. Make your system and the module more intelligent , by good design. Try to  a optimized the code for better performance.
    Hope this helps you out.

    Take Care


    Helping People To Solve Technical Problems
    • Proposed as answer by PROGRAMMERLIVE Wednesday, October 28, 2009 5:28 AM
    Sunday, October 25, 2009 3:06 PM
  • Thanks a ton PL. This gives a clear insight of factors that needs to be considered while designing the system.
    Tuesday, October 27, 2009 6:47 AM