locked
Scaling Up by using many BizTalk Groups RRS feed

  • Question

  • Is it a vaible option of not?

    There is no requirement for high reliability. Failed messages can be resubmited by the source party.

    BizTalk scales badly, so the idea is to multiply the simple BizTalk environments in the independent "pipelines". Load Balancing can be implemented on the IIS (or something similar) because the inbound messages are delivered with SOAP / REST / HTTP.

    What are the issues with such architecture?


    Leonid Ganeline [BizTalk MVP] BizTalk and 0MQ (ZeroMQ)
    Monday, January 30, 2012 7:27 PM
    Moderator

All replies

  • Some things to think about:

    - Tracking: your tracking data will be in different databases => harder to find someting (don't know whether tracking is a requirement)
    - Deployment: do you need to deploy the same artifacts on multiple groups?
    - Maintenance, different groups will make it slightly harder (more databases, more SSO masters, ...)

    There is nothing wrong if it suits your needs :)

     


    If this post answers your quenstion, please mark it as such. If this post is helpful, click 'Vote as helpful'.
    Monday, January 30, 2012 8:15 PM
  • To decide upon scaling , we first need to figure out the bottle neck within the landscape.

    depending on that we can scale up the biztalk landscape.

    IIS can be load balanced so that all the incoming traffic is routed to the servers attached to the load balancer instead of single server.

    You can look at option of increasing the number of biztalk servers instaed of a different group with different DB.

    If DB is the bottleneck then the approach mentioned by you would be viable.

     

     

    Tuesday, January 31, 2012 4:00 AM
  • Hi Leonid,

    Can you please let us know more details about the NFRs. So that this would be a interesting discussion.

    If we get an idea about system architecture, volume of the transactions & message size then it would be easier to start the analysis. 


    Thanks With Regards,
    Shailesh Kawade
    MCTS BizTalk Server
    Please Mark This As Answer If This Helps You.
    http://shaileshbiztalk.blogspot.com/
    Tuesday, January 31, 2012 4:44 AM
  • Hi guys,

    thank you for discussion!

    Requirements are:

    • hundreds messages per sec
    • several K per message

    Pretty much Pub-Sub, i.e. one way data flow.

    High-availability below load-balancer. (Idealy several identical services)

    Reliability: several services could process the same message, it is OK, because DB under these services tolerates duplicates.

    I think about several independed identical boxes; BizTalk+SQL Server(MessageBox) in one box, all Standard licenses. No clusters.

    Scalability should be near linear. In Dev VM the performance is 20-50 messages/sec. I predict >100 m/s on 4 core i5 box. In 2-3 years the performance should be 500-1000 m/s, then more. All this scalability should be O(N) not O(N*N) regarding price.


    Leonid Ganeline [BizTalk MVP] BizTalk: Advanced Questions
    Tuesday, January 31, 2012 5:17 AM
    Moderator
  • Hi Leonid,

    As per my experience the volume and the message payload that you are talking about can be achieved with single group but yes the configuration of the servers must be high. I am sure you already know what i am talkign about.

    Linearity in the scalability is definitely questionable though.

    I have couple of other pointers on this.

    1) Is it possible for you to setup the Non Functional Environment for this ?

    2) Use single group and scale out the biz in-proc servers within the same. Keep load testing your application. Make sure you test same scenario multiple times.) My experience in these type of testing says that even for same scenario the results might be different drastically. So we have to use the average of the multiple results. I am sure i am saying things which you are already aware :).  

    3) If the results are not as per your expectation (and linear in nature as per your needs) then you can still go for another group parallely working along with first single scaled out group.

    4) Another pointer that i see is if we go for single group scale out and it solves your purpose you will save the cost of additional SQL server liceses (which you might need in multiple group scenario).

    I hope these pointers would be useful for you.


    Thanks With Regards,
    Shailesh Kawade
    MCTS BizTalk Server
    Please Mark This As Answer If This Helps You.
    http://shaileshbiztalk.blogspot.com/
    Tuesday, January 31, 2012 7:25 AM
  • Hi Shailesh,

    Thank you for discussion.

    My shame, I don't know what is it "the Non Functional Environment". :(

    My goal is to go out of the single group limits. It scales by specializing the architecture and far away from linear by performance and cost. I cannot just "add another copy" and get the additional perfomance.

    Not sure that "adding one more identical copy" approuch will add more configuration cost. My idea is quite opposite. Add one more box (VM) with Standard licenses and get additional performance power. (I know the BizTalk problem with cloning, all this hassle with hard-coded computer name.)


    Leonid Ganeline [BizTalk MVP] BizTalk: Advanced Questions
    Tuesday, January 31, 2012 4:29 PM
    Moderator
  • inbound messages are delivered with SOAP / REST / HTTP

    Pretty much Pub-Sub, i.e. one way data flow.

    High-availability below load-balancer. (Idealy several identical services)

    Reliability: several services could process the same message, it is OK, because DB under these services tolerates duplicates.

     

    So what do you think of using Windows Server AppFabric in this scenario. I think it would help you in achieving high throughput by skipping the message box.


    Please mark the post answered your question as answer, and mark other helpful posts as helpful, it'll help other users who are visiting your thread for the similar problem, Regards -Rohit Sharma (http://rohitbiztalk.blogspot.com)
    Wednesday, February 1, 2012 4:17 AM
    Moderator
  • inbound messages are delivered with SOAP / REST / HTTP

    Pretty much Pub-Sub, i.e. one way data flow.

    High-availability below load-balancer. (Ideally several identical services)

    Reliability: several services could process the same message, it is OK, because DB under these services tolerates duplicates.

     

    So what do you think of using Windows Server AppFabric in this scenario. I think it would help you in achieving high throughput by skipping the message box.


    Please mark the post answered your question as answer, and mark other helpful posts as helpful, it'll help other users who are visiting your thread for the similar problem, Regards -Rohit Sharma (http://rohitbiztalk.blogspot.com)

    Hi Rohit,

    Yes, I'm thinking of something like this now.

    It will shift the development and deployment to different platform. It is not a great difference, as it looks like. Of course it is shift from visual editors to the code. Then, if I'm on this way, time to think about "more code" approach. And about FOSS (Rabitt, 0MQ, Spring Integration, etc.)

    Funny, but the very low-level code with ZeroMQ seems to manage all my requirements. Pieter Hintjens said after showing a small cut of such code "...people have charged houses for code that did less..."  , I  LOL then... agree. A big wave of integration FOSS is on horizon.


    Leonid Ganeline [BizTalk MVP] BizTalk: Advanced Questions: have fun - test your knowledge
    Wednesday, February 1, 2012 5:17 AM
    Moderator
  • One more option to scale up the BizTalk application is to use the Branch edition. It works for one application but it is OK in my case

    Leonid Ganeline [BizTalk MVP] BizTalk: the Naming Conventions in Examples

    Sunday, March 3, 2013 2:17 AM
    Moderator
  • IMHO, multiple BizTalk Groups would be a nightmare to manage.

    If the message submission if through SOAP/REST/HTTP those are functions which an IIS/Web Server does in an NLB mode and can sustain (even better the requirement of hundred message/sec). Instead of trying to implement Services in BizTalk and exposing them to handle this, I would focus on achieving a scenario where the submission is decoupled from BizTalk. Clearly the message delivery/reliability is NOT in question here as you stated that messages can be resubmitted. So multiple IIS Server load balanced would achieve your message rate. Where in the scenario would you leverage all the higher functions such as transformation, routing, multiple end-points and such?

    Regards. 

    Sunday, March 3, 2013 9:33 AM
  • IMHO, multiple BizTalk Groups would be a nightmare to manage.

    ...

    Agreed.

    The completely independent BizTalk servers working in parallel with the load balancing on the inbound might work fine.


    Leonid Ganeline [BizTalk MVP] BizTalk: the Naming Conventions in Examples

    Sunday, March 3, 2013 7:34 PM
    Moderator