locked
What is the recommended WF scaling solution for the "Job/Batch" servers vs web server? RRS feed

  • Question

  • I am confused regarding the scaling story for doing batch processing using workflows.  For instance, I want to allow users to upload documents that kick off a workflow that processes the document by converting it and burning a file stamp into the actual document and then sending it off to some external system.  The user doesn't need to wait on the entire process completing and doing all of that work within the actual web request obviously wouldn't scale very well.

    I essentially want to have one large workflow that describes the entire process, but after the document has completed uploading, I want to send a response to the user and then continue with the batch processing on another machine.  This makes sense correct?  You definitely don't want the application pool that is handling web requests to also be processing the long running batch processes.

    I know I could just put a delay activity after the send response activity and have the workflow aggressively unloaded.  Then the WorkflowManagementService could pick it up on another machine when the delay expires.  This seems hacky though.  The other solution I see would be an activity that creates a bookmark and allows the workflow to unload.  A custom windows service could then watch for these and resume the bookmarks on that machine.  That requires building in all the robustness and scalability that the WMS already has.

    Am I completely missing something here?

    Thursday, January 6, 2011 8:10 PM

Answers

  • Create two WorkflowServices

    One for the user to submit the job

    Another to process the job (runs on different machine/farm)

    User submits job, workflow does initial processing and sends response then submits job to the second workflow service which actually processes the work.

    Another option would be for the second workflow to be hosted in a Windows Service and have it poll a database for jobs to run


    http://blogs.msdn.com/rjacobs
    • Marked as answer by Andrew_Zhu Friday, January 14, 2011 8:47 AM
    Friday, January 7, 2011 1:35 AM
  • Hi David,

    In additional to Ron's reply, the business process can be encapsulated into small business oriented processes (WorkflowServices) such as manager and workers driven by events. This is a push event model, where each business process will create an event for next process. The Manager has a responsibility for creating an event for worker respectively for more workers based on the business requirements. On the other hand, the worker represents an executive unit (mediator) for specific business transformation driven by event message.

    The Manager can be designed as a state machine to organize a root business process logical decomposed into small worker jobs. Note, that the business process is decomposed via MSMQ event messaging.

    For business composition (forwarding events between the workers and/or manager) can be used WCF Routing Service to enable scaling a business process across a cluster.

    The following picture shows a composition of the business process via Router, more details can be found in my article Routing Manager for WCF4

     

     

    Thanks

    Roman

     


    Roman Kiss, MVP Connected System Developer
    • Marked as answer by Andrew_Zhu Friday, January 14, 2011 8:47 AM
    Saturday, January 8, 2011 1:41 PM

All replies

  • Create two WorkflowServices

    One for the user to submit the job

    Another to process the job (runs on different machine/farm)

    User submits job, workflow does initial processing and sends response then submits job to the second workflow service which actually processes the work.

    Another option would be for the second workflow to be hosted in a Windows Service and have it poll a database for jobs to run


    http://blogs.msdn.com/rjacobs
    • Marked as answer by Andrew_Zhu Friday, January 14, 2011 8:47 AM
    Friday, January 7, 2011 1:35 AM
  • Hi! you can host your workflow as a WCF Service in IIS.. you can host it in another application pool or even a different server if you like.. Inside the workflow, you receive the message from the client, send a response to the client and keep executing the workflow... then you can have more than 1 host and you can use WCF Routing Services to do load balancing... I hope it helps.. cheers from argentina! ps. sorry my english sucks!
    Friday, January 7, 2011 5:20 PM
  • Hi David,

    In additional to Ron's reply, the business process can be encapsulated into small business oriented processes (WorkflowServices) such as manager and workers driven by events. This is a push event model, where each business process will create an event for next process. The Manager has a responsibility for creating an event for worker respectively for more workers based on the business requirements. On the other hand, the worker represents an executive unit (mediator) for specific business transformation driven by event message.

    The Manager can be designed as a state machine to organize a root business process logical decomposed into small worker jobs. Note, that the business process is decomposed via MSMQ event messaging.

    For business composition (forwarding events between the workers and/or manager) can be used WCF Routing Service to enable scaling a business process across a cluster.

    The following picture shows a composition of the business process via Router, more details can be found in my article Routing Manager for WCF4

     

     

    Thanks

    Roman

     


    Roman Kiss, MVP Connected System Developer
    • Marked as answer by Andrew_Zhu Friday, January 14, 2011 8:47 AM
    Saturday, January 8, 2011 1:41 PM