locked
Scheduled external process outside workflow RRS feed

  • Question

  • Hi all,

    I am designing a system in which a workflow needs to be intiated after every 8 minutes
    and will be working on the changed data which will be fetched from database during this 8 minute
    period. I don't want to schedule workflow as it will affect my scalability and hence want some outside
    process to take data from database after eery 8 minutes ,awake the workflow and submit data
    to the workflow and go to sleep for another 8 minutes.

    What Technology should be best fit to implement this external process(e.g. windows service,
    batch file with windows service, an exe etc).

    I want to clarify here that i don't want my workflow to run under the external process as workflow will be running under Windows Server Appfabric.

    Cheers
    TicArch

    Monday, April 5, 2010 6:22 AM

All replies

  • You're running sql server?

    Run a job every 8 minutes.

    Dumps whatever your data is out into a file or adds an entry to a message queue.

    Your workflow watches the queue and kicks off when a new entry arrives. 

    I would think ssis could start a workflow and that would be scalable than waiting for an event.  Depends on what your workflow is doing really.

    You might find that ssis could do the whole lot and you don't need any wwf.  Which would likely be more efficient again.

    Depends on how long your piece of string is.

    Monday, April 5, 2010 1:51 PM
  • whether the workflow is part of the main system or is part of an external progress the end result will be the same - that

    a) the database is going to be invoked in order to retrieve the data and

    b) some internal or external progress must process said data. the processing is going to cost you no matter where it is

    there is no free lunch.

    any tech is suitable


    MickyD | http://mickyd.wordpress.com/ Help others by voting my post as 'Helpful' if you think it is so.
    Monday, April 5, 2010 2:10 PM
  • Hi ,

    Thanks to both of you.My answer to your suggestion is as follows:-

    1. Yes i am using sql server 2008

    2. A Job putting changed data in a queue and workflow taking changed data from queue might not be feasible for me as queue works on FIFO so workflow will receive data and process it sequentially.This might hamper the system performance.

    Can we have some mechanism by which workflow
    gets initiated when we get event of new data.All the data will be provided to workflow as a collection in single go and workflow will enumerate the collection in for loop and assigning one  thread for each row of data.This will be processed asynchronously, hence would be faster

    3. My query to you is can we initiate a workflow based on SSIS. If yes, what function of SSIS should be used for this and what activity of workflow should be used to listen to this event sent by SSIS


    4. I am using workflow inside appfabric because by that workflow will be scalable.
       If i host workflow inside windows service, it would be not that scalable.So want workflow to be seperate from windows service but not sure that if i call workflow hosted inside appfabric from windows service, will this whole will act as workflow hosted in windows service as its instance will be created inside windows service only.

    Let me know if i have to clarify my query a bit more

    Cheers

    TicArch

    Monday, April 5, 2010 2:27 PM
  • You can just start a workflow from ssis. 

    http://www.codeguru.com/columns/experts/article.php/c15635__2/

    Like I said, you might be able to obviate WWF entirely.  Depending on what the workflow has to do.  A lot of shops use sql server jobs to drive processes rather than using wwwf at all.  SSIS is very powerful, you can use drag and drop to design a package and you can run pieces of T-SQL, Stored procedures and code.

    You wouldn't need to listen to events coming from ssis if you just started your workflow up from there.  I would guess there might be some complications in security but I have zero experience with appfabric.

    On the queue thing.  You can have the workflow listen for a new entry in the queue,  But if you need things to happen in a particular sequence and scalability is a concern then minimising the number of different servers and application interfaces is going to be more efficient.    Does the order really matter though?  Will the different workflows interact? 

     

     

    Monday, April 5, 2010 5:11 PM
  • Hi Andy,

    Thanks for been so helpful.Have you ever used the sample posted by you.In that sample SSIS activity and SSIS_Service is hosted inside Workflow runtime.

    SSIS_Activity is to take out items from queue for processsing, that is understood. But what is the use of SSIS_service which is hosted inside workflow runtime is not clear to me.Also will i be able to host these in a workflow which is hoste inside appfabric.

    My requirement is like there are 20 records of sales order is added to the database.Workflow will create a seperate thread and workflow instance inside it for each order.If we take out this data from queue, we will end up processing each record (out of 20) sequntially which would be less performing.Can we replace queue by something like BlockingCollection in .Net4.0 which is PUb/Sub implementation of collection.

    Kindly guide

    Cheers

    TicArch

     

    Monday, April 5, 2010 5:47 PM
  • Sorry, Sloppy googling there.  Although you could start up a package from workflow that's probably not what you want to do ;^)

    I would think you can start a workflow from ssis, never done it myself and I must admit I can't find an example easily but pretty much anything you can do in .net code you can do using ssis. 

    If you need precise control over the order things run in then you don't really need a net 4 based collection, you could just write to a table.  You could watch the table with your workflow if you need to decouple the two parts. 

    What do your workflows have to do once a new order arrives? 

    If you have the option to write the workflows using .Net4 then you want to since there are significant changes between 3.5 and 4.

    Have you thought about the maximum number of orders you might get in 8 minutes?

    Monday, April 5, 2010 6:22 PM
  • Okie let me ask you my question in a bit less complicated fashion, as i generally make query too long.

    If i create a windows service/console application, which invokes a workflow .Workflow will start with query to the database for new records by using timestamp, then process that data inside workflow and then enter a wait state for a predefined time say 8 minutes, then wake up again and do database check again and ......

    Do you think this is a professional solution as i am using console application/windows service only once to start the workflow .Thereafter workflow manages the show and no use of windows service/console application is left.

    What do you say.

    One additional query if you find that above solution is feasible.What activity from the visual studio 2010(wf4.0)toolbox should i use to implement "wait" and "check database"

    Cheers

    TicArch 

    Tuesday, April 6, 2010 6:15 PM
  • What's the workflow going to do?

     

    Tuesday, April 6, 2010 7:12 PM
  • It will have a polciy activity which will check N number of rules in different rulesets.

    Based on these rule executed in this policy activity, i will set some global variable which will be checked in following ifelse statement.Thereafter, conditional activities gets fired, and workflow will be again go to wait state for next 8 minutes.

    Wednesday, April 7, 2010 3:57 AM
  • These activities.

    Do they interact with other systems and data or are they to do with data on your sql server?

    Will some power user be rewriting the WWF scripts?

    What's WWF adding?

    Because ssis can do conditional branching, send emails, start processes, sql server jobs can have multiple steps and setting a status flag on the order is more robust than a global variable.

    20 orders every 8 minutes is very low numbers of records to process.  If you had hundreds of thousands then you'd have a problem.

    Wednesday, April 7, 2010 8:44 AM