Workflow Persistence, Idle Workflows and External Events

Question

I've been doing some thinking about how the default workflow persistence services work and how it affects those of us writing complex event-activities (i.e. IEventActivity implementations), and I was curious about something.

I was reading Paul Andrew's comment on how the default workflow persistence service will only load persisted workflows that are "ready to run", and how it worked with locking when multiple runtime instances across a farm were using the same persistence database (all very informative stuff). However, what I'm not quite clear on yet is what the "readiness to run" of a persisted workflow is determined. Could someone expand on this a little bit?

Here's what I'm trying to find out: For example, if an activity had been idled and unloaded because, say, it was waiting for a delay activity to fire and the delay time is close to being done, that surely that would seem like a candidate for "ready to run" status. But what about an activity waiting on an external event, such as one using the HandleExternalEvent activity to wait for an external component to fire an event, or a custom IEventActivity implementation waiting on something completely different? Obviously, the workflow runtime (or really, the persistence service) would have no way of knowing what this external event means and when it might fire.

So what happen on those cases? Is the workflow instance considered to always be "ready to run" and loaded anyway? Or perhaps are just the workflow queues for the workflow recreated and the workflow just reloaded/executed when the event arrives at said queue?

Basically I'm trying to understand how the persistence services interact with custom runtime services that are associated to custom activities implementing IEventActivity in event-driven scenarios (if indeed there is any interaction at all!). I think this is a topic where not much has been said, and one where there are a lot of "obscure" details yet because of that.
 I've posted some thoughts about what I've noticed about this in a recent blog entry, but so far have not received any comments on it, so no idea how offbase I'm here :)

I would sure appreciate any clarifications on this topic...

Answer 1

From what I can tell from looking at the tables and stored procs - here is the scenario:

Imagine a host running workflows.  These workflows are not idled but running.  Imagine the host dies.  When the host is restarted  - the workflows are "running" from the POV of the persistence database - since they are in the executing state when they last persisted.

As to the queue issue - queue information is persisted with a workflow instance - so when the workflow de-persists - the queue is recreated.

Answer 2

Hi Jon,

Thanks for the info. WRT the queue issue, that's the behavior I was expecting, so at least I wasn't that far off on what I was thinking.

 So let me tweak the question a little ;)
What makes a workflow go from the "running" to the "idle" state? Obviously, I guess if the workflow is sitting there waiting for a delay activity to fire with more than X time that would be a candidate, but what about other kinds of wait? Does a workflow instance waiting for a custom event activity to fire become idle?

Answer 3

Tomas - a workflow becomes idle when:

a) There is one or more exeucting activities (i.e. an activity that returns executing from its Execute method - which Delay and HandleExternalEvent do)

b) there are no more activities currrently schedule to execute.

So if there is a parallel with three branches - all with a HandleExternalEvent - the idle doesn't happen until all three HEE get to execute and return executing from the Execute method. 

At least - that is the behavior I have observed.

Answer 4

Hi Jon,

That's interesting, thanks. It's somewhat unintuitive that having an schedule in the "executing" state means the workflow can become idled, but such is life :)

The parallel task issue is interesting as well, and it makes a lot of sense. How about a listen activity instead?

Answer 5

I think it kind of does make sense - Idled is essentially the workflow saying "I don't have any more to do right now - just sitting here waiting for a message".

I believe the Listen activity would behave the same way

Answer 6

Jon,

Humm, that wouldn't make sense. If it behaved the same way, then the workflow could never become idle once you reached a ListenActivity, since only *one* of the event activities in the Listen branches will get their Execute() method called (the one whose event fired off first). So either the workflow becomes idle after the Listen activity has subscribed to the event activities in it's branches but before the event is triggered, or it doesn't become idle at all.

At least that seems to be my understanding of how the Listen activity works, I might be wrong!

Answer 7

Tomas - if listen didn't work that way - how could you have three HEE activities as the first activity in each listen branch?  Each of them has to have their Subscribe method called - so that they can setup the queues to listen on.  Once one of the messages comes in - the listen finishes.  That is its only different with parallel - parallel lets all the sequences finish - but all of the start - otherwise you coudl create the queue to listen on.

Answer 8

I guess the other difference between parallel and listen - is that listen requires each child of the eventdriven to implement IEventActivity - whereas parallel doesn't.  So the children of listen only have the execute method called if the message comes in.  The listen activity itself is the one that returns executing in this case and would cause the Idle.

Answer 9

Jon: OK, so we're saying the same thing: the workflow can become idle after the ListenActivity subscribes to the child activities, then it returns the Executing Status and all is well. Cool.

So, allow me to expand the scenario a little bit (I hope I'm not being too much of a bother!): If the workflow instance becomes idle and is persisted and unloaded by the runtime, what will trigger it getting reloaded? Will something arriving at a workflow queue the workflow is waiting on trigger the runtime to reload the workflow instance?

If it happens that you had two workflow hosts connected to the same persistence DB, how would locking affect this? I think I understood how locking worked when the runtime was started and persisted workflows were reloaded, but can a workflow instance unloaded from one appdomain be reloaded again at another time by a second runtime instance in a different appdomain (possibly different machine) while  the first runtime is still running, or does it remain locked so that a different runtime won't pick it up unless the original runtime terminates?

Answer 10

Ok so let's name things.  Assume we have HostA and HostB.

HostA starts workflow #1.  Workflow#1 becomes idle and persists (we assume the settings are this way) and unloads.  When an instance unloads there are no locks (meaning any host can reload it).

Now - how does a Host recreate that workflow?  When you call WorkflowRuntime.GetWorkflow - the WorkflowRuntime first looks for instances in memory - since we know #1 is unloaded it isn't in that list - so the WorkflowRuntime looks for a persistence service.  Since we have a persistence service - it loads the workflow from the persistence store and GetWorkflow returns a live in-memory workflow to which we can call EnqueueItem on (or in the case of using ExternalDataExchange - we can fire an event to - which ends up calling EnqueueItem).

Either Host A or B can do this - assuming they both point to the same persistence database and both have all the assemblies necessary to serialize and deserialize the workflow.

Now - where does locking come in?  Assume HostB called GetWorkflow on #1 - at that point it would be locked.  If a message came into the HostA for workflow#1 - it would be locked by HostB - either until it idles again - or until HostB decided to have it persist (WorklfowInstance.Unload) at which point the lock would be lifted and HostA could load the workflow.

The bad scenario is if HostB dies before it can persist the workflow again (and unlock it) - the workflow will be in "limbo" until the ownership timout hits.

Answer 11

Jon Flanders wrote:

Now - how does a Host recreate that workflow? When you call WorkflowRuntime.GetWorkflow - the WorkflowRuntime first looks for instances in memory - since we know #1 is unloaded it isn't in that list - so the WorkflowRuntime looks for a persistence service. Since we have a persistence service - it loads the workflow from the persistence store and GetWorkflow returns a live in-memory workflow to which we can call EnqueueItem on (or in the case of using ExternalDataExchange - we can fire an event to - which ends up calling EnqueueItem).

Ahh, excellent, that makes perfect sense. So since the underlying service for my activity that handles the subscriptions will always need to call GetWorkflow() to dispatch the notification through the workflow queue, things should "just work" in the general case. I was hoping that was the case, but for some reason failed to realize that the GetWorkflow() call would be the key to this. Thanks for the pointer[

Jon Flanders wrote:

Now - where does locking come in? Assume HostB called GetWorkflow on #1 - at that point it would be locked. If a message came into the HostA for workflow#1 - it would be locked by HostB - either until it idles again - or until HostB decided to have it persist (WorklfowInstance.Unload) at which point the lock would be lifted and HostA could load the workflow. The bad scenario is if HostB dies before it can persist the workflow again (and unlock it) - the workflow will be in "limbo" until the ownership timout hits.

Interesting scenario. So it's pretty much up to the underlying service handling the subscription to guarantee that it doesn't loose the event if the call to GetWorkflow() (or putting the event data in the workflow queue) fails, right? This is something I'll want to check in my MsmqActivities that I'm handling correctly.

Thanks for all the help, Jon, it is much appreciated!

Answer 12

This is why the BizTalk messagebox is really a very brilliant thing.  Since all messages are routed to the message box - it is easy for the messagebox to route messages to the correct host instance.

If you say had a WCF application listening on an MSMQ queue - where messages might need to be routed to the correct workflow instance - and more than once process was listening on that queue - you have a synchronization issue you have to fix in your applicaiton.

Answer 13

Yep, I soo agree. All this time I've been thinking just how amazing and easy the BizTalk Pub/Sub engine in the MessageBox makes things! I'm very looking forward to seeing how WF gets used in a future BizTalk version.

I'm a big fan of the BizTalk Messaging architecture, it has a lot of really really cool things (and not just the messagebox). Heck, I wish the WCF team had taken a couple of ideas from the biztalk stack and used them instead of going in a different direction! it would sure make a few things easier :)