Using the Rules Engine with complex data containers
We are currently working on an implementation of the WF Rules Engine to validate complex data containers. These data containers contain not only simple properties, but collections of other data containers which themselves may also contain collections of other containers. We are evaluating what a suitable business rule set would look like to validate these containers, and the feasibility of business end-users [or developers] maintaining the rules associated with these containers.
Jurgen's article which introduces the workflow business rules engine talks about a pattern for iterating over a collection and evaulating rules for each member in that collection. The basic pattern looked like:
1. Create Enumerator and assign to activity/workflow-level variable [pri 2]
2. MoveNext on enumerator [pri 1]
3. Validate Data in current enumerated instance [pri 0, 1..n]
4. Update Enumerator to cause [2] to be reevaluated [pri -1]
Looking at what this pattern looks like in the supplied Rules Editor, I have concerns that such an arrangement falls short on a couple fronts.
1. It is very hard to understand how the rules for a given enumerated list fit together
2. Enumerating a list requires that activity/workflow level variables are introduced for each such possible enumeration.
I see these two issues as pretty serious shortcomings in the proposed approach. I am very concerned that explaining how to setup business rules to a business user [or a junior developer for that matter] will be difficult, and potentially leave them unable to realistically create a ruleset which can validate the business rules.
My main question is whether or not the provided pattern is the recommended approach when dealing with the validation of data containers which contain collections of data to be validated. If not, is there another pattern which could be applied which is simpler to understand and does not require the introduction of code/variables to track the progress of the iteration?
Currently I see two methods of implementing complex business validations on such containers:
1. Build the looping/iteration logic into the workflow itself, inserting the appropriate policy objects to validate individual items in the collection. This seems to trade complexities in the business rule set for complexities in the calling workflow. I'm not really sure this is a good tradeoff, since it makes things like loading the validation logic from an external store more difficult.
2. Using a pattern such as that suggested in the business rules introduction. This keeps the workflow simple, but for complex containers with multiple collections, the rulesets can get really ugly and require that there are placeholder variables sitting around to hold the intermediate state of all the enumerators.
3. Using another business rule centric pattern which is simpler to understand and implement :)
4. Build some sort of "structure" around the validation logic, coupled with a custom activity that handles loading in all the appropriate ruleset bits and applying them to the appropriate portions of the data container. This seems to keep rulesets and workflows simple, but introduces the need for a more complex rule set management application and accompanying code infrastructure to handle the validation execution.
The additional wrinkle in all this is that whatever solution is chosen, it must be able to integrate with an external rule set store, since throughout the system, the business rules to run [and any other associated parameters] can vary based on the particular deployment's configuration.
Answers
There is not currently a way to automatically iterate over a collection. We are investigating a capability like this for the future, though (post v1).
Can you tell me a little more about your pattern for using the containers? Are you planning on iterating over multiple collections in a single ruleset evaluation? If so, are those collections nested? e.g. "given this.Orders.Items, loop over the Orders collection and for each Order loop over its Items. Some rules are written directly against the Order instance and some against the Item instance"
Some thoughts on your points:
TheJet wrote: 1. Build the looping/iteration logic into the workflow itself, inserting the appropriate policy objects to validate individual items in the collection. This seems to trade complexities in the business rule set for complexities in the calling workflow. I'm not really sure this is a good tradeoff, since it makes things like loading the validation logic from an external store more difficult.
Putting the Policy activity (or whatever activity you use to execute the rulesets) inside a Replicator in the workflow would seem to be a pretty straightforward approach. You're right that this adds some complexity to the workflow but I'm not sure that it necessarily makes externalization of ruleset any more problematic. It shouldn't, for example, dictate any hard dependency between the workflow and the ruleset. I may be oversimplifying your scenario, though.
TheJet wrote: 2. Using a pattern such as that suggested in the business rules introduction. This keeps the workflow simple, but for complex containers with multiple collections, the rulesets can get really ugly and require that there are placeholder variables sitting around to hold the intermediate state of all the enumerators.
3. Using another business rule centric pattern which is simpler to understand and implement :)
You need to provide some way to iterate over the collections, though. You could use other standard patterns as you do in code (e.g. increment a counter variable "i" and access myCollection[ i] until you get to the end of the collection), but this doesn't really address your concern.
A couple other options that you could explore:
1) Author the rulesets against the target type (e.g. item #3 above - Validate Data in current enumerated instance [pri 0, 1..n]) and then interject the rest of the pattern into the ruleset in your activity/rules management infrastructure (at compile/execution time). This does not diminish overall complexity but does limit the scope of what the rule author needs to understand.
2) Create a parent ruleset that is responsible for implementing the iteration and then calls other ruleset(s) that are then executed against specific instances. This could be done in the parent ruleset either by having it call out to some helper code you define or by defining a custom RuleAction.
Most of these suggestions revolve around simplifying the rule authoring. Most still require fields to hold intermediate state. If you want to limit the fields that are needed to hold this state you could, of course, always handle the active enumerators for multiple collections in a Dictionary.
Hope this helps. As I mentioned, we have been doing some thinking about richer ways for supporting collection iteration, so I would love to hear the kind of authoring experience that you would like to see.
All Replies
There is not currently a way to automatically iterate over a collection. We are investigating a capability like this for the future, though (post v1).
Can you tell me a little more about your pattern for using the containers? Are you planning on iterating over multiple collections in a single ruleset evaluation? If so, are those collections nested? e.g. "given this.Orders.Items, loop over the Orders collection and for each Order loop over its Items. Some rules are written directly against the Order instance and some against the Item instance"
Some thoughts on your points:
TheJet wrote: 1. Build the looping/iteration logic into the workflow itself, inserting the appropriate policy objects to validate individual items in the collection. This seems to trade complexities in the business rule set for complexities in the calling workflow. I'm not really sure this is a good tradeoff, since it makes things like loading the validation logic from an external store more difficult.
Putting the Policy activity (or whatever activity you use to execute the rulesets) inside a Replicator in the workflow would seem to be a pretty straightforward approach. You're right that this adds some complexity to the workflow but I'm not sure that it necessarily makes externalization of ruleset any more problematic. It shouldn't, for example, dictate any hard dependency between the workflow and the ruleset. I may be oversimplifying your scenario, though.
TheJet wrote: 2. Using a pattern such as that suggested in the business rules introduction. This keeps the workflow simple, but for complex containers with multiple collections, the rulesets can get really ugly and require that there are placeholder variables sitting around to hold the intermediate state of all the enumerators.
3. Using another business rule centric pattern which is simpler to understand and implement :)
You need to provide some way to iterate over the collections, though. You could use other standard patterns as you do in code (e.g. increment a counter variable "i" and access myCollection[ i] until you get to the end of the collection), but this doesn't really address your concern.
A couple other options that you could explore:
1) Author the rulesets against the target type (e.g. item #3 above - Validate Data in current enumerated instance [pri 0, 1..n]) and then interject the rest of the pattern into the ruleset in your activity/rules management infrastructure (at compile/execution time). This does not diminish overall complexity but does limit the scope of what the rule author needs to understand.
2) Create a parent ruleset that is responsible for implementing the iteration and then calls other ruleset(s) that are then executed against specific instances. This could be done in the parent ruleset either by having it call out to some helper code you define or by defining a custom RuleAction.
Most of these suggestions revolve around simplifying the rule authoring. Most still require fields to hold intermediate state. If you want to limit the fields that are needed to hold this state you could, of course, always handle the active enumerators for multiple collections in a Dictionary.
Hope this helps. As I mentioned, we have been doing some thinking about richer ways for supporting collection iteration, so I would love to hear the kind of authoring experience that you would like to see.
Thanks for the quick reply.
Jurgen Willis wrote: There is not currently a way to automatically iterate over a collection. We are investigating a capability like this for the future, though (post v1).
Can you tell me a little more about your pattern for using the containers? Are you planning on iterating over multiple collections in a single ruleset evaluation? If so, are those collections nested? e.g. "given this.Orders.Items, loop over the Orders collection and for each Order loop over its Items. Some rules are written directly against the Order instance and some against the Item instance"
Yes, this is exactly what we are attempting to do. We are trying to determine from a number of standpoints, where the right balance between ruleset scope and management/maintenance simplicity. Containers can be arbirtrarily complex, but all conform to a fairly standard tree-based structure [i.e. no loops in the graph]. The containers represent the data contract instances in our messaging architecture.
Jurgen Willis wrote: Some thoughts on your points:
TheJet wrote: 1. Build the looping/iteration logic into the workflow itself, inserting the appropriate policy objects to validate individual items in the collection. This seems to trade complexities in the business rule set for complexities in the calling workflow. I'm not really sure this is a good tradeoff, since it makes things like loading the validation logic from an external store more difficult.
Putting the Policy activity (or whatever activity you use to execute the rulesets) inside a Replicator in the workflow would seem to be a pretty straightforward approach. You're right that this adds some complexity to the workflow but I'm not sure that it necessarily makes externalization of ruleset any more problematic. It shouldn't, for example, dictate any hard dependency between the workflow and the ruleset. I may be oversimplifying your scenario, though.
That is true, however, the entire rule validation process itself needs to be swapped in and out. Really the workflow needs to be able to point to a single "validation unit", whether that be a sub-workflow or ruleset, which can validate the incoming data. That validation unit must be loaded at runtime based on the application configuration [and varies per deployment]. The only fixed aspect is the fact that a particular validation unit will always exist. Currently, we are thinking that using a ruleset as a validation unit is probably easier than using a sub-workflow, but I haven't fully investigated what it would take to use a workflow as a validation unit.
Jurgen Willis wrote: A couple other options that you could explore:
1) Author the rulesets against the target type (e.g. item #3 above - Validate Data in current enumerated instance [pri 0, 1..n]) and then interject the rest of the pattern into the ruleset in your activity/rules management infrastructure (at compile/execution time). This does not diminish overall complexity but does limit the scope of what the rule author needs to understand.
This is the path we are currently investigating, allowing the rule author to choose the sub-item in the container and write rules against that sub-item. However, this doesn't address the issue of saying things like "this employee must have exactly one home address", which still need to be handled at the container level, and for which I don't see a nice, single rule type check.
Jurgen Willis wrote: 2) Create a parent ruleset that is responsible for implementing the iteration and then calls other ruleset(s) that are then executed against specific instances. This could be done in the parent ruleset either by having it call out to some helper code you define or by defining a custom RuleAction.
We had thought about going this route, and this is part of the solution we are currently investigating, but comes back to the question of "how do we link these distinctly defined rule sets into one cohesive unit". The two options I currently see are:
1. Inject the iteration/looping rules/variables around the authored rules, resulting in one large ruleset. The benefits of this are rentention of full forward-chaining semantics, but this requires actually "rewriting" the rules that the author provides as part of the merge process to basically replace "this" with the appropriate path to the container item in question.
2. Create a custom activity that essentially loops over the container and queries the "rule metadata" for the appropriate ruleset to apply to the specific container item. This seems to be the easier approach, but removes the possibility to have any sort of cross ruleset chaining. If any of our rulesets required such chaining [and I don't know that they do], this would not be a valid solution.
Jurgen Willis wrote: Most of these suggestions revolve around simplifying the rule authoring. Most still require fields to hold intermediate state. If you want to limit the fields that are needed to hold this state you could, of course, always handle the active enumerators for multiple collections in a Dictionary.
Hope this helps. As I mentioned, we have been doing some thinking about richer ways for supporting collection iteration, so I would love to hear the kind of authoring experience that you would like to see.
Using something like a dictionary is not so good, it makes authoring slightly more complicated and you don't get the benefit of "compile time" validation. From a "what would be nice to see" aspect, I'd love to see things extended to support what I would expect from a logic system in reference to collections, operations like:
1. bool IfExists(IEnumerable<T>, Predicate1, ..., PredicateN): This essentially says, if there exists an item in the collection which matches the predicate
. Similar to the List<T> function, but without the need to explicitly create the delegate [the rule engine would do that for you].2. bool IfForAll(IEnumerable<T>, Predicate1, ..., PredicateN): This asks the question if all the items in a collection match the given predicate(s).
3. T First/Last<T>(IEnumerable<T>, Predicate1, ..., PredicateN): Returns the first/last item in the collection which matches the predicate(s).
4. In conjunction with something like "named subsets" of rules, having a simplifying call which is something like: ForEach(IEnumerable, RuleSubSetName). This would simply run the named sub-set of rules for each item in a collection. Keeping chaining semantics intact and all that stuff.
For all these, the specification of predicates would not be the passing of delegates [as seen in List<T>], but rather actually specifying things like:
if !Exists(Employee.Addresses, item.AddressType == AddressTypeEnum.Home)
introducing another keyword like "item" or some such to make parsing/intellisense possible for such a statement would be ideal.
Thanks!
As a side question, it doesn't appear that there is a way to integrate custom rule actions into the current rule editor dialog similar to how the Update/Halt actions are integrated. Is that a correct statement, and if so, what is the recommended method of adding that type of action, is code/XAML the only option?
TheJet wrote: As a side question, it doesn't appear that there is a way to integrate custom rule actions into the current rule editor dialog similar to how the Update/Halt actions are integrated. Is that a correct statement, and if so, what is the recommended method of adding that type of action, is code/XAML the only option?
At this point you would have to edit the object model or XAML directly. We have changed this, though, so that in v1 you'll be able to enter the custom RuleAction directly into the editor.
TheJet wrote: 1. bool IfExists(IEnumerable<T>, Predicate1, ..., PredicateN): This essentially says, if there exists an item in the collection which matches the predicate . Similar to the List<T> function, but without the need to explicitly create the delegate [the rule engine would do that for you].
Thank you for the input on the collection processing semantics. In the meantime, you might be able to achieve part of what you're looking for using custom expressions (i.e. a custom implementation of IRuleExpression). You mentioned custom RuleActions, so you may have already looked into custom expressions. You could, for example, define a custom IfExists expression that takes a reference to a collection. Inside your expression implementation, then, you would do the iteration over the collection.
The challenge, as you know, will be in authoring the predicate portion, since there's nothing surfaced to provide an "item" concept. You could try various ways to get around this; as an example, you could have the predicate entered as a string and then you parse this to create the condition. If you want to leverage our intellisense you could add a field of the Type of the instances in the collection onto the workflow and then author against that such that you have IfExists(this.Order.Items, this.Item.ProductCode == "ABC123"). As another example, you could create a dummy class with static members that mimic the members on the collection items, e.g. IfExists(this.Order.Items, ItemDummyClass.ProductCode == "ABC123"). Since you evaluate the expression, you could then transform the expression into an executable one (e.g. replace the CodeTypeReferenceExpression into a CodeThisReferenceExpression in the expression).
If the separation of the logic is acceptable, you could also have your custom expression take a condition name, e.g. IfExists(this.Order.Items, "ProductCodeCheck"), where the condition, "ProductCodeCheck", is authored separately against the Item type. I realize that none of these are exactly what you're looking for - just some suggestions.
Feel free to drop a note to JWillis here at Microsoft if you'd like to explore any of these in more detail.
