Snake Oil

It's really funny, talking to people sometimes. There's this big debate about "rules.". The funny part is how, in the absence of knowledge we move immediately to implementation. I guess it's just human nature.

Business rules, as a technology category and type of engine, is an implementation of a way to encapsulate logic. It's another way to code something up that is, at times, more efficient than alternatives. But not always. Sometimes it is less efficient.

Let's take configuration of a complex telecom switch. If you had to write all the rules and figure out all possible configurations of the switch, you would spend more time writing the application than it's worth. It was also very costly to do this for every quote... It required many highly skilled people. Enter high-end rules engines as a good way to model the complex system so that a valid configuration can be made by a relatively inexperienced user. (The price paid was a VERY experienced author of the model, not to mention the expensive configuration technology itself, and the vendor's services costs...very high and they go on forever...).

There's an economic model here that is important: the issue isn't that telco's used a configuration/rules engine to "externalize the business logic" for its own sake (by the way, it isn't - it's just locked inside a different type of container... for proof, look at the follow-on consulting revenues of rules engine vendors, or ask them to export the rules to a competitor's engine...); the issue is that cost of startup and maintenance of rules encoded in that manner allow you to scale the human use, and (presumably) sell more switches because you give superior or broader customer service.

So far so good. When you have a highly complex problem, using a complex technology solution is OK, because it extends access and enables organizational scale. The high fixed cost of dealing with the rules engine paid off because it replaced a fairly high variable cost of many manual configurations.

Note that the magical word "agility" is not in and of itself a meaningful corporate goal: agility should ALWAYS be considered only to the extent it cost-effectively furthers reach (higher numbers of less skilled people can now do things) or improves your customer's experience (thereby giving you some competitive advantage). Rules-centric approaches are useful when the problem is so complex that you want to trade variable complexity (cost) for fixed complexity (cost).

But what about less complex problems?

The equation with complex problems like configurations and credit scoring and underwriting decision engines was that at some point in the early 90's, it became more efficient to code up those algorithms using rules engine technology than using other means to solve the problem.

But does it follow that EVERY rule or algorithm should be coded up in this way? Of course not. In fact, rules engines as a specific implementation actually impede agility because of the difficulties they place in front of the developer. There are some pieces of business logic that MANY people could understand graphically, but that very FEW people could understand in a rules context.

For example, a process application (like any computer program) CAN be depicted as a set of rules. But is that easier than just drawing, say, a line? And if the line is interpreted by a purpose-built process engine, isn't that more efficient, too? (Answer: yes)

There is a point at which what you are trying to express is simply easier to do in an alternative manner than in a rules-driven system. CAN I represent a user interface as a set of rules? Sure, but why would I want to in a world with rich, easier-to-use tools?

And in the best BPMSs, the engines are metadata-driven (no coding and compiling), therefore they are just as technically agile as rules-driven systems. Make a change and the system changes behavior - now.

So there are TWO factors that determine agility: a system's ability to change in real-time (without compile/reboot intervention) and a human's ability to change the system (who is able to understand the system, make the change, and test it). And there are two factors that drive the need for agility: scale (cost) and customers (revenues). The closer you can align the needs for agility with the means of agility, the better off you are. That is: the closer you can align the humans who want the change with the humans required to effect it, the better off you are.

Peter Drucker said that every human touch is like a repeater, 1/2 signal loss for each one... So being able to have a lot of people who are able to understand the system very close to the people who need the system changed is key to process management in the 21st century.

More people can fully use a good process-centric BPMS than a rules-centric one. By a factor of A LOT.

Welcome to the 21st century process! A little process and a lot human...

Another problem with a rules-centric approach is that rules engines have an inherent problem in the areas of simulation, and historical analyses of WHY certain rule paths are traversed. This lack in ability to introspect is a massive failure. You might notice that rules vendors talk down this need... But if you talk to business people, the ability to do what-if analysis is an imperative.

Rules engines know THAT a rule has fired, but no context as to why... There is no inherent understanding of process or flow.... Just rules firing.

This isn't the case with the best process-centric engines. To a rules engine, a process is a bunch of rules. To a process-centric system, a process is a process. It is a rich semantic, and it delivers rich understanding.

And this understanding is key to optimization...

Rules engine vendors would have you believe that it's pretty easy to just "not automate a bad process.". They say: "just automate a good one. Take your 12 step process to 6!". If it were that easy, AA would have way more members. Heck, why not take the 6 to 3. And then on down to one. "Great encapsulation" they'd all say...

There ARE obvious improvements that can be made to many processes. You should absolutely make those changes. Using whatever technology you choose! But one of the breakthroughs we've made is that today's "process" isn't a simplistic deterministic workflow. We're not here to make Henry Ford's production line out of your nurse onboarding process! The humans are in the process for a good reason... What we need to do is provide the means to help them discover the rules that are being applied in non-obvious ways.

It may be obvious, for example, that all orders from Wal-Mart over $xx.xx should be auto-approved. But it's not so obvious that 87% of all shipments from xyz in belgium arrive late when they are ordered within 15 days of quarter's end. Or that a human approves orders from 2nd tier customers 99% of the time when an inside sales person is the rep and not a channel partner. To write that rule is easy in either system.. But it's only easy to DISCOVER it in a process-centric BPMS.

The cool thing about great process management is that you can automate human processes and then watch what the humans do - or at least the inputs to what they do and the decisions they make - without getting in their shorts about HOW to make the decision! Use a wiki? Fine! Use the phone? Great! Just git 'er done!

It's not ABOUT automating every decision... It's about automating efficient flow and information and gathering the results of all those completed instances. Level 1 in most new Process Maturity Models say "get a handle on the As Is; stabilize and structure your workloads."

And then, IF you granularly track those process instances, use advanced data mining technology to understand and correlate what data led to what decisions.

Breakthrough!

Business process management is now about DISCOVERING the complex business rules it would normally take many years to understand, AND quickly insert those rules into the process with easy to understand methods.

And so that's what is funny. Today, some people would rather talk about rules as an implementation methodology, even before giving you the means to understand what rules should be in place.

Visibility is key to business process management, and metadata-driven, easy-to-grasp authoring is key to the implementation of BPM.

So don't buy the snake oil they're selling! Rules engines play a key role in the enterprise... But they are ineffective and inefficient process systems. Remember, it's just cod liver oil with a prettier bottle, basically...

- Phil Gilbert (via mobile)

Technorati Tags: BPM, Business Rules

Comments

[Paul Snow]

I agree with much of your rant on Rules Engines. This is why I wrote a Rules Engine from the ground up for the Texas TIERS project. All of the policy Rules had already been put into Decision Tables by another Vendor to serve as our specification. All I had to do was load up those Tables and execute them. If the policy failed, it would quite clearly be because the specification was in error.

"All I had to do" is of course tricky. It involved writing a language that was "near English" but that I could none the less compile and execute, and do so with performance in Java.

The Rules Engine had to be fast (there are 3000+ decision tables representing 10 to 30 conventional rules per table). Sets of these tables are executed over and over for each case evaluated.

And the Rules Engine had to be tracable (that is, we had to be able to easily explain the context and result of every decision executed in the tables).

As far as implementing the policy, these rules have been impressively effective. And without any of the flaws you seem to expect from Rules Engines.

[Phil Ayres]

Phil,

My feeling, having far greater experience with BPMS than strict business rules is to agree with much of your post.

Of course, there are specific examples where business rules exercised at a point in a process make absolute sense - for example in financial services: credit checks, money laundering risk ratings, suitability reviews. These can take diverse sets of data from many sources and use that to create a result that can be used within a process.

But to reinforce how I view BPMS and rules: business process management drives the process; rules influence decisions at a point in time.

And within this system, resorting to a human knowledge worker is not always a bad outcome.

Phil Ayres
http://improving-nao.blogspot.com/

[Phil]

Good question and that's really my point... "business rules" vendors would have you believe that the magic to their agility is in the "business rules," but, in fact, the magic is that they have engines that interpret the metadata that describes the rules in real-time. My point was that other _types_ of engines could also do this. In fact, some process-oriented engines do this now (ours included).

Since either engine can therefore "be changed in real-time" the other parts of the change cycle will be the guarantors (or inhibitors) of success. That is, the humans. So if it's easier for a human to change a metadata-driven process system, then that system is more agile than a metadata-driven rules system.

[Danny]

I'm having trouble seeing where you place the line between rule-based systems and non-rule-based systems.

e.g. "...in the best BPMSs, the engines are metadata-driven (no coding and compiling), therefore they are just as technically agile as rules-driven systems"

How does the metadata (or perhaps rather the part of the system that determines how to operate based on the metadata) - differ from rules?