Workflow Engines and Products

I am in love with workflow products, workflow engines in particular. Listed below are some of workflow engines and workflow based(*) products that I came across and looked at over last 3 years.

BPM
* Tibco iProcess
* BizTalk
* Savvion
* Lombardi
* Oracle BPEL Process Manager

Other Products
* IBM Tivoli Intelligent Orchestrator
* JOpera
* Full Armor Workflow Studio
* K2.Net
* LogicBase Harmony
* Skelta
* Adobe LifeCycle Workflow
* Opalis - oh I know this engine well
* overview pdf
* Taverna
* TurboWorx - look ma, they lost their domain! Oh, don't even think of going torboworx dot com ...

Workflow Engines
* YAWL
* Bossa
* OpenWFE
* Enhydra Shark
* ActiveBPEL
* Windows Workflow Foundation

Hords of open source Java engines are listed here and here (with discussion). Interestingly, there is no open source C++ workflow engine. I googled hard, but only found a dead one in C. I'll be happy if you proves me wrong.

Here you could imagine me doing a product review. But let's save your time and cut straight to the bottom line.

Over last 10 years everyone and his brother had been building a workflow engine. Most of the products listed have their own ones. Developing them has been a lot of fun. This time is over. Now in 2007 workflow engine becomes a commodity. Who are the leaders?

I nominate ActiveBPEL and WinWF

Is Agile Any Good for a Startup?

Agile is about reducing risk, improving predictability, and having more fun. Startup is about taking risk, making surprises, and having more fun. Is agile any good for a startup?

Startup can’t reduce risks, they take it.
Startup can’t improve predictability – there is none.
Startup can’t be any more fun – it is already too much to bear.

Despite rumors, agile is a disciplined process. It promise future benefits, but comes with a cost. Bad trade-off: start-up is a life boat that needs to get to shore before short supplies run out.

Optimizing dev process? Why bother improving speed in a future, get it going now! Code maintainability? Why bother, if it goes down the toilet faster then it’s written. Continuous integration, auto build, unit testing… Why bother? These best practices proved themselves in the industry. So what? A research lab is not like a production line.

The only thing from agile that suits really well is the motto: "Every day is a good day to die".

Ok, there is more. They share fundamentals. Focus on people, working software, customer collaboration and embracing change.

I am being controversial to stir the pot. Those who know me are not surprised. I don’t know if agile is any good for a start-up. Do you know? Leave a comment.

WCF+WF in ORCAS: Deep Dive Training at Redmond

I spent 3 days at Platform Adaptation Center in Redmond, taking a dive in hot Microsoft technologies. The focus was on Orcas, the upcoming version of Visual Studio and .NET 3.5. We also got insights of the current .NET 3.0, and features beyond ORCAS.

My professional focus was on Windows Workflow Foundation. My geeky side was fascinated by LINQ. Windows WF worths a dedicated post. For details on LINQ I’d rather refer you to someone who can tell the story: Anders Hejlsberg! Watch a video interview from here. To taste it, follow "The road to Linq: C# 3.0 for mere mortals" paved by Alex Thissen.

It is intriguing: ADO.NET team comes with Entity Framework, new and cool, along with ESQL query language. It directly competes with LINQ to SQL (aka DLINQ); and we got competing evangelists from the two competing teams presenting. The obvious question "what Microsoft’s recommendation on choosing between the two" was answered with pretty much "the jury is out".

Every time I see PowerShell, I am excited and ready to install & play & learn & use it right away. Never gotten beyond installation yet . After David Aiken’s show, I will do it. David had delivered Using Powershell to manage WF & WCF – great demo! Later he spoke about application manageability, WMI, events and their play in M$ Dynamic Systems Initiative.

Rules rulez! Moustafa Ahmed made it clear. Rehost a rules engine whenever you pleased, and what a great power. Rules Editor, on the other hand, is not so cool. Technically it is rehostable, but only developers can possibly use it. For normal users, you’re on your own.

What else? Three presentations on Windows Communication Framework. Justin Smith was talking about Syndication and later on WCF performance. Pravin Indurkar shown in-depth interplay of WCF and WF. I liked Justin’s thinking broadly of syndication, beyond blogs and news. Think of an event log as a syndication of events, a record set is syndication of data rows, a web page as a syndication of data backed controls, and so on. WCF is promising support for Atom & RSS.

AJAX: two presentations, by Steve Marx and Eugene Osovetsky, and the lab on JSON & AJAX win WCF. Workflow Driven User Interface: web application “HelpDesk” example demoed by Shelly Guo. Compact framework and Office/Sharepoint – not interested.. Federated Identity was presented in greatly understandable way by CardSpace team – now I can explain it myself! Brian Keller was showing off VS Team System. It was off-beat for the general theme, but for me both the presentation and after-talk with Brian was so interesting… I gotta blog on this separately.

The audience of around 50 was a mix of innovators from blogging and training cutting edge, on one side, and pragmatics from the dusty field of daily industrial development, on the other. Innovator Alex wanted to know where LINQ is going after ORCAS. I wanted to figure if I could use WF right now. Satisfying both sides presented a challenge to the organizers. At the end, though, both sides got their questions answered.

Overall – good organization, smart guys, great event.

OpenLazlo

A friend of mine gave me a buzz: “Check it out, OpenLaszlo”.

A free open source AJAX toolkit, OpenLazlo gives an XML based language with embedded JavaScript that compiles into Flash or DTHML. Originally optimized for FireFox, but claims full support for Internet Explorer, including 7.0. No JSON, but XML based dataset with XPath query is provided instead. Animation is OpenLaszlo’s sweet spot. Demos are sexy, and “Laszlo in 10 minutes” is fun. Although I still more like fine-grain control, like with Atlas, OpenLaszlo declarative style ramps up web development time.

Morale: AJAX is cool, Open Source rules, friends matter.

Effects of Defects: Grey Scope Creep

Is boosting productivity simple? Yes! Just cut out craploads of code! One problem here. The gap between code complete and feature done is bigger then it appears. That was my thought as I was looking at statistics.

On average, a developer cuts off 300 lines per day, some true heroes claim more. This “productivity” comes with the price of 100 defects per 1000 lines of code. Finding defect and fixing it is on average 4 to 16 hours [1] [2]

Do the math: Your kick ass star developer produces 500 lines a day. This creates up to 80 hours of extra work to himself, the dev team and their tester buddies. 10 damn days! I call it “grey scope creep”.

At first, this statistics seems totally off. But can’t you remember such a kick ass code monkey? you surely know a few. Now go over all these steps to fix a defect: cutting the build, installing it in QA, testing, finding the bug, writing a bug report, assigning it back to development, reviewing, trying in vein to reproduce… “Works on my machine”, sending it back to QA”, and finally, after a couple of round-trips, it’s fixed. Than regression sticks out an ugly head, and the high 16 man-hours per a bug looks too good to be true.

So, is boosting productivity simple? Yes! Less bugs in, more bugs out! One problem here. But enough on problems, let’s get positive and move on to how fight grey scope creep.

First, if the kick ass developers only care about the joy of cutting code, not shipping the software, they got the boots. Don’t regret, they produce more work then they get done. It is not about developers developing and testers testing. It is developers and testers working cooperatively to ship a software. We got space for local optimization.

Second, good software practices are to the rescue. Pair programming, design and code reviews, test driven development, etc… They surely slow down the LOC/day pseudo-performance (less bugs in) and reduce the number of defects. Pair programming by itself brings 15% less bugs, with healthy 15% slow-down of LOC performance [3].

Third, redefine “done”. Fresh features are marked done, and disappears somewhere in QA to eventually fire back at unknown time with unknown bugs. Grey scope creep. Stop it. Instead, insist on taking less but making it “done” within an iteration. Done reads fully developed, thoroughly tested, debugged, fixed including regression, and accepted by product owner. If this bring development to a deadlock, it’s one of the two. You may be blessed to fail faster and cheaper. After all, if you can’t get “done” just one feature, how would you ever be done with the entire project? Or the team overcomes the crisis and changes the working habits. The shortened feedback loop and fixing bugs as you make it leads to less time per bug fix. The grey scope doesn’t creep beyond the iteration. Problems become obvious early enough to to deal with them.

Finally, until you fix the grey scope creep, your estimations could be off, up to 10 times! Measure well.

So, is boosting productivity simple? Yes! One problem here. Trying to address it may expose the stakeholders to too much truth about the state of the project. Figure out how to deal with this truth. Or wait to my next blog, I have some thoughts about it.

Reader's Digest #2: Writing, Receiving Criticism, and Hiring

From Paul Graham, again. I read his essey on writing Writing, Briefly. Then I printed two copies - one for me, one for my son who is into writing, to read it daily till it sticks. If you write, tell me I'll print you a copy. Or serve yourself: http://www.paulgraham.com/writing44.html

"How to give and receive criticism" By Scott Berkun. Common sence, structured. Nothing new but the fact this may be news to someone.

How does receiving criticism applies to software development? Directly and profoundly: It rases team productivity and improves the quality of software. Steave McConnel stresses technical and desing reviews as premium quality assurance techniques. Conducting reviews effectively is all about giving and receiving criticism. Handling criticism well indicates higher morale and better team dynamic that guarantees productivity improvement. Convinced? Think of probing this skill when hiring.

Speaking on hiring:
* another essey from Scott Berkun "How to interview and hire people". Again, common sence, structured. A few tricks to learn, but it's not about tricks, it's about interviewer's goals and attitude. I'm going to use it to refresh my perspective on hiring before plunging into massive interview race. For candidates, it worth reading to see what to look forward on the other side.

* also, check out "The Guerrilla Guide to Interviewing" which Joel Spolsky updated to v3 in October 2006, reflecting changes with time passed since the first version. I like it even better now.

* and finally, Extreme Interviewing by Software Factory fellows.

Migrating ASP.NET AJAX based WinFX Monitor

The most interesting examples around Windows Workflow Foundation is John Flander's article on hosting Workflow Monitor in ASP.NET Web App with AJAX. One problem: since his post ASP.NET AJAX went to RC, then RTM, lost name Atlas, and support of some features. So John's app doesn't work on ASP.NET 2.0 AJAX Extensions 1.0 RTM. To figure out why, RTFM here and here. Main trouble is Microsoft.Web.Extension.dll has been renamed to System.Web.Extension.dll, luckily token is the same. So the fix was to rename it all over the Web.config, plus a few little changes. To save you troubles going through this routine, get a modified Web.config for John's example.

Disclaimer: Works for me :-) aka no testing. Enjoy.

Windows Workflow Foundation - Love at First Sight

I am falling in love with Windows Workflow Foundation (WF). Formely known as WinFX, it is recently released as part of .NET 3.0, becoming a core for a number of Microsoft products, and actively promoted by Microsoft Evangelists along with WCF and WPC.
I took a dive at Windows WF, and loved it. Microsoft got it right. Well, mostly. Here is where they I think they missed:

1) Designer is terrible. Yes it is polished and pretty, but it is a usability disaster. It’s not worse then a typical BMPN product, though. They are all guilty of overwhelming a user with all the complexity of notations and workflow patterns. As a result, most workflow tools suck just like UML based code generators. Too bad Microsoft chose to suck with the most.

A minor usability issue: Activities for a traditional sequential workflow and a state machine workflow are all mixed in the same toolbox. Can I drag this activity to this workflow? Makes a beginner to guess, makes an expert to remember.

2) Enterprise level high availability is either not addressed or hidden. I have not found any stories about failover, clustering, and load balancing.

3) Workflow in web clients is not there yet. But looks like it’s coming, check out www.netfxlive.com/

Despite these points, I love Windows WF, at least at first sight. This love is easy to test: just have a second sight. That is what I am going for – a good second sight at Windows WF along with WCF. Meet me at Redmond next week.