Matt Heusser’s Blog
Testing at the Edge of Chaos
How we make decisions
Last week, Apple announced it’s iPad. If you haven’t heard the annoucement, you can go watch the eight-minute YouTube advertisement, or Steve Job’s 30-minute public announcement. It’s okay. It’s worth it, and I can wait.
… time passes …
Now, I didn’t come away from that video wanting to buy an iPad – it’s for consumers, people who surf the web, watch video, listen to tunes, etc. My focus is less on consuming and more on putting out new material. So it’s designed to support an entirely different group of people.
No, what I wanted to do was to buy shares of Apple Computer Corporation.
Which brings me to the topic of today’s blog post – how we make decisions. You see, we like to think that we make decisions rationally, in a sort of cold, hard, unemotional way. Line up the pros and cons, evaluate each one, and off we go.
But it’s not like that all. Sure, we may use facts to guide our decision making, but even then, as my friend Michael Bolton likes to point out, we tend to decide on how we feel about those facts. For example: Tell any executive that you expect 80% of your paying “members” to renew next year, or that you expect 20% to not renew, and you’ll likely get two different reactions.
So here are the facts I responded to about the iPad:
(1) Apple computer has a history of integrating it’s hardware and software offerings – the iTunes store, for example, has both music and video. Downloadable books seem like the next logical step.
(2) Amazon’s Kindle product has proved the market for a book-reading device. I also have some confidence that Apple can do it better; after all, they’ve been a hardware/sw company from year one. (There is even an old “kindle 3″ parody, worried about the dumbing-down of society that ends with tag line “It plays movies instead of books”)
(3) Apple has a history of innovation, new products, and even new categories. The iPod, for example, has sold 250 million copies – and Apple is incredibly good with price points and using new technology, to get users to upgrade every two to four years. The have a fan base so strong that, as another parody claimed “I’ll buy almost anything that is shiny and made by Apple.”
Sounds like strong reasons to invest right?
Well, there is another side. About a year ago I got another device – a netbook. It was shiny, small, a neat. It was also slow and hard to sync my files back and forth from different devices. Within a few months, I realized that I had a thin, small, cheap laptop. It wasn’t a category killer at all.
Moreover, Apple has been hyped lately; it’s trading very close to it’s 52-week high, which could mean it’s over-priced. If the iPad doesn’t take off, it might just be time for Apple’s stock to a hit. And Apple does have misses, or at least near misses – remember the MacBook Air?
If I wanted to, I could easily line up a set of facts against Apple just as strong as the reasons for investing it. On some level, investing is a bet that involves prediction, and telling the future accurately is probably best left to the weather guy on channel eight … but don’t go out past a couple days; then things start to get fuzzy.
But there is one more trick to decision making: If you know the value system or the culture and a little basic information, the decision they will reach is often predictable. I won’t say it’s entirely satisfying, but I have said to senior managers and executives things like “when nine months go by and this software hasn’t shipped, you can always call me” or “If this software doesn’t exist by February, just ask me and I can build something that can ship in a month.” (Both of those organizations had cultures that led to analysis paralysis.)
Most of us gather information in a way that leans toward Organic or Mechanic. When we have enough information to make a decision, the additional facts don’t matter – we rationalize them to fit the decision we have already made.
In that case, gathering additional facts is a kind of waste.
And that knowledge helps me make my own decision.
So, if you’ll excuse me, I’m off to purchase some stock in Apple Computer.
The tyranny of the deadline
Hang around software development or testing long enough, read enough blogs and books, and sooner or later, you are likely to hear the great crisis talk. It typically goes something like this:
(A) Software Development is in a state of crisis!
(B) (Quote the Chaos Report)
(C) Something must be done!
(D) Therefore – whatever the presenter is selling
The presenter could be selling anything – from process improvement, to ‘getting the requirements right’, to ‘prevention’, or ‘inspection’ – it doesn’t really matter.
The point is, software development is in a state of crisis and we must be fix it!
Wait a minute. Let’s hold our horses. What is this Chaos report, exactly?
It turns out that a business think-tank called the Standish group conducts a series of surveys to determine the overall state of the health of IT; the 1995 report is available on-line free. At the very least, it is an interesting read.
The Standish’s report (the “Chaos Report”) takes IT projects and divides them into three buckets:
Three Project Outcomes
Resolution Type 1, or project success: The project is completed on-time and on-budget, with all features and functions as initially specified.
Resolution Type 2, or project challenged: The project is completed and operational but over-budget, over the time estimate, and offers fewer features and functions than originally specified.
Resolution Type 3, or project impaired: The project is canceled at some point during the development cycle.
Three buckets. Really?
Imagine, for a moment, we used the same criteria to evaluate movies. Did you know that the movie Titanic had it’s original release date of June 1995 pushed back to December? Under these definitions it would be “challenged”, even though it was the single largest grossing film in the history of cinema.
Forget Titanic; Star Wars was six months late! That’s right kids; the single largest iconic geek movie of the 20th century – the one that launched five sequels, countless action figures, lunch boxes, breakfast cereals, toy light sabres that make whoosing sounds and it’s own version of “Monolopy” – should be caulked up as a failure. (It did run $3 million over budget, after all, costing $11 million instead of $8)
Oh, bull pucky.
Even worse than late or over-budget, the Chaos report considers projects that were “canceled early” a failure, when in my career I have worked on projects that delivered 90% of the value in 50% of the allotted time. Likewise, some projects should be canceled if the competitive environment changes and the project becomes irrelevant.
What all this means is the report doesn’t allow for the possibility of change – positive or negative. Instead of measuring the execution of the software team, it actually measures the ability of the estimators at the beginning of the project; not only on their estimation, but also on their ability to predict the future (changes to the environment) that will happen before the project finishes.
Pictured this way, the ’shocking’ and ‘abysmal’ ‘failure’ rates actually mean that, as an industry, we aren’t so good at predicting the future.
I don’t think this is all that shocking, nor would I call it a crisis.
Where did we go wrong?
The biggest problem I see is that the chaos report is only looking at a single dimension – “did the project hit the deadline” instead of asking a more reasonable question, like, for example “did the delivered software have a good return on investment?”
Or, perhaps, one single question just isn’t enough. It’s like asking “what’s your weight” to someone; it’s hard to draw any conclusions without knowing their height. Even then, a weightlifter can come off “overweight”, you need body fat index to really tell. And that might not be healthy; you’d need to know cholesterol
.. you get the point. Drawing conclusions with a single dimension can be dangerous.
How can we do it you one better? Well, the usual. Develop in small chunks. Adapt to change. Plan to re-plan. Think while we do. Limit our work in progress, so that if a new opportunity arises, we have to throw away less work. Plan phase gates that release working software – not documents that may or may not be useful.
And, as testers, try not to be fooled by happy-talk – or even crisis-talk.
Meanwhile if you want to build a new framework to analyze projects in our industry – one with depth, that, to paraphrase Gandhi, is the change we want to see in the world – please, drop me a line; I’d be interested.
