Matt Heusser’s Blog
Testing at the Edge of Chaos
How we make decisions
Last week, Apple announced it’s iPad. If you haven’t heard the annoucement, you can go watch the eight-minute YouTube advertisement, or Steve Job’s 30-minute public announcement. It’s okay. It’s worth it, and I can wait.
… time passes …
Now, I didn’t come away from that video wanting to buy an iPad – it’s for consumers, people who surf the web, watch video, listen to tunes, etc. My focus is less on consuming and more on putting out new material. So it’s designed to support an entirely different group of people.
No, what I wanted to do was to buy shares of Apple Computer Corporation.
Which brings me to the topic of today’s blog post – how we make decisions. You see, we like to think that we make decisions rationally, in a sort of cold, hard, unemotional way. Line up the pros and cons, evaluate each one, and off we go.
But it’s not like that all. Sure, we may use facts to guide our decision making, but even then, as my friend Michael Bolton likes to point out, we tend to decide on how we feel about those facts. For example: Tell any executive that you expect 80% of your paying “members” to renew next year, or that you expect 20% to not renew, and you’ll likely get two different reactions.
So here are the facts I responded to about the iPad:
(1) Apple computer has a history of integrating it’s hardware and software offerings – the iTunes store, for example, has both music and video. Downloadable books seem like the next logical step.
(2) Amazon’s Kindle product has proved the market for a book-reading device. I also have some confidence that Apple can do it better; after all, they’ve been a hardware/sw company from year one. (There is even an old “kindle 3″ parody, worried about the dumbing-down of society that ends with tag line “It plays movies instead of books”)
(3) Apple has a history of innovation, new products, and even new categories. The iPod, for example, has sold 250 million copies – and Apple is incredibly good with price points and using new technology, to get users to upgrade every two to four years. The have a fan base so strong that, as another parody claimed “I’ll buy almost anything that is shiny and made by Apple.”
Sounds like strong reasons to invest right?
Well, there is another side. About a year ago I got another device – a netbook. It was shiny, small, a neat. It was also slow and hard to sync my files back and forth from different devices. Within a few months, I realized that I had a thin, small, cheap laptop. It wasn’t a category killer at all.
Moreover, Apple has been hyped lately; it’s trading very close to it’s 52-week high, which could mean it’s over-priced. If the iPad doesn’t take off, it might just be time for Apple’s stock to a hit. And Apple does have misses, or at least near misses – remember the MacBook Air?
If I wanted to, I could easily line up a set of facts against Apple just as strong as the reasons for investing it. On some level, investing is a bet that involves prediction, and telling the future accurately is probably best left to the weather guy on channel eight … but don’t go out past a couple days; then things start to get fuzzy.
But there is one more trick to decision making: If you know the value system or the culture and a little basic information, the decision they will reach is often predictable. I won’t say it’s entirely satisfying, but I have said to senior managers and executives things like “when nine months go by and this software hasn’t shipped, you can always call me” or “If this software doesn’t exist by February, just ask me and I can build something that can ship in a month.” (Both of those organizations had cultures that led to analysis paralysis.)
Most of us gather information in a way that leans toward Organic or Mechanic. When we have enough information to make a decision, the additional facts don’t matter – we rationalize them to fit the decision we have already made.
In that case, gathering additional facts is a kind of waste.
And that knowledge helps me make my own decision.
So, if you’ll excuse me, I’m off to purchase some stock in Apple Computer.
The tyranny of the deadline
Hang around software development or testing long enough, read enough blogs and books, and sooner or later, you are likely to hear the great crisis talk. It typically goes something like this:
(A) Software Development is in a state of crisis!
(B) (Quote the Chaos Report)
(C) Something must be done!
(D) Therefore – whatever the presenter is selling
The presenter could be selling anything – from process improvement, to ‘getting the requirements right’, to ‘prevention’, or ‘inspection’ – it doesn’t really matter.
The point is, software development is in a state of crisis and we must be fix it!
Wait a minute. Let’s hold our horses. What is this Chaos report, exactly?
It turns out that a business think-tank called the Standish group conducts a series of surveys to determine the overall state of the health of IT; the 1995 report is available on-line free. At the very least, it is an interesting read.
The Standish’s report (the “Chaos Report”) takes IT projects and divides them into three buckets:
Three Project Outcomes
Resolution Type 1, or project success: The project is completed on-time and on-budget, with all features and functions as initially specified.
Resolution Type 2, or project challenged: The project is completed and operational but over-budget, over the time estimate, and offers fewer features and functions than originally specified.
Resolution Type 3, or project impaired: The project is canceled at some point during the development cycle.
Three buckets. Really?
Imagine, for a moment, we used the same criteria to evaluate movies. Did you know that the movie Titanic had it’s original release date of June 1995 pushed back to December? Under these definitions it would be “challenged”, even though it was the single largest grossing film in the history of cinema.
Forget Titanic; Star Wars was six months late! That’s right kids; the single largest iconic geek movie of the 20th century – the one that launched five sequels, countless action figures, lunch boxes, breakfast cereals, toy light sabres that make whoosing sounds and it’s own version of “Monolopy” – should be caulked up as a failure. (It did run $3 million over budget, after all, costing $11 million instead of $8)
Oh, bull pucky.
Even worse than late or over-budget, the Chaos report considers projects that were “canceled early” a failure, when in my career I have worked on projects that delivered 90% of the value in 50% of the allotted time. Likewise, some projects should be canceled if the competitive environment changes and the project becomes irrelevant.
What all this means is the report doesn’t allow for the possibility of change – positive or negative. Instead of measuring the execution of the software team, it actually measures the ability of the estimators at the beginning of the project; not only on their estimation, but also on their ability to predict the future (changes to the environment) that will happen before the project finishes.
Pictured this way, the ’shocking’ and ‘abysmal’ ‘failure’ rates actually mean that, as an industry, we aren’t so good at predicting the future.
I don’t think this is all that shocking, nor would I call it a crisis.
Where did we go wrong?
The biggest problem I see is that the chaos report is only looking at a single dimension – “did the project hit the deadline” instead of asking a more reasonable question, like, for example “did the delivered software have a good return on investment?”
Or, perhaps, one single question just isn’t enough. It’s like asking “what’s your weight” to someone; it’s hard to draw any conclusions without knowing their height. Even then, a weightlifter can come off “overweight”, you need body fat index to really tell. And that might not be healthy; you’d need to know cholesterol
.. you get the point. Drawing conclusions with a single dimension can be dangerous.
How can we do it you one better? Well, the usual. Develop in small chunks. Adapt to change. Plan to re-plan. Think while we do. Limit our work in progress, so that if a new opportunity arises, we have to throw away less work. Plan phase gates that release working software – not documents that may or may not be useful.
And, as testers, try not to be fooled by happy-talk – or even crisis-talk.
Meanwhile if you want to build a new framework to analyze projects in our industry – one with depth, that, to paraphrase Gandhi, is the change we want to see in the world – please, drop me a line; I’d be interested.
Estimates – III (Serious this time)
I’ve re-titled the last Estimates III as “Models – I”, as it really deserves it’s own serious.
Pressing on …
(AKA – Stupid Project Management Tricks; a True Story)
The code is big stinking mess, all over the floor. The team is dependent on a vendor who hasn’t delivered a good build to us yet. We have no idea when anything is going to be ready. My best bet is three months out, but nobody really knows. Senior Management is Antsy. What will the project manager say?
Ok, so I wasn’t there, but I know what happens: He walks into the big office and says “THE PROJECT WILL BE DONE IN FIVE MONTHS.”
Where did that number come from? I have no idea. There was probably a gannt chart somewhere that added up to five months, but nobody really knew, and nobody technical trusted the vendor.
All I do know is that senior management went home that night – and they slept well, because they had a date. A firm commitment. None of this “gee, we don’t really know, how long does it take to find your lost keys?” business. A firm date.
It was bogus, but, whatever. They could sleep.
… time passes …
Four and a half months later, and it becomes obvious that the team won’t deliver in two weeks. I’ve been pulled off to work on another, different death march(*), but I take a general, cynical tester’s interest in the project.
Again, the word comes down from on high “WE’VE HAD ANOTHER SLIP. THE PROJECT WILL SHIP IN FOUR MONTHS …”
Of course, four months goes by …
Eventually, the project was cancelled: Over a year late.
Now, I criticize a lot of things about that project. 99% of the time, I think the project management was irresponsible, incompetant, and (possibly) unethical.
Then again, 99% of the time, senior management was able to sleep at night, because, dang nab it, they had a date. None of this “uncertainty” business.
Which reminds me of an old quote by DeMarco/Lister – all too often, people would rather be certain and wrong than uncertain. After all, it’s better to be a team player than be negative.
So, the basic technique is this: If you have no idea when you’ll be done, make up a date so far in the future that you’ve got a good shot. When it becomes obvious that you cannot hit that date, well … make up a new one.
Or, in other words: When uncertainty is politically impossible, do what gets you rewards.
Like I said before, 99% of the time, I think this is an incompetant and bogus way to “lead” projects. But that 1% of the time, I think: Man, if you have no idea when it’s going to be done, making up a date real far away and slipping if you have to is a decent approach. (It can’t be too far out – otherwise you’ll get push back.)
All of that presupposes that you actually can ship at all. The problem in the story is that we couldn’t, and anyone with decent systems thinking skills could see that, advise management, and would, er, well …
… “find other projects.”
–heusser
(*) – Actually, to be honest, that one was a relief; it was a death march, and I worked massive overtime, and it was painful, AND IT SHIPPED.
Estimates – II
As usual, some of my commenter’s have written my article for me …
Seriously, Ben Simo points out that is it always possible to give an estimate, but that all estimates are wrong (if they were right, they would be commitments). Shrini points out that we get lousy requirements for our estimates. For example, when asked:
“When will it be done?”
You probably have a vague and fluffy definition of “IT” and a vague or unrealistic definition of “done.”
For example, let me re-translate:
“When will it be done?”
Could be:
“How long will it take you to do everything I’ve written down here, with scope creep that _I_ deem reasonable, without a single defect or flaw – given that anything over a month is too long?”
Yikes!
Of course, I promised some actual answers. So here goes …
The “GAP” between what the software actually IS and how it was analyzed is a serious pain point in software development. That gap widens with time – so my first suggestion is to never run a project more than about a month. (Yes, require/dev/test/prod in thirty days.) Schedule any “big” project as a series of small ones.
A worst, you might be two weeks late. That’s not the end of the world.
If you have to run longer than a month, then periodically bring your code up to production quality. Develop features end-to-end, in thin slices, and add them to the system.
That’s not really an estimating hint – it’s a project organization hint. Estimates of less than a month are drastically easier than more. Moreover, they are small enough that an employee can feel a real personal commitment to the date, and work a little overtime to make it. (On a ten-month project, once you realize you’re late, overtime will just kill you, not bring the project back on track.) So you organize the project so you never have to estimate more than a month at a time.
Plus, you can prioritize the customer’s feature requests, so you build the most important thing first.
So what if that is not an option?
Ok, next idea. Do two types of estimates – realistic and pessimistic. Make the realistic estimate your goal date, and make the pessimistic a commitment back to the business. Within the team, talk about the goal; with management, talk about the commitment.
A third – Estimates can be done in two different ways. The first is an analysis task where break down the features into chunks and add up the chunks, or you take the spec and do some math to come up with an approximate number of test cases. The second way to do estimates is to actually start doing the work – at a very high level. Go ahead and _DO_ the design, or start to decompose the test cases, or write “Tracer bullet” objects to do the highest levels of disk, screen, and database interaction. Leave comments or “TODO” markers at where the next step will be. When you’ve finished that, come back to the TODO markers, estimate them, and add them up. Finally, add a little extra for risk – the bigger the risk, the bigger the buffer.
The thing is, the further you go into the project, the more you’ll know. I have one colleague who simply avoids making any commitments or estimates at all. He’ll say things like “if you say so” or “that’s certainly our goal” but never make a promise. I’m not advocating that – but certainly the further you get into the project, the more accurate your dates will be.
(In Rapid Development, Steve McConnell actually suggests that you communicate estimates in a range that gets smaller over time, with the larger number first. “Six to four months” sounds strange – but if you say the smaller number first, people tend to forget the larger one.)
Looking back, most of this is not how you should do the estimates – but how to communicate them so they are not abused or mis-understood. This can be very challenging; as my old friend Eric likes to say “Sooner or later your three paragraphs of very specific if-thens and assumptions are going to turn into a PowerPoint bullet with a date on it. Tread lightly.”
If that’s the case, then you have a few options. One is to hit any arbitrary date by structuring the project so that you can go to production every month. Another is to make no commitments and blame someone else. A third is to way is to continually and consistently communicate risk-adjusted dates back to your customers.
My final thought is to express the cost of every change. It’s usually not the project that kills us – it’s the feature changes at the end. And that’s fine – it’s good – it makes us provide software that has better fitness for use. When that happens, though, either make the cost explicit (by adding the feature and insisting on moving the date) or pay for it with your nights and weekends.
The choice is yours ….
Estimates – Laws
I’m a big fan of “Laws” of software development – Rules that express complex ideas in very few words. Moore’s Law, for example, that computing power doubles (roughly) every eighteen months.
Or Jerry Weinberg’s advice to not attribute to malice what can be explained by incompetance.
Or Paul Jorgensen’s rule of large numbers in software development: Big Numbers times Big Numbers equals REALLY big numbers.
Now me, I’m working on a theory for software estimates, but here’s another interlude, Heusser’s Rule, which explains a lot of the … odd behavior you’ll see on software projects:
Most people really like to be able to sleep at night
This explains why people would rather be certain and wrong than uncertain and right. It explains why people hold on to bad ideas that should have been discredited long ago. It explains why people can force themselves that they will meet a deadline, despite a mountain of evidence to the contrary.
And it explains why they shoot the messenger. It’s not that you’re wrong; no, you are very much right. But by calling a spade a spade, you’ve robbed the entire room of the ability to sleep at night.
More soon.
Estimates – Interlude
The previous post was on estimates for technologist in general.
Software Testers, things are a little more … challenging. For example –
What quality will be the code be when it is delivered to test?
Most of the time, we don’t know. Vast differences in quality could make testing take a longer – or shorter – length of time.
What is the management expectation for the software when it is released?
Most of the time, these expectations aren’t quantified – if they are articulated at all.
Once identified, how quickly will the developers fix the bugs?
Unless you’ve had the same team for several projects, and the projects are similar, you probably won’t know this. If the answer is “not that fast”, then it’s probably not a testing phase – it’s a fixing phase – and it’s not bound by the testers.
For every one hundred bugs the developers fix, how many new bugs will be injected?
I’ve seen teams where this number is fifty, and I’ve seen team where this number is much less – but I can’t think of a team where this number is zero. Obviously, if this number is bigger, testing will take longer.
I could go on – the point is that we’ve got a whole lot of variables that are beyond the control of the testers. In fact, it’s probably impossible for the testers to even know what those variables are. So how can you provide an estimate?
Offhand, I know of only one career field that has to deal with a similar problem. That is — your friendly 401(k) investment advisor. I always thought that “advisor” was a bit of a misnomer, because providing advice is the one thing that guy doesn’t do. Instead, he carefully explains that he doesn’t know where social security will be in fifty years, and that he doesn’t know the value of your house, how it will appreciate, and weather you will pay it off early or take home equity loans. He doesn’t know what inflation will be, or how you investments will perform.
So he gives you a piece of paper with a big, complex formula, and tells you to figure it out for yourself. After all, he will say “It’s your money.”
Testing is an investment of time by a business owner. It is senior management’s money, yet we couldn’t get away with this, now could we?
Don’t worry. I have a few ideas and solutions for you.
More tomorrow.
Estimates – I
If I had to think of one subject that was not taught in school, and only covered in industry certifications in the most naïve way – it would be estimation.
It seems so simple. You figure out what needs to be done, figure out how to do it, break the how down into tasks, and add them up. This is pure functional decomposition, right? Easy as pie.
Sadly, In the real world, not so much. Do any of these sound familiar?
Forced Due Dates. In this case, nobody really does estimates. Or perhaps the boss does something quick, sloppy, and “aggressive.” In any event, the boss needs the project done in two weeks. Period.
‘Someone else’ does the estimates. Estimation is a skill, and it’s a skill that few junior staffers have. So having a senior staffer (or manager) do the estimates for a junior person might be helpful. The problem is that is separates the estimate from the person responsible. In other words, making someone else’s estimate is like spending someone else’s gift money – it is just detached enough from reality that you won’t be careful enough.
“Four weeks? Really? I could do it in one …” This is a manipulation technique used at the big status meeting, and I hate it. In some cases, the bully really can do it in one week, and it’s a great win for the company to switch. More often, all it takes is an offer to switch to shut the bully down. Junior people tend to buckle, promise the impossible, ‘fail’, and then look bad.
Bring me a rock estimates. “Hmm … three months? Can you shave a little time off that? … two months? We were looking for something a little more aggressive … One Month? OK, that sounds good. Remember, that is YOUR NUMBER, and we’ll hold you to that estimate!”
Slippery-Slope Requirements. Perhaps, by some miracle, you are allowed to estimate the project with your own date. Halfway through, the customer realizes that the requirements won’t meet his needs. This leaves you with two options – (A) Add the features and slip the date, making the project “late” or (B) Deny the features and deliver a product that doesn’t fit it’s purpose.
Why Estimates Are Important
In many cases, the “success” or “failure” of the project depends on weather or not we hit the date. In most of the scenarios above, that’s not a judgment of the technical contributors – it’s an evaluation of the person who made the original SWAG compared to reality.
In my next post in the series, I will discuss what we can do about it.
