Matt Heusser’s Blog
Testing at the Edge of Chaos
The tyranny of the deadline
Hang around software development or testing long enough, read enough blogs and books, and sooner or later, you are likely to hear the great crisis talk. It typically goes something like this:
(A) Software Development is in a state of crisis!
(B) (Quote the Chaos Report)
(C) Something must be done!
(D) Therefore – whatever the presenter is selling
The presenter could be selling anything – from process improvement, to ‘getting the requirements right’, to ‘prevention’, or ‘inspection’ – it doesn’t really matter.
The point is, software development is in a state of crisis and we must be fix it!
Wait a minute. Let’s hold our horses. What is this Chaos report, exactly?
It turns out that a business think-tank called the Standish group conducts a series of surveys to determine the overall state of the health of IT; the 1995 report is available on-line free. At the very least, it is an interesting read.
The Standish’s report (the “Chaos Report”) takes IT projects and divides them into three buckets:
Three Project Outcomes
Resolution Type 1, or project success: The project is completed on-time and on-budget, with all features and functions as initially specified.
Resolution Type 2, or project challenged: The project is completed and operational but over-budget, over the time estimate, and offers fewer features and functions than originally specified.
Resolution Type 3, or project impaired: The project is canceled at some point during the development cycle.
Three buckets. Really?
Imagine, for a moment, we used the same criteria to evaluate movies. Did you know that the movie Titanic had it’s original release date of June 1995 pushed back to December? Under these definitions it would be “challenged”, even though it was the single largest grossing film in the history of cinema.
Forget Titanic; Star Wars was six months late! That’s right kids; the single largest iconic geek movie of the 20th century – the one that launched five sequels, countless action figures, lunch boxes, breakfast cereals, toy light sabres that make whoosing sounds and it’s own version of “Monolopy” – should be caulked up as a failure. (It did run $3 million over budget, after all, costing $11 million instead of $8)
Oh, bull pucky.
Even worse than late or over-budget, the Chaos report considers projects that were “canceled early” a failure, when in my career I have worked on projects that delivered 90% of the value in 50% of the allotted time. Likewise, some projects should be canceled if the competitive environment changes and the project becomes irrelevant.
What all this means is the report doesn’t allow for the possibility of change – positive or negative. Instead of measuring the execution of the software team, it actually measures the ability of the estimators at the beginning of the project; not only on their estimation, but also on their ability to predict the future (changes to the environment) that will happen before the project finishes.
Pictured this way, the ’shocking’ and ‘abysmal’ ‘failure’ rates actually mean that, as an industry, we aren’t so good at predicting the future.
I don’t think this is all that shocking, nor would I call it a crisis.
Where did we go wrong?
The biggest problem I see is that the chaos report is only looking at a single dimension – “did the project hit the deadline” instead of asking a more reasonable question, like, for example “did the delivered software have a good return on investment?”
Or, perhaps, one single question just isn’t enough. It’s like asking “what’s your weight” to someone; it’s hard to draw any conclusions without knowing their height. Even then, a weightlifter can come off “overweight”, you need body fat index to really tell. And that might not be healthy; you’d need to know cholesterol
.. you get the point. Drawing conclusions with a single dimension can be dangerous.
How can we do it you one better? Well, the usual. Develop in small chunks. Adapt to change. Plan to re-plan. Think while we do. Limit our work in progress, so that if a new opportunity arises, we have to throw away less work. Plan phase gates that release working software – not documents that may or may not be useful.
And, as testers, try not to be fooled by happy-talk – or even crisis-talk.
Meanwhile if you want to build a new framework to analyze projects in our industry – one with depth, that, to paraphrase Gandhi, is the change we want to see in the world – please, drop me a line; I’d be interested.
Testing Philosophy II -
About every four months, Shrini Kulkarni convinces me to drop the term “Test Automation” from my vocabulary. After all, testing is a creative, thinking process. We can automate some steps of what we are doing, speeding up repetitive work – but those don’t turn out to be the investigative, critical thinking steps(*).
Still, I want to use a word to describe when I use automation to go faster. I use awkward terms like “automating the repetitive steps” or “build verification tests” for a few months, until I end up calling it test automation. Then Shrini emails me, and the cycle repeats.
Since I am on the down-stroke of one of those cycles, I thought it would be appropriate to link to an old paper of Brian Marick’s:
I want to automate as many tests as I can. I’m not comfortable running a test only once. What if a programmer then changes the code and introduces a bug? What if I don’t catch that bug because I didn’t rerun the test after the change? Wouldn’t I feel horrible?
Well, yes, but I’m not paid to feel comfortable rather than horrible. I’m paid to be cost effective. It took me a long time, but I finally realized that I was over-automating, that only some of the tests I created should be automated. Some of the tests I was automating not only did not find bugs when they were rerun, they had no significant prospect of doing so. Automating them was not a rational decision.
The question, then, is how to make a rational decision. When I take a job as a contract tester, I typically design a series of tests for some product feature. For each of them, I need to decide whether that particular test should be automated. This paper describes how I think about the trade offs.
The paper is “When Should A Test Be Automated?” – and it is available on the web.
Now, that isn’t exactly my philosophy, but I think it’s good readin’.
Speaking of good readin’; if you are doing test automation by writing code in a true programming language to hit a GUI, you might enjoy this classic by Michael Hunter. (If you are not, it’s still an interesting read, but everything after about page 12 is very specific about writing code libraries for application ‘driving’cripting.)
And now, Shrini, I’m back on the up cycle. Really. I promise.
–heusser
(*) – It doesn’t help that a lot of “test automation”, especially in the 1990’s, were snake oil products, designed by marketeers, to show how testing could be “sped up” with massive ROI numbers. I can’t tell you how many boxes of GUI test automation software I have seen sitting, unused, on shelves. Lots – and even more than that are the stories I’ve heard from colleagues.
When I talk about test automation, that’s not what I mean – see philosophy I for a better description.
My theory of software testing – I
What’s the right mix of exploratory testing, “planned” manual testing, and test automation?
My short answer is “it depends.” Now, you don’t need to point out that “it depends” is non-helpful – I realize that – and I am going to try to go beyond it.
The reason I say “it depends” is that it depends on the type of problem you have. So one thing I can do to go farther is to list a bunch of possible problems, along with kinds of testing I have seen that fit.
1) The build verification test, or BVT. These are tests you run immediately after every build to make sure the software isn’t completely hosed. In a typical windows app, this is – Install / File New / Add some stuff / File Save / Quit / Re Launch / File Open / The Screen should look like THIS. You add more tests over time, but the point is – if these tests fail, it was probably one big massive regression error, any additional testing information is suspect. The test team will work on other projects until the devs can get the BVT to pass.
BVT tests are incredibly boring, and often worth automating.
2) The Fahrenheit-To-Celsius Conversion test. Sure, you could have a human being test a hundred examples manually, but if you have a spreadsheet that you can use as a source of Truth, why not write a program that loops through all values from -10,000 to 10,000, by increments of 0.001, calls the function, does the math in the spreadsheet, and compares the two? Note that this does not provide information about Boundaries, which may be best explored – but it can help you build some confidence about the happy middle.
3) Model-Based Testing. Similar to #2, if the application is simple enough you can create a model of how the software should behave, then write tests to take “random walks” through the software. (Ben Simo has several good blog posts on this topic, including here.)
Despite the appeal of Model-Based Testing, it requires someone with true, deep testing expertise, true development expertise, modeling skill, and, usually, a gui-poking tool. So, despite all it’s promise, the actual adoption of model-based testing has been … less than stellar.
4) Unit Testing. This is as simple as “low-level code to test low-level code” – often much further down in the bowels than a manual tester would want to go, it provides the kind of “safety net” test automation suite makes a developer comfortable refactoring code. And devs need to do this; otherwise, maintenance changes are just sort of hacked onto the end, and, five years later, you’ve got a big ball of mud.
5) Isolation Problems. If you have a system that requires a stub or simulator to test (embedded system), you may need to want/need to write automated tests for it.
6) Macros. Any time you want to do something multiple times in a row to save yourself typing, you may want to do automation. Let us say, for example, you have a maintenance fix to a simple data extract. The new change will stop pulling X kind of employees from the database, and start pulling Y.
You could run the programs before and after, and manually compare the data.
OR you could use diff.
And, for that matter, you could write a SELECT COUNT(*) query or two from the database and see that:
The total number of new rows = (old rows + Y members – X member);
This is test automation. Notice that you don’t have to have a CS degree to do #6 – and most of the others can be done by a tester pairing up with a developer.
So When should I not do automated testing?
1) To find boundaries. As I hinted at above, automated system-level tests are usually pretty bad at finding bounds. So if you’ve got a lot of boundary errors, by all means, test manually.
2) As Exploratory Testing. Sometimes when we test our goal is investigation; we play “20 questions” with the software. We learn about the software as we go. This kind of “exploratory testing” can be extremely effective; much more effective than mine-field automated testing.
3) When we don’t need to repeat the tests. Some things only need to be tested once – Tab Order. Or may be best tested by a human, because they require some kind of complex analysis.
Sapient Testing
Human beings can be sapient; they can think. Humans can look at a screen and say “Something’s not right …” where a bitmap compare would simply fail or pass. Humans can look at a screen and intuit the most likely parts to fail. So the key is to use the technology to help automate the slow, boring, dumb tasks.
For example, at GTAC this year, one group said that it had 1,000 manual tests that ran overnight – recording a screen capture at the end of each. The next morning, a test engineer sits down with his coffee and compares those screen captures to the previous night – using his brain to figure out which differences matter, and which don’t. After he finishes those build verification tests (in about a half hour), he can then get down to pushing the software to it’s knees with more interactive, exploratory testing.
Now, that morning coffee .. Is that automated testing, or is it manual? My short answer is I don’t really care.
These are just ideas, and they focus almost exclusively on the relationship between Dev and Test. I haven’t talked about the relationship between Analyst and Test, for example. Don’t take this as Gospel. More to come.
Certifiable – III
I am drafting a reply to the agile-leadership group, but posting it here first.
Several people (including me), asked what problem certification solves, or who the “customer” is for the certification. To which Alistair Cockburn replied:
Back in message 416 or thereabouts I wrote: “In both cases, the market is so hungry for an agile PM certificate, that the market will snap it up (witness the response to the 2-day “Certified ScrumMaster” course).”
and in message 458 I wrote: “Looking at project managers in general, a company hiring PMs will follow two paths at the same time: they will send some of their existing PMs, or people about to become PMs, to PMI school to get their PMP certificates as part of general training. …”
To me, this argument seems to boil down to: “That Market Wants It. If we don’t do it, someone else will, and they will do a much worse job.”
Please keep in mind, I respect Alistair and what he has done, and I mean this in the best possible sense, but there is just no easy way to say this.
“If we don’t do it, someone else will” is not a reason – it is a rationalization. It is the same rationalization that people use to sell Crack, Heroin, and other drugs. And rationalization no reason, it is a logical fallacy – the fallacy of deceptive alternatives. (Logic and Rational Thought, Frank Harrison III)
Letting another vendor grab title-share is one alternative. Having the ALPN create the certifications is another. Education and Dialogue is a third.
My proposed solution is dialogue. It won’t make me rich, or even popular, but that way, at least I have chance to be Good.
And that’s a trade that I would make any day.
Regards,
–heusser
Certifiable – II
Another post, this one to the agile-management list …
Ron Jeffries Wrote:
A leadership certification, one would imagine, would perforce be far less specific and far more general, but just as likely to be interpreted as indicating that the certificate holder is qualified, surely more qualified than her colleagues around her who do not hold the certificate, to be chosen to “be a leader”.
Yes. It’s very hard for a certificate program to avoid becoming a Confidence Game in Software Engineering.
Not that it’s impossible, but it is hard…
The guy at the front of the room, the guy who is the leader in the field, he is out *INVENTING* a new BOK (body of knowledge), not studying someone elses. I want to be that guy.
If you would like to assess me, read what I write, or talk to me about it:
http://www.xndev.com/articles/articles.htm
I realize that some people do not want to be that guy; some people don’t like to write or speak, they want to do the same thing, over and over again, and do it well.
But if that’s the case, is a certificate in “Agile Leadership” really the right fit?
Certifiable?
The Agile Project Leadership Network is working on a certificate program in Agile Leadership.
This has been discussed a bit on the Agile Leadership Yahoo group. I just emailed a reply and thought I would share it here ..
Ron “The Don” Jeffries Wrote:
>But rightly so. My lack of comments on the topic should be read as
>coming from a similar frame of mind, oddly conflated with “if you
>can’t say something nice, don’t say anything at all”.
>
What I don’t get is what pushes _everybody_ toward certification.
It’s … odd. It’s like …
1) Form a club (Association, Company, Institute, Take Your Pick)
2) Create a Code of Ethics
3) Run a Conference
4) Create a certification program
5) ?
6) Profit!
It doesn’t matter if you are scrum or agile or a software tester, everyone seems to push in this direction. Aside from step 6, which, I must admit, sounds nice, I don’t get it.
I went to seminar on perl certification at the O’Reilly Open Source Conference in 2003. The moderator was Tim Mahr; you can read his opinion here: http://minimalperl.com/consultix/publications/perlcert_article.html
His main talking point seemed to be that certification allowed the employer to reduce the risk of a bad hire — if you hire a certified java guy, you know you are going to get a certain level of competence.
This seems, to me, to be like outsourcing the function of discernment.
If your managers of software development don’t have discernment, well … you’ve got a problem. Maybe instead of a crutch, you should be working on that?
Now, a certificate in Agile Leadership, used for the same purposes … on it’s face, that concerns me, but I just don’t have enough information to tell at this point.
The ALPN feedback cycle is now apparently private; does anyone have any contact information, so I can find out more without mucking the waters?
–heusser
Agile Shark Jumpin’ – IV
I turns out that my Collegue, Jim Brosseau has been going through some of the same issues that I have. Jim and I don’t totally agree on every issue (who does?), but he does say some things in a recent newsletter that I think are rather profound, and I’m going to provide a couple of exerpts.
Here’s the first, which talks about his experience at Agile Vancouver:
I have no problems with an idea that fills a room and gets people talking about techniques that help them be more effective. What I do have problems with is the over inflation of the value, or even the knowing neglect to correct some flawed assumptions about an idea, all in the name of further padding of the wallet.
At one point in the final panel session, there was the suggestion that teams starting down an Agile path should enlist the support of an expert for training. There were more than 2 of the experts that I counted at the front of the room that clearly recognized this as a “cha-ching” moment. This was expressed either politely as a satisfied smile and nod, all the way to the blatant Tiger Woods pumping of the arm for the hole in one. Who is the primary beneficiary of this Agile movement, anyways?
Consumer Tip: If any company or organization suggests a specific methodology, tool or framework as a reasonable solution for you before you can safely say that they truly understand and empathize with you, your culture, your challenges and your products, do your best to make the door hit them on the ass on the way out!
Even if that tool or methodology is what you called them about in the first place.
What is Agile, if not a fully fledged product? Perhaps an enabling technology, perhaps a series of patterns, perhaps a philosophy. Good stuff, but currently over-hyped.
What I saw at the conference was well beyond a philosophy, it was a religion, bordering on a cult. There was a common enemy that everyone could rally against, with an ‘If you are not with us, you are against us’ fervor. All things non-Agile were seen as bad, and generally lumped into the category of Waterfall or hacking. Plenty of ad hominem discussions about evil managers.
There were a large collection of ideas that were embraced by and expressed as Agile, even though the vast majority have been around and practiced in effective ‘waterfall’ projects for decades. Retrospectives are Agile. Estimation based on size is Agile. Estimation Poker is Agile (hello…Wide Band Delphi, anyone?). Early stage focus on test is Agile (and has been a well known approach for early stage validating scope for years).
Perhaps the current push extends some of these ideas a bit further, but there is nothing that I would call disruptive technology. Hell, we wrote what looks a lot like XUnit over a dozen years ago on a huge ATC project. Decidedly pre-dating Agile, but we did some smart things.
What is happening is that a group of people are re-discovering things that work, assuming that they can be generally applicable, and evangelizing a bit too aggressively. I wrote an article for the Cutter IT Journal in June of 2004 titled Beyond the Hype of a New Approach, a cautionary tale expressing many of the concerns I had then and still have, at the time in response to Jim Highsmith’s Agile Project Management book and the corresponding movement. Then and now, a lot of good ideas are being thrust upon us in a manner that will cause the mainstream to cringe rather than embrace. We need to understand the market before we can sell the product.
The argument is not that the ideas in Agile are bad, but that we technologists thrive in a Boolean world while reality is analog. The best answer before we have all the data really is “it depends.” Euthanize the dogma, please. Don’t get me going on the suggested prospect for a Unified Agile approach. It’s not going to happen.
There are a couple of things I keep noticing in the software world, a big one of which is a desire for simple, straightfoward, “easy” approaches that allow practitioners to turn their brains off and follow the process.
I have a real problem with that, because software development is knowledge work, and “easy” approaches make the process responsible for the outcome, not the people. That might work fine at McDonalds, where everything is consistent (just not very good), but do you really want software like that?
Sadly, where there is a market, businesses will fill it. So not only do you get Agile Dogma (mouthing agile buzzwords and assuming they will always work without thinking), but you get Agile pushers, who are more interested in the advancement in the prominence of “Agile” than in any lasting contribution to the field.
My response to the “Give me a cookie-cutter soltuion” question is, well, long. I like to try to start a dialogue. After all, it takes a very long time to say “Life is pain highness. Anyone who will tell you otherwise is selling something” without losing a client.
In the end, the reality is that software development is hard, and the cream will rise to the top.
I wouldn’t have it any other way.
Shark Jumpin’ – II
Ben Edwards commented yesterday that in TV Shows, the characters age and the writers need to introduce new characters or plotlines to keep things interesting. Those things may make the show jump the Shark, argues Ben, but Agile has “just been around for a while and is gaining followers ad people, tired of the old ways of doing things, are looking for something that works better.“
There’s nothing wrong with that, and I applaud it, but I’d like to take a few moments to talk about the system effects of a mass movement.
First of all, it’s now much less of a career risk to pursue agile development. Lots of companies are doing it, and “Agile RUP” or “Agile CMMI” is far less threatening than Extreme Programming. Automated Unit Tests, TDD, and xUnit frameworks are hitting the early mainstream, and vendors are adding refactoring tools to IDE’s like Visual Studio.
Second, the original Agile movement was a re-action to the heavyweight, documentation-centric processes and methods of the 1980’s and 1990’s. More than a few people noticed that it’s really hard to move a heavy boat, and that extra documentation adds mass. They also noticed that the “Crystal Ball” of a project beyond 90 days doesn’t work. So a bunch of guys got together at the snowbird conference and suggested lean artifacts, planning in short increments, and adjustment.
There are several problems that the agile manifesto just plain punts on: For example, the idea that you can be on-time, on-budget, high-quality, feature complete. Forget about it – at the beginning of a large project, the customer doesn’t even know what features he wants – who are we kidding?
But … there’s a problem. Lots of companies want to be able to predict all that stuff, to the point that it is better to be certain and wrong than to be uncertain. I have experienced this first-hand, and DeMarco and Lister comment on this in “Waltzing With Bears.” Another Example: Extreme Programming doesn’t have a concept of “architecture” or a role of Architect. It simply doesn’t address the whole, er, problem space, that, um … Enterprisy-Architecty-Modelling-y things address. (Whatever)
By “punting” on problems that it can’t solve, the agile manifesto makes it possible to deliver great software regularly with considerably less waste.
The problem is that by saying “Embrace Change”, we are also saying “Get over your fear of loss of control”, and there are a whole lot of people in this world who don’t want to. They want to be told that they can have their pie and eat it too. And they have titles like VP of Development, CIO, CEO, or CTO.
This means there is a market, with money, who want to be told how they can have all this agile stuff and also have CMMI, or Architecture, or Portfolio Management, Long Range Planning, or a Crystal Ball.
In fact, one of the consistent things I hear on software discussion lists is “We want to be agile but how do we solve problem X.” Where problem X is something that is addressed by one of the technologies above.
For the first couple of years, the answers I saw on the lists were consistenly something like “Gee, we’ve been doing agile for two years and never had a problem with issue tracking. We talk about it at the standup, and we fix it.” Eventually, though, I started to see answers more like this:
“Yes, my company thought of the same problem before we switched to Agile, so we use AgileBugTracker, by SuperAgileSoftware. It’s great!”
It shouldn’t be surprise; Adam Smith tells us that someone will start a business to exploit that opportunity.
So we get Agile RUP and Agile CMMI and Agile Portfolio Management, Agile Issue Tracking and Resolution, Agile Systems Architecture, and get slowly pulled back into the world we were trying to escape from.
The tent is too big and we’ve given credence to ideas and concepts that we should not have, to the point that it is very hard to tell “good Agile” from a bunch of consultants that can’t ship software but know the right buzzwords.
Personally, I am a member of the American Society for Quality, and I have read Crosby, Drucker, Juran, and Deming: I went through the 600-page books and know the difference between “Getting It” and using the buzzwords. And I have real concern that Agile is in danger of becoming TQM or Six Sigma: Inherently good but misunderstood more often than not. That is what I mean by jumping the shark.
Next time: Agile CMMI, Jim Brosseau’s comments, and more …
Agile Jumped the Shark? – I
On Happy Days, there was an episode where Fonzie jumped a shark tank on his motorcycle. Many people consider that the “high water” mark of the show, and believed that once the show reached that pinnacle, it had no where to go but down. There is even a website, JumpTheShark.com, devoted to chronicalling when TV shows start to go downhill.
That said, I’d like to share a few facts:
- Agile was originally a cosortium of a bunch of like-minded groups: Scrum, DSDM, Crystal, and Extreme Programming. The goal was to make a bigger tent.
- The tent is getting bigger each year. You can now google for “Agile RUP” or “Agile CMMI” and get a considerable number of results. The agile conference grew to 1,100 people last year and I believe is predicted at 1,500 this year. Instead of a band of rebels, it’s now ‘hip’ to be agile. (That means it will attract an entirely different group of people than it did four years ago, when the only people who knew what agile were active readers of the wikiwikiweb.)
- Jon Kohl and I have been concerned about the state of the agile world for about a year now.
- Yesterday, Elisabeth Hendrickson posted Inside the Secret Fears of Agilists, which stated as a top concern that the big global outsourcing companies would adopt Agile as a buzzword, just like Service-Oriented, Business Intelligence, Global Sourcing, and ‘Enterprise’
- My friends who are certified scrum masters are now using the term “use what works” to justify studying for the PMP exam
- Yesterday, my collegue Paul, the web manager, threw his copy of XP 1st Edition into the trash can, telling me “We’re doing back to BUFD” (Big Up-Front Design)
There are a lot of reasons this is happening, and I would like to discuss it, but first I’d like to say that I think Jon and Elisabeth are spot-on. Go read them.
As for me, more to come later.
XP, Dear XP …
Extreme Programming in Japan?
You’ve got to see this video.
Don’t worry, it has subtitles, and it’s only three minutes long.
