Matt Heusser’s Blog
Testing at the Edge of Chaos
The Fishing Maturity Model
Imagine, for a moment, you own a fishing company. Perhaps it is in Louisiana, something like Bubba Gump’s Shrimp Company from the movie Forrest Gump.
What you do is very simple; every day, you go out, cast your nets, bring in a few hundred pounds of shrimp. Minus the cost of gasoline and maintenance on the boat, you can “make a good living.”
But the Blue family were not professional shrimp fishers; they were shrimp cookers.
Enter me, the Management Consultant, and my fishing maturity model.
I point out that you’ve been running your business in an ad-hoc fashion – just running out into the water and dragging the nets. Why, you don’t know if you’re doing well or not, and have no plan for improvement. I’d like to help.
The five levels of the fishing maturity model:
1 – Ad-hoc. Fishing is an improvised process.
2 – Planned. The location and timing of our ships is planned. With a knowledge of how we did for the past two weeks, knowing we will go to the same places, we can predict our shrimp intake.
3 – Managed. If we can take the shrimp fishing process and create standard processes – how fast to drive the boat, and how deep to let out the nets, how quickly, etc, we can improve our estimates over time, more importantly.
4 – Measured. We track our results over time – to know exactly how many pounds of shrimp are delivered at what time with what processes.
5 – Optimizing. At level 5, we experiment with different techniques; to see what gathers more shrimp and what does not. This leads us to continual improvement.
Sounds good, right? Why, with a little work, this would make a decent 1-hour conference presentation. We could write a little book, create a certification, start running conferences …
The problem:
I’ve never fished with nets in my entire life. In fact, the last time I fished with a pole, I was ten years old at Webelo’s camp.
You see, I actually have no idea what I am talking about. I made it all up by inserting a process improvement framework over a specific activity. It’s Gunk; it’s Garbage.
Or is it?
The beauty of Maturity Model Mania (the MMM) is that it’s a non-falsifiable argument. If you boil it all down, all it’s saying is do the same thing every time, measure what you do, experiment, and then do what works best. How can you argue with that?
I’m not going to argue with the idea that evaluation when picking changes to technique is good. But that’s not what we’re talking about; we’re talking about a specific framework that, when you get down to it, introduces a series of wasteful and expensive processes that don’t really pay off until level five.
I would argue that, as testers – and quality experts – or heck, anyone doing knowledge work, it’s our job to not be fooled – not just by the software, not just by the project plan, but even by the people in suits with fancy ideas and power point. If Forrest Gump wants to run a successful fishing business, he probably wants to listen to just enough of the idea to see what parts of it apply, then use his own good judgement and discernment on what to implement. (On the other hand, if the guy actually fishes every day and has been wildly successful, you might want to listen to him.)
There are a lot of bogus ideas in software development. This is just one little reminder to trust your gut; don’t be afraid to say “This whole thing smells fishy to me.”
Comments (12)
1 Trackbacks/Pingbacks
at October 8, 2009, 2:07 pm:
Matthew,
Excellent post, as usual. [I'm glad you like it. Thank you.]
You raise very good points. Testers (and other IT executives) should be leery of snake oil salesmen and use their judgment about “experts” who lack practical hands-on experience. While I completely agree with this point, I offer up my own experiences as a “counter-example” to the problem you pointed out here.
3-4 years ago, while I was working at a management consulting and IT company, (with a personal background as an entrepreneur, lawyer, and management consultant – and not in software testing), I began to recommend to any software testers who would listen, that they start using a different approach to how they designed their test cases. Specifically, I was recommending that testers should begin using applied statisitics-based methods* designed to maximize coverage in a small number of tests rather than continuing to manually select test cases and rely on SME’s to identify combinations of values that should be tested together. You could say, I was recommending that they adopt what I consider to be (in many contexts) a “more mature” test design process.
The reaction I got from many teams was, as you say “this whole thing smells fishy to me” (or some more polite version of the rebuttal “Why in the world should I, with my years of experience in software testing, listen to you – a non-software tester?”) Here’s the thing: when teams did use the applied statistics-based testing methods I recommended, they consistently saw large time reductions in how long it took them to identify and document tests (often 30-40%) and they often saw huge leaps in productivity (e.g., often finding more than twice as many defects per tester hour). In each proof of concept pilot, we measured these carefully by having two separate teams – one using “business as usual” methods, the other using pairwise or orthogonal array-based test design strategies – test the same application. Those dramatic results led to my decision to create Hexawise, a software test design tool. [Point Taken ...]
My closing thoughts related to your post boils down to:
1) I agree with your comment – “There are a lot of bogus ideas in software development.”
2) I agree that testers shouldn’t accept fancy PowerPointed ideas like “this new, improved method/model/tool will solve all your problems.”
3) I agree that testers should be especially skeptical when the person presenting those PowerPointed slides hasn’t rolled up their sleeves for years as a software testing practitioner.
Even so…
4) Some consultants who lack software testing experience actually are capable of making valuable recommendations to software testers about how they can improve their efficiency and effectiveness. It would be a mistake to write them off as charlatans because of their lack of software testing experience.
[I agree with the sentiment that sometimes, people out of the field can provide insight. I even hinted at that with the comment that at least, Forrest should listen, then use his discernment on what to use. I'm not entirely ready to, as the expression goes, throw the baby out with the bathwater.]
5) Following the “bogus ideas” link above takes readers to your quote that: “When someone tells you that your organization has to do something ‘to stay competitive,’ but he or she can’t provide any direct link to sales, revenue, reduced expenses, or some other kind of money, be leery.” I enthusiastically agree. In the software testing community, in my view, we do not focus enough on gathering real data about which approaches work (or -ideally- in what contexts they work). A more data-driven management approach would help everyone understand what methods and approaches deliver real, tangible benefits in a wide variety of contexts vs. those methods and approaches that look good on paper but fall short in real-world implementations.
[Hey man, you can back up your statements with evidence, and you're not afraid to roll up your sleeves and enter an argument. I may not always agree with you, but you're exactly the kind of person I want to surround myself with, to keep each other sharp. Thank you for the thoughtful and well reasoned comment.]
- Justin
Company – http://www.hexawise.com
Blog – http://www.hexawise.com/wordpress
Forum – http://testing.stackexchange.com
*I use the term “applied statistics-based testing” to incorporate pairwise, orthogonal array-based, and more comprehensive combinatorial test design methods such as n-wise testing (that can capture, for example, all possible valid combinations involving 6-values).
at October 8, 2009, 2:19 pm:
Matt,
Nice article. Very similar idea to one I have in my drafts but mines to do with food shopping and not fishing
.
There are too many new ideas, best practices and models to adopt which causes a huge amount of waste but more importantly, creates a vast amount of confusion. And this confusion not only leads to lost time and money but the endless arguments we see on forums over what terminology, phases, exit criteria and methodology we should use. It’s where the ’standard’ and ‘certification’ people have a field day.
Nice post.
Thanks
Rob..
at October 8, 2009, 2:23 pm:
I like the analogy and it’s an interesting point about maturity models being a non-falsifiable argument. I think that’s true so long as the argument in favour of them is kept at the superficial level you’ve presented. However, I think that if the argument in favour is expanded to give just a bit more detail then assumptions start to appear that can be challenged and rejected.
E.g. are there implicit assumptions that all activities can be defined as processes, and that all processes can be considered of equal significance, and of equal relevance in different organisations? Saying yes to these assumptions shows that you don’t really understand what’s going on.
Saying no acknowledges that the experience and judgement of the professionals at the front line have to be taken into account, and that they’ve got to consider the context, the specific circumstances, in which they’re working.
If you acknowledge that then the underlying framework of a 5 level maturity model starts to fall apart. What’s the value of working to tick off all the boxes at each level?
I need to give this a bit more thought. Thanks for stirring the brain cells Matt.
at October 8, 2009, 2:28 pm:
Matthew,
Quick follow-up to my 5th point, above: an article I co-wrote with some good, hard data supporting the assertion that combinatorial test design methods like pairwise have worked in multiple diverse real-world projects to improve efficiency and effectiveness:
http://csrc.nist.gov/groups/SNS/acts/documents/kuhn-kacker-lei-hunter09.pdf
Thanks again for posting. It’s thought-provoking.
- Justin
at October 8, 2009, 2:44 pm:
Best Post Ever! [Well, thanks Lanette. I can appreciate the enthusiasm..]
Did I mention on the advice of someone I respect at work I joined ASQ. I hang my head in shame that I am a member who paid money for their publication and each one that arrives terrifies me more.
[Hey, I'm a senior member of the ASQ. Who can be against Quality? What we can be against is brain dead, process-and-paperwork-isized interpretations of Deming, Juran, Crosby, and TQM. I think the ASQ has potential, and I'm in the fight. But I certainly can understand your position when, once a year, they pass the offering plate. Appropos of nothing whatsoever, you might like the new DeMarco/Lister book. ]
I read this week AGAIN that the quality at Toyota is slipping yet again, still people copy that variation on a theme endlessly when it isn’t even working in this context where it started.
[Again, I /like/ lean manufacturing and the Toyota Production System. As I see it, the problem comes when you try to "go agile" or "go lean" (whatever that means) without being willing to adapt your world view. Try that, and you get Cargo Cult Software Engineering. (Also as I see it: A large chunk of the "Lean SW Dev" literature bears little resemblance to what my understanding of lean would actually be. Tread Carefully; I think the Poppendieck's "get it"; I'm not so sure about the 2nd tier consultants).]
Six Sigma companies are doing NO BETTER than those who have not wasted the time and money on it.
[Part of my graduate Cohort did their capstone on Sig Sigma for Software Development. I'd be happy to have a long and thoughtful discussion on it. My take away is: (A) It has specific, narrow application for software, and (B) You can't just graft some numerical tools onto a broken worldview, DANGNABIT! That's not going to work. I'm getting too excited for this ...]
The part about the process improvement snake oil that is scaring me the most? Not picking one! Not jumping off the train when it is heading to purgatory? And firing the people who ask, “Hey guys, why are we in this hand basket and where is it headed on a greased pole?”
[The good news is, companies that take this approach tend to end up with what they deserve. Some have enough institutional inertia that it takes a few decades.]
These aren’t harmless ideas. These ideas are COUNTER to the essence of human nature.
[Oh, no, I disagree. People doing silly - but easy - things in order to claim success and get a "exceeds expectations" at the end of the year? People doing what is easy for their careers at the expense of the organization? I'd suggest to you that Original Sin is the most empircally verifiable of the Christian Doctrines. But it doesn't have to be that way. (And I think you mean counter to good work, which I agree with.)]
Why? When I’m logging and planning what I do everyday to a micromanaging process application I am NOT TESTING. It’s simple. I have X numbers of good thinking in a day. X minus your bullshennanigans= less for what matters. Prove it helps first. Prove it nearly ALWAYS helps in real life first. Then you can sell it. Not before. Before it is a theory, not a valid process improvement. We need good theories, but then we need to test them in a valid way and prove or disprove them, not while creating havoc on companies, developers, testers, managers, and worst of all software users who get the junk turned out during “Implementing Fishing Maturity Model”.
[Indeed. I've got a process improvement method designed to maximize time spent butt-in-seat, hands-on-keyboard, actually testing. I'll be talking about it at STPCon at a speed geeking session, and maybe blogging about it afterward. Thank you for the comments!]
at October 8, 2009, 4:52 pm:
Excellent!
I’m expecting a follow-on to this post entitled “FMMI – The Fishing Maturity Model Integration”.
To paraphrase Kenny Bania: “That’s gold, Matt! Gold!”
at October 8, 2009, 4:55 pm:
You say “How can you argue with that?” But, it’s easy to argue against it, because no evidence has been offered to suggest that such models actually help. A nice sounding story is not the same as a sound argument.
[Well, I do take that as rhetorical position, then go on to argue with it James, as I would assume the reader can tell quite clearly when he reads it.]
As for Justin’s point, the real problem there is that Justin, not knowing much about testing, is telling us that he’s not qualified to evaluate whether his methods actually improve testing. It may be that he is optimizing for certain shiny and pretty variables while eroding the true underlying value of the process. In fact, this is Nicholas Taleb’s constant complaint about financial analysts in his book The Black Swan.
[It's interesting that Justin's bug-counts are one dimension. Bug or not a bug; it does not include show stoppers vs. typos, for example. What I find helpful is that Justin is willing to debate and share his data, an intellectual position I accept as valid.]
Last night I spoke with an earnest but confused test manager who tried to tell me that counting test cases would tell her about the progress her team was making on their test strategy. While it is *possible* for that to be true, it’s more likely that the test count is telling her nearly nothing about test progress. She (temporarily) lacks the skill to observe and evaluate the work of her test team.
[Agreed. These kind of things are like straws to grasp. They create the illusion of control, kind of like a gannt chart might when you simply take the end date, divide by four, and create requirements, desgin, dev and test 'stages.']
The kind of maturity I’m interested in is the maturity of the humble scientist, who strives to be aware of the many sources and potentials for error in his work. The false rigor of maturity models is, to me, a level 0 practice. They distract us from learning. Leave them behind.
[I submit they these models are prevalent and popular enough to be worth discussing, and that finding weaknesses in them, as an intellectual exercise, is yet another way we can sharpen our calling as testers to 'not be fooled.' If memory serves, I first heard that term from James. I like to believe that one way to do that is with humor. Even nonsense can sharpen the intellect, but I have to agree with the sentiment that having these arguments over and over again slows down the actual advancement of our field.]
at October 8, 2009, 8:06 pm:
I understand the criticism presented here, as I have been a victim of “here is the solve all process” and “I can fix what is wrong with FMM.” Please see this not as a direct criticism of what I see as a good article and a very valid point. I think this idea needs some fleshing out.
The real problem is not that FMM has no value or that its processes are inherently wasteful. It is not that the Consultant does not understand Fishing with Nets. It is because this Consultant does not take the time to understand the wants and needs of Bubba’s family business. Another problem is that Bubba has no desire for or understanding of improvement for his family business. The Consultant applies FMM mechanically, without any real understanding of how that tool might be used to actually help without adding waste and so it has little or no value to Bubba.
I am not a lover of highly detailed process definition. I will be the last to say, “what we really need here is FMM level three certification”. I do love when someone can give me their outside view of what is happening, comes up with valuable tools (including, possibly FMM) to resolve and improve the work I am doing, because I want to improve the process. What is fishy to me here is the process slave.
I do also agree that there are too many suits selling books and stating the obvious solutions to process problems (I have yet to find a valuable process / business book that did not tell me what I already knew. I have yet to leave one of those books / seminars having applied nothing.).
[Dan, one expression I've heard is that the best moral teachers all tell us truths that, on some level, we know are true - but we do not act as if they were true. I think it's the same here. Not only did I get a lot out of Stephen Covey, I periodically re-read his works and evaluate if they are impacting my behavior. So, sure, some guru's spout cliches and obvious things - but they might just be worth listening to anyway.]
However, there are many people (me included) that are too close to their problems to see the obvious solution. I know we have all been there. Often it takes someone that thinks deeply and simply about solving problems to give us that push to simplify and improve.
[I also agree with the sentiment that sometimes an outsider can help us connect the dots. I hope my readers did no take my statements to this sort of logical extreme.]
I am currently working with a Project Manager that is CMM certified and presented himself as no great lover of the agile process we were using on the project. He has been very flexible and his understanding of CMM has been incredibly valuable for identifying places where we could eliminate waste and improve our agile process. We have not become CMM slaves, we have eliminated waste and he has come to appreciate what Agile has to offer.
[Dan, my humorous post was meant as a light metaphor; notice I didn't pick on ANY maturity model by name, and some are better than others. Of course, CMM can be done better or worse, and, distilled down to it's simplest form, it's ideals about observe, experiment, and improve are things I personally agree with philosophically.
I can't help but notice that you say the Project Manager was CMM certified. How exactly does a single /individual/ become CMM certified? Is he a SCAMPI-Cerified lead /appraiser/? Or something else?]
at October 9, 2009, 4:02 pm:
I misspoke about the certification. It was part of an edited sentence that obviously turned out badly in spite of my editing.
My point was that this guy is sold on CMM and often speaks about how it is misunderstood. I attribute his knowledge of process maturity to helping us cut the wasteful work and unnecessary process from our project work, in spite of our perceived maturity level.
I was interested to see that the FMM was more of an ontology than a process. That is my limited understanding of many of the maturity models – they tell us little about how and much about what. I do see many that turn such models into a rigid, wasteful process (”FMM compatible processes”), though I would also point out those that use them as effective tools. I just was not seeing that brought out in the metaphor, or in the following comments. Though I saw the process as the main complaint of the article, it seemed to point at the model as at fault. I see from these comments that was not the intent.
[I'm not exactly sure what you mean here, Dan, and taking guesses about interpretations on the intarwebs tend to get me in trouble. Do I think the folks the SEI 'got it wrong', mis-interpreting Deming/Crosby an failing to grasp key dynamics in development? Yes, I do. I can also agree with articles like this one, talking about how it is possible to do some good with a framework like CMM. I also respect your perspective and would like to hear more about it. I'll be in GR next Wednesday, likely lunch at Calvin College with a few people. Can I buy you lunch to talk about it?]
at October 9, 2009, 4:21 pm:
One thought about the FMM is that it might actually work, IF the right goals are set at the beginning.
[If you're saying it's possible for someone with good general systems skills to use elements of the FMM for success, sure. But if he's got good general systems skills, does he need the FMM at all?]
What’s the goal here? Fish as many shrimps as possible per day? Fish for 3 days a week, then lie on the beach for the other days?
My point is that the all important question that hasn’t been asked at the beginning is “What do you want to improve?”
Of course if you skip that question you can then apply it to many other questions or build a fantasy framework around it.
I’ll assume that this was one of your points?
[All good questions. Victor Basili at the University of Maryland has another framework, Goal-Question-Metric, that is, in my mind, at least a little bit more interesting, but it has it's troubles, too. Another day.
]
The reason why I think methods like Justin, the FMM and others can work is that, simply put, people sit down maybe for the first time and have a good look at what they’re really doing, think about what they want to change, then implementing a change, then observing if it has the desired result. The important bit is the CHANGE, not the methodology or even what the change is.
What Justin describes sounds reasonable. What I don’t know if reduction in time to create test cases and increased productivity (double the number of defects) is what was the goal was in the first place. Were the tests that were designed in this way applicable to the system under test or were the tests designed by the SME actually more worth for the final system in production? Please note that I’m NOT saying that the “applied statistics-based testing” approach was wrong or not worth it. What I’m questioning is if the value of both sets of tests are the same and which set has the higher one for the customer.
To get back to the FMM, of course it’s a lot easier to pick up a model that sounds reasonable, then follow it rather than trying to figure out the goals in the first place. It might bring up thing like “We wasted time and money in the last couple of years”, “I don’t have the skillset to implement that change so rather not mention that”, etc. The old with change comes fear. Implementing an “Acknowledged model” (whatever that is) one can ask for time, resource and training without too much fear and still make a change.
Oh, and last not least, nice blog! [Thanks, I'm glad you liked it.]
at October 10, 2009, 6:58 am:
Hi Matt,
I’ve since given this blog and even my own initial reaction of it more thought. You are right, I am glad to be part of ASQ and also to keep thinking of overall quality. I am FOR quality, and I’m also for process improvement, but I just want to be careful and see more scientific rigor in the application of process improvement.
[What do you mean by 'rigor', Lanette? Measurement? I suspect you mean 'actual thought and observation over cliche and platitude', and I'm with you.]
Not just doing it to see IF it will make improvements, and because it sounds good on paper, but applying improvements for meaningful reasons, for real company pain points rather than petty things like keeping up with the neighbors, or because Toyota did it well and it worked for them.
[Ahh, we're running into an incentives problem. Because ferreting out the real company problems - as opposed to doing some 'stuff' to get an exceeds on your annual evaluation, is a politically unpopular decision. Human nature being what it is, when doing what's right for the company interferes with personal advancement ... you can probably guess what usually happens.]
Sort of in the way that some people considered me “anti-automation” because I felt poorly designed step-by-step automation was frustratingly not uncovering useful bugs and also was not maintainable,
[Well, so-called step-by-step automation isn't actually automating what we human beings do when we test the software, is it?
I'm against bogus an anti-intellectual trivializations of the testers work. If that makes me anti-automation, so be it. Or, to put it another way - you show me your bug counts, I'll show you mine.]
so it wasn’t “adding” value, but instead stealing time away from testing, I think there is a big risk of process improvement done badly, or applied to the wrong areas being harmful.
[Sure. Part of the question is "who gets to define improvement?" I've seen process improvement initiatives designed solely to impress auditors. So actually velocity decreased dramatically, but the auditors sure were happy. I would submit: Your business doesn't get sales by making auditors happy.]
I think that a topic like this would make a good article submission, especially with a few more specifics and examples, because for me it was very thought provoking and it made me want to think more, evaluate more, and look more carefully before I jump into any process improvement boat to make sure that the process improvement is likely really lead to a meaningful and sustainable quality improvement. In some cases, I’ve been personally involved with process improvements which were unquestionably helpful and the right thing to do. Anyhow, if you did write an article on this, expanding on telling the Fishing Maturity Model from “just copy Toyota” to a real customized and adjustable and well implemented process improvement. Stagnation and failing to innovate in the area of process isn’t a good answer either, so how do we quality minded individuals solve process problems without running into the problems you mention here?
[Sounds more like a series of articles, or possibly a book to me. Email sent.
]