Matt Heusser’s Blog
Testing at the Edge of Chaos
Off to vacate
I’ll be in vacation for the next few days, but in the mean time, Paul Carvalho has an interesting post comparing skilled (exploratory) testing to fencing. I thought it was insightful; I hope you enjoy it.
Interview?
Software Quality Engineering recently sent me a bunch of inteview questions for a StickToolLook e-newsletter. Writing a reply was hard, as I wanted to provide real information that was worth reading, not just pontificate or repeat the same old thing. My replies are below; I’m interested in what you think.
SQE: What are the benefits of a tool developed for an entire market? What about a tool designed to focus on a particular need?
That’s a really tricky question, because within every market there is a smaller market. If we start with tools designed for everyone who ever touches software development, we can find tools inside of that. Say, tools for just testers – then just performance testers. Inside of that market are tools for web performance testers, java testers, and database testers. Inside of that we have tools for each specific flavor of database.
Overall, I like tools as niche as a I can get them, because the vendor can solve a specific problem and solve it well. The higher-level tools offer some level of integration and file sharing, and workflow that just can’t exist in a specific tool. Also, the more general a tool, the better chance there will be to find a user group, consultant, or support when things go bad.
SQE: What are the cons to using either type of tool?
Well, for example, if you use a general tool for all databases, calling your sales rep and ask about Oracle support, and he replies “yeah, there’s a driver for that.” That’s bad news. At best, you’re going to muck about for a day or three before finally getting the software to mostly work. At worst, it’ll go back on the shelf. General tools generally have a main market; like ordering the fish in a steakhouse, you never know quite what you’re gonna get.
I have heard of a few general tools that work in lots of environments; McCabe has amazing support and I’ve heard good things about Worksoft certify. You just have to be certain to check a vendors claims carefully – yourself.
Of course, if you use the specialized, Database-Specific tool and write custom code for it, then when you convert environments you might have to throw all that work away and start over.
SQE: Is it possible to develop a tool or a tool suite that can be all things to all people? Or even all things to one person?
I think it’s a matter of philosophy. UNIX, for example, is a collection of small tools that can be tied together in really interesting ways to do just about anything; I think the “Swiss Army Chainsaw” is my favorite term to describe it. Is UNIX a single tool or a collection of tools? (Grins) Who Cares?
There are certainly performance testing tools, record/playback tools, and development environments people spend all day in, and some of them have very passionate and active users. My colleague Mike Kelly is fond of the phrase “Role Players”, to mean that the person likes to perform a function and do it well. If someone wants to be an excellent performance tester for websites, then sure, that person could live in one tool, align himself with that vendor, and probably make a very good living as an employee or consultant. That just isn’t me, and you never know when a specialty is going to be destroyed – say, for example, when websites start to support wireless devices and the bottleneck for performance shifts.
SQE: On the other hand, can a tool be too specific?
Certainly! I don’t know anyone who uses awk or sed anymore, for example. Those were great little programming languages, but perl cam along and offered all the exact same features plus more.
Of course, if the tool is open source, it doesn’t really matter if the tool is too specific, because it’s free, and you can put it away and pull it out if and when you have the need again. If it’s for-purchase, the market will sort it out – the vendor will either expand the product or fail in the marketplace. (You have to love capitalism, after all.)
SQE: Which type of tool is more prevalent in the market right now?
I think vendor-supported tools tend to go big and open source tools tend to be small. The open source tools are often developed by a specific person to solve his own problems, and growing the tool won’t make him any money. So open-source tools grow through some other guy adding features that he needs. For-purchase tools are driven by companies that want to make money; they have an economic reason to try to reach as many people as possible. I hear a lot of good things about Visual Studio Team System, which is big, big, big general purpose tool. I think the market is still figuring itself out, but right now most of the new tools I see are specific tools for a small market, created by small companies who have expertise in that area – and I think that is very good thing.
SQE: Where do you see the market headed in the future?
I think software testing is an immature field, and software quality even less so. That means we need to try a bunch of different approaches, see what works, and then build up tools around that. Like I mentioned earlier, I see little companies doing just that. If they succeed, economic forces will expand those products a little bit, while hopefully the “big” products will get out of the clouds and come down to the real world a little bit more. Eventually, I predict a fight for the middle.
Context Driven
I posted this to the Software-Testing Yahoo Group yesterday, and I thought Creative Chaos Readers might enjoy this. The background is a post saying that context-driven thinking was universal, and there was no value in putting people into different boxes such as the “Analytic” (Academic, Telecom, Finite State Machines) school of testing or the “Factory” School (ISTQB).
In reply Scott Barber wrote:
*If* that is the case (and I’m not saying I agree, I’m just doing a thought experiment), then who is it promoting things like “Best Practices” and flaming things like Exploratory Testing? Is it managers? Developers? Tech Magazine columnists? Because *someone* is promoting these ideas and it isn’t anyone who believes in the context sensitivity of testing.
And I replied:
In my experience, when people are fighting XP and promoting best practices, they are often willing to pay lip service to the concept that software environments are different, and what works for you might not work for me, and vice versa.
Then they go back to the chalk board and continue explaining the rational, unified model thingy that applies to everyone. 
Seriously, it’s hard to argue with context-driven principles, and few people do. It is a lot easier to ignore them – and many do. (I think that’s allready been said on this list, so I labelled this post …)
Against Systems
As a military cadet, I had a few occasions to design systems – generally point systems. For example, the number of points required to graduate from a summer encampment, or a merit/demerit system.
I typically would write a page that gave guidelines and concluded with “Plus or minus (some big number) for items of exceeding excellence or discredit”, and failed to meet expectations. What my superiors wanted was a complex, detailed, organized, predictable system. They wanted something comprehensive.
That always amazed me. First of all, if that could be done, someone would find a way to game it. Make Public Displays of Affection (PDA) a 1-point demerit, and some cadet would end up embracing his girlfriend during pass-in-review, collect the demerit, and reply “it was worth it.” OR, for encampment graduation, a cadet could do the absolute bare minimum to graduate, reflecting a negative attitude that was undeserving, while a handicapped cadet (we had a few) might do his best but not quite make it.
It also happened on promotion forms. You would have a cadet that just didn’t get it, but you’d be forced to use the CAPF 50 (leadership eval form) to do an objective evaluation. Sadly, “Gets it” level was not on the sheet, so the overall score would be too high. What do you do? Pass the cadet, or systematically mark him down in everything to fail, or spend a hour trying to figure out how to “fairly” complete the feedback form, with accurate feedback, that resulted in the outcome you desired?
This problem isn’t just limited to me. One of the themes of the movie Thirteen Days, which is about the cuban missle crisis, is the desire of the Military Establishment to escalate the cuban missle crisis to war. To do this, they get the president to agree to a set of “rules of engagement”, then try to use brinksmanship to escalate the level of conflict until US Troops are in harms way. Then the rules of engagement would require a counter-attack to “defend” out troops.
I’m all for decision support systems (DSS). There’s a diffence, however, between a DSS that provides information and one that makes the decision for you. As a Decision Maker, why would you force yourself into a system that limits you? Why develop a system that takes over, limiting your ability to use common sense and good judgment? Why would you want that?
Often times, there are some perfectly good reasons for this. Some ERP systems, for example, can do a better prediction of trends than a human can, thus limiting the amount of excess inventory that needs to be carried but ensuring the shelves are stocked. In other cases, like hiring, there is a real risk of being sued for picking one person over another. Objective systems that decide for you, say for example, a point system, will limit legal risk.
Or, I dunno, say you are running a conference and you want to provide feedback to the people you rejected. Having a templated form that every member of the committee fills out that you can average looks a lot more impressive than saying “Well, we talked about it, and you didn’t make the list. Try again next year.”
But, there’s a problem, and that is this:
Most first-pass objective systems suck. They really, really, suck.
I’ll say it again: Most first-pass objective systems suck.
I’ve known this since childhood, at age 15, when I tried to make my first merit/demerit system, but I did not find out why until much later. Wording my explanation is hard, but I’m going to try, so please bear with me:
1) Modeling system effects is hard. When you reward something, you get more of it, but you get more of exactly what is measured. You want people to get leadership training, so you give them points for taking the class and they are going to take the class. But you really want them to learn about leadership. Did that happen? Maybe. Maybe not.
2) The more complex the system, the more variables. Modeling a 2-hour-a-week-plus-some-weekends-and-encampments military environment, with the same rough goals and training schedule is hard. Imagine the workplace, where each project is different!
3) The more variables, the more interactions, reactions, and unintended consequences.
In the 1970’s and 1980’s, the great solution for this was going to be Artificial Intelligence (AI) and neural networks; computers that could learn. It’s actually pretty easy to build a system in LISP that can look at what books you like, find other books that other people like, and make recommendations. When you are dealing with a closed system with a few million books, it’s pretty easy.
Real life is not a closed system
It turns out that it’s easy to make a CASE system that requires a requirement for every check-in, or a requirements template to be complete before coding begins.
But the judgment that the template is filled out well? That is best done by a human. It is very hard to make an AI program capable of assessment, or any of the higher levels of bloom’s taxonomy, except in very specific applications, in which case the computer is really just parroting back what some human told it.
Why I am talking about AI?
Because most processes and systems are really really bad AI – AI programming in a vague and ambiguous language that is much closer to BASIC than LISP.
This is a huge part of the problem that CMM(I) and ISO 9000 have. They want to be one-page descriptions that say “Do the Right Thing” or “Do Good Work”, but you need to define “Good” and “Right”, and to try to do that a crappy language like English which is worse than BASIC, while dealing with all of the variables in software development is, well … hard.
To my knowledge, there is only one computer than can synthesize all this information, and it is the human brain. The role of the human brain is to make sense and integrate the world around it. If you’ve ever had intuition, or a gut feeling, you know that can be a lot more than emotion. It can be the left and the right sides of your brain working in concert to solve a problem on the subconscious level. And where process descriptions fail, the human brain can be surprisingly good at solving problems. (For example, to quote Michael Bolton: “If your project has dug itself a hole, your process ain’t gonna pick up the shovel.”)
The job of collecting, synthesizing, and making a judgment is a craft. Like art, writing, development and testing, judgment can be improved with practice. In future entries I would like to explore a few of these exercises.
For the time being, here’s my $0.02: Be skeptical of systems that spit out answers about behavior and judgments. Ask questions about the weighting.
Remember this: If you are a decision maker overseeing such a system, it can exist to make the decision for you. To quote Richard Bach:
“If it’s never your fault, you can’t take responsibility for it. If you can’t take responsibility for it, you’ll always be its victim.”
The Software Concept Maturity Model (SCMM)
Here’s a gift for the holidays; a software concept maturity model to lighten your mood.
Level 1 – Initial
At this point, someone has an idea. It seems to work for them, to solve the problems they have. The person actually uses the idea with some success. There are lots of ideas at level 1 right now, but we’ve never heard of them.
Level 2 – Defined
Someone gives the idea a name. Exploratory testing is probably at this level right now, although there are a few tools starting to emerge, and that is the leap to level 3. “Blink” Testing is probably at this level now. The key thing about level 2 is that once an idea has a name, it will spread.
Example of jumping from level 1 to level 2:
Level 1: “As a final check, Joe eyeballs the data, looking for irregularities. He does this mind-meld thing where he takes a step back and looks for patterns.”
Level 2: “Our shop does blink testing as part of it’s quality process. Does yours?”
Level 3 – Vendorified
At this point, the idea is popular enough that people start to do it, and ask for tools to make it. Vendors begin to provide tools to make the idea easy. The Enterprise Service Bus is probably at this level.
Level 4 – “Gotta Have It”
By level 4, vendors have invested a significant amount of time and effort in creating tools. They now create a market for the product. Instead of telling you the ROI of the product, they tell you that “everyone else is doing it” or “In Six Months, companies that aren’t doing X will be unable to complete.” Enterprise Resource Planning (ERP) hit level four about three years ago. Web Services and CRM are probably at this level.
Not to be a jerk, but level 4 often involves marketers and executives who don’t really understand the idea at the nuts-and-bolts level.
After level 4, the tree splits …
Level 5A – Ubiquitous
After all the hype, it turns out that the concept doesn’t solve all your problems and never will. Instead, hopefully, it has some broad specific applicability that solves a lot of problems like X, and can be used by people when the concept makes sense. Object-Oriented Development is probably at level 5 – People use it that use it, people that don’t don’t feel bad about it. Sure, there is a design patterns community and classes on Object-Relational Mapping, but no one is saying you “have” to use design patterns. (Any More) Agile Development is pushing it’s way from level 4 to 5A right now, but I don’t have a good feel for what is happening to test driven development.
Alternatively, some ideas go to level 5B …
Level 5B – Craptacular
Some ideas do not have broad but specific applicability and do not solve problems very well, yet a small group of companies and consultants continue to insist that you “just gotta have it” or that it is the wave of the future. RUP, UML, and some aspects of MDA are probably level 5B. It is interesting to note that many level 5B ideas are re-labelled or re-named to something more “hip”, then re-cycled at a lower level. Examples: The Agile Unified Process, The Enterprise Unified Process, Agile Model Driven Development, and HeavyWeight SOAP Web Services, which in many ways is dressed up CORBA. I am not 100% sure about EUP and Agile UP, though, as they may be part of some elaborate joke. ISTQB testing is probably level 4, going to 5B.
Note that this is just a model, and it’s incomplete. For example, several of the standards set by the Object Model Group have no reference implementation, as they were never actually used in the first place, skipping level 1 entirely.
What is my point, you ask?
Well, in the 1970’s a gentleman named Mark Hanan wrote a really interesting book called “Consultative Selling.” In the book, he suggested that instead of traditional selling, which he calls “Vending”, companies should help the customer solve it’s problem and increase bottom line performance – the ROI argument. When you see big, full-color ads talking about the 500% ROI from test automation, someone in the line of inspiration probably read Hanan.
The problem comes when people stop trying to understand the problem and really solve it, and instead ask “How can I use this (solution) to make money?” – then it turns into a solution seeking a problem, and you begin to climb the ladder toward craptacular. At best, you hit 5A, which really isn’t that bad, OO has really helped some people solve real problems in the real world.
And, know that I think about it, that is the danger with Agile right now, the Shark that could be jumped.
As for me, I intend to try to stay in the Agile Community, to make sure we go to 5A and not 5B.
I suppose this was a little bit more than a joke post, after all.
References, Holidays, and Enterprise Architecture!
First of all, Agile Shark Jumpin’ got a reference last week on InfoQ.com. Nifty!
Second, I am taking a short sabbat from Creative Chaos over the next few days. There may be a couple of posts next week, and more serious stuff after the new year.
When we come back: More from Jim Brosseau, I will discuss enterprise architecture (a little), and perhaps maybe finally deconstruct blue man group.
I’m no good with sappy Christmas sign-offs, so I’ll just be honest. I’ve had an awful Advent. It’s _supposed_ to be a season of preparation for the coming of Jesus Christ. I haven’t prepared. I haven’t prayed enough. I skipped Bible Study. Lately, I have even been short with the kids.
Here’s the thing – knowning all that, he’ll still come for us anyway next week – and knowning full well what he will do for us on Good Friday.
I suppose I do have something to celebrate, after all …
Four Schools of Software Testing
Most readers of “Creative Chaos” probably know that allthough I do all things software, my predominant focus is software testing. In fact, a few weeks back Elisabeth Hendrickson labelled my blog “Mostly Testing”, and I was vaguely annoyed that I did not make her “Mostly Agile” list … then again, seeing how my thoughts on Agile Jumping the Shark played out, I should probably be flattered.
I hope that it is no suprise that I ascribe to the Context Driven School of Software Testing. Brett Pettichord has a presentation on the Four Schools of Software Testing which provides some more background on the competing ideas in software testing.
James Bach posted this yesterday to the context-driven yahoo discussion group; It matches my feelings relatively well, and I thought it was worth sharing:
I first attacked the CMM because they labeled my community “Level 1: Initial” and called it “heroic”. The first label was clearly not descriptive of what we were doing at Borland, the second label *was* descriptive, but they seemed to think heroism was a BAD thing. This caused me to speak out in favor of my school, and to seek better ways to decribe it. For a time I called it “Market-driven software engineering” but that didn’t stick. Then I called it the Cognitive paradigm (as opposed to the Clerical paradigm), but when Cem suggested context-driven, that captured it better, for me.
I think the strategy of some consultants, such as Rex Black, is to label their school in such a way as to encourage the belief that their school is the only school– and thus that there is no controversy. I think that’s why, in Rex’s certification advertisments, he writes as if his ISTQB school of “elite” software testers represents the pinnacle of testing achievement for all of us, instead of a peculiar formulation that looks to my eye like an example of sloppy Factory school thinking.
To me, to deny controversy is terribly arrogant. I may have a different view of humility than Rikard, but to me I express humility not by saying I don’t have a strong idea of what is right and wrong, but rather that I have a strong idea, that my strong idea may be mistaken, and that I strive to encourage the kind of criticism I need in order to improve it.
I’m upset with people like Rex, who fight for the ascendency of their faction in the wider community of testers while pretending not to be part of any faction. That turns it into a propoganda war instead of a straightforward debate. It’s trying to manipulate people instead of persuading them.
I want to see more use of the schools concept so that we can have a free market of ideas where consumers can make informed choices.
If you don’t know much about the Context-Driven School and would like to know more, you can subsrcibe to the Yahoo Group. We also talk about context a fair bit on SW-IMPROVE, my email discussion list, where the volume is considerably less.
Two Roads in a wood – III (Tone)
Brian Marick once wrong something to the effect of “Methodology design is an extension of personality.”
There is a large percentage of the population to who having things be stable, predictable and repeatable are very important. They tend to be attracted to, and succeed it, larger companies with established “ground rules.” They tend to struggle in smaller, start-up, and entreprenurial enviroments.
And, to turn a phrase, that just ain’t me.
In the methodology discussions in the 80’s and 90’s, the stable, predictable voice was the loudest one in the room. Interesting exceptions that were’nt quote as loud were little voices like Extreme Programming, The Open Source Community, Jerry Weinberg, and DeMarco/Lister.
Stable, Predictable, Repeatable, Measured does yield a certain set of results. I’m just not sure that everyone wants that, and I think it should be okay to say that. I’ve drawn the images in “Two Roads” in a certain way to make the distinction clear.
My goal is not to insist that one is right or wrong, no, it’s to help you decide to make a concious choice.
To help with that, I’ve drawn a characture, but the loudest voice in the room has been doing that for twenty-five years now, so I hope you’ll allow me the opportunity.
Two Roads in a wood – II
“Deal or no deal?”
“Is that your identity?”
“Let’s play one verses one hundred!”
“Is that your final answer?”
These catch-phrases all come from simple, formulaic game shows. The shows are the same, every time, right down to the catch-phrases and witty banter. How many different ways are there to say “And we’ll find out, right after this short break” anyway?
You might even say that these game shows are stable, predictable, and repeatable. Moreover, the plan generally works – find a formula that brings in viewers and stick with it until the formula is no longer profitable. Whether it’s asking increasingly challenging questions for more and more money, like the latest gameshows , or having two teams compete and kicking a member off the losing teams, like most reality shows, you basically just do the same thing again and again, and it works.
… but imagine what it’s like to manage Saturday Night Live. That show is comedy; it is new every morning. Comedy shows are funny and entertaining when they are different every time. In that world, the challenge moves from picking a formula and finding sponsors to hiring and inspiring great talent. Late-light shows like Leno and Conan are somewhere in the middle; they have a packaged format, butdifferent guests. A predictable opening monologue, but nobody likes to hear the same joke two nights in a row. Or, twice … ever.
Of the two, which is software development? Well, probably, both. Somewhere out there there is a database-backed website company spitting out the same templated solutions, again and again, and somewhere else there is a company doing things that are “new every morning.” Most companies that develop software are somewhere in the middle between the two, stuck somewhere on the great spectrum.
The bigger question is: What kind of company do you want to work for, and would your organization benefit from moving closer to (or further from) one of the two poles?
“Deal or No Deal” is not going to have a great positive cultural impact, but it could make a lot of money for NBC and small group of writers. Saturday Night Live, on the other hand, has helped define popular culture for twenty-odd years (for good or ill), including The Blue’s Brothers, Wayne’s World, and a Dozen More.
Saturday Night Live is a talent incubator; it brought us Chris Farley, Adam Sandler, Tina Fey, and countless other talented comedians, given them a stage to perform on, and support and ideas to improve.
Picking one of these strategies as an individual can be even easier than doing it as a CIO. On one hand you could pick a mainstream, know-what-you-are-going to get path like education or getting PMP, Java, or other certification. On the other you could try to be different, which probably means lasting and unique contributions to a field.
This season NBC has two shows (a drama and a comedy) that follow the antics of an SNL-like show, mostly off-stage. Despite the cowboy-like mentality I hint at above, the shows do have some bounds. For example, they do need to fit into a specific time slot with commercial breaks. One common theme on Studio 60 is the planning board – a big cork board with 3″x5″ index cards in it that represent skits. They can add skits the board until it’s full, take them down, tear them up, develop them later, and show on.
The board is called a story board. I suppose that makes the cards … story cards?
By now, I hope it is clear that I have chosen my path. It’s nice to know that we share a similar set of tools with other people on that path.
As for what the game shows do, I don’t really know. I suppose it’s not interesting enough to support a profitable “life on the set” drama or comedy.
Go Figure.
Considering Conferences
I used to go to conferences to get new ideas, and my emphasis was on the “big” ones like The SD Conferences or the Open Source Conference. Thanks to the internet, the public library, and wooden speakers who just stand up and read the bullet-points, I can now get most of that value from home. (No, they aren’t all that bad, but you get the point.)
Over the past couple of years I have come to realize that little fact, and yet I still go to conferences. In fact, I desire to attend many more than I can actually get out to. Something changed.
Oh, I still try to pick up ideas, but instead of getting them from powerpoint slides, I get them from talking to people. In fact, it is people, relationships, and opportunities to sharpen my own saw that keep drawing me in to conferences; not fancy theoretical ideas about how to reorganize a 100-person IS shop, or new policies to implement top-down. To borrow a line from “Lessons Learned in Software Testing”: Conferences are for conferring.
So here’s two assertions for you:
1) It’s generally better to meet people at conferences who actually live somewhere near you,
2) Having those people around, and talking to them, can often make ’sharpening the saw’ (your mind) a lot easier
At this point, you might agree, but in the back of your mind you are thinking “but my company won’t send me to a conference.”
Well, let’s connect the dots. The easy way to meet people who you have a chance of catching at lunch sometime is to go to a regional conference. Regional conferences are all over – there’s GLSEC in Michigan, IQAA in Indiana, PNSQC in Portland, YAPC all over the place, the Simple Design and Test Conference in Pensylvania, a ton of barCamps, Ohio and Chicago both have conferences as well. Check out QAIWorldWide or the Agile Alliance for a user’s group near you, and see if they have a conference.
Most of these conferences are run by a non-profit, so the price will be cheap – usually less than $300 per day. They may be close enough that you don’t need a hotel room. If your company won’t spend a dime, email them and find out if you can volenteer, and earn your own slot. Then you go for free.
If there is no such conference in your area, you can start one. Really, if you live in any kind of medium metro area and you have a few friends who will commit to it, it’s not about being “hard”, it’s more a committment of time.
So, like I hinted at before, after the conference a few people who really care typically go out for a beer, and that’s where the real learning starts. Amazingly, there are actual conferences that are structured this way. They limit the conference to something like 15-25 people and expect heavy involvement for all attendees; no sitting in a chair for you. My two examples are the Indianapolis Workshop on Software Testing, or IWST, which I’m making a real effort to get to this year, and the Simple Design and Test Conference. (SD&T)
Peer conferences are especially interesting because they go beyond non-profit to non-commerical! Everyone is a speaker and the conference is often free for all attendees. SD&T is the largest that I have seen, and it really intrigues me.
If you look at the SD&T website, you’ll see on the bottom-right that they drew some reasonably big names to be “just” participants, er, speakers when everyone is a speaker. One of them is the guy with the hat, who just looks really familiar. It turns out that he’s George Dinwiddie, a regular reader and occasional commentor on Creative Chaos, who now has his own blog. It turns out George lives in Maryland, my old stomping grounds, and the more-or-less home of the Agile Conference this year.
If you look at my blogroll at right, those people are more than just minds I respect; I consider them friends. I’ve met every single one of them at a conference or user group meeting (Except for Jerry and John Bruce) – and Jerry and John I have each read at least 180,000 words of. (That’s about 3 printed novels)
So first, seriously, let’s encourage each other. If you want help getting plugged in to a development community near you, drop me a line. And second, here’s to meeting George at a conference soon, so he can qualify for my blogroll. (Seriously, drop me a line.)
