Matt Heusser’s Blog
Testing at the Edge of Chaos
Complete, Correct, Consistent, Unambiguous Requirements?
This morning, Shrini Kulkarni sent me an email asking what I thought of requirements based testing approaches. Specifically, his email has this line in it, quoting from a web page:
“XXYYY Inc is the world’s leading expert in Requirements Based Testing (RBT). The RBT process first ensures that the specifications are correct, complete, unambiguous, and logically consistent”
Which made my spidey-sense go off. Shrini asked me to elaborate on why and what that means, and liked my reply enough to encourage me to blog about it.
Here goes:
1) On day 90 of a 150-day project, the business senses an opportunity that will allow them to double revenue. They change what the software must do. This renders the perfect spec imperfect.
2) On day 100, the VP of engineering is fired. The new VP of engineering thinks that the fribble feature is “all wrong.” The spec is now imperfect.
3) On day 110, the CEO promises BIGCO that the software will do X. The spec is now imperfect.
4) On day 120, the VP of sales interprets a line item as doing X+1, and promises it to the customer.
5) On day 125, the new VP of Engineering brings back the famous consultant Rickaro BlenderBurger, to explain how he personally inspected the specifications to make sure they were unambiguous, and the X+1 promise was not right. The customer responds “But he promised us.”
6) On day 126, the CEO creates a time warp field and goes back to day 120, and personally sits in the meeting with an invisibility cloak, to figure out how it is possible for the VP of sales to get the unambiguous document wrong. He brings along a linguistics professor.
The linguistics professor watches the meeting and says “Oh, yeah, that is going to happen. English was pretty much hobbled together by illiterate peasants, and codified by writers like Chaucer who knew just enough Latin to grok the letters. It is a pre-rennesaince, pre-age-of-science language. Anyone who tells you they can write an unambiguous spec in english is selling you something.”
—> In other words, to quote James Bach:
“Here is a simple litmus test for an ambiguous spec: It is written in English”
Seriously. Ask an engineer the difference between and English spec and a CAD drawing.
—-> People who think you can write an unambiguous spec, so you can slide it under the wall, are denying how most human beings actually communicate and collaborate.
It’s not about hand-offs. It’s about working together. Or, in Japanese “Gung Ho.”
Hence, my spidey sense.
Regards,
–heusser
UPDATE: I am not saying that we should completely reject the idea of requirements or specs and only go by word of mouth. As with many things, the danger is in the extremes. Instead of just picking an extreme, we could have a more healthy conversation about which way we move the needle – to do a little more conversations or a little more documentations.
UPDATE TWO: Over the great back-channel of email, Jonathan Kohl points out that even though the claims of XXXYY Co. may be unrealistic, the ideas and concepts that XXXYY uses could be valuable and might be worth looking into. There are exercises that can be used to get less ambiguous documentation, and they can be helpful. As the expression goes “Don’t throw the baby out with the bathwater”, and I think he has a point. Thanks, Jon.
100% Test Automation
A recent post I put out to the Agile-Testing List:
The original poster asked:
> I have been asked to work out a system to make a 100%
> automated testing solution to suit the agile development
> process that the dev team are using.
Perhaps I don’t understand what 100% automated testing solution means. When I hear those words, I think that means that every test should be able to run at a click of a button – that no test should ever involve setting up test data or checking the screen personally to make sure the results are “right.”
Did I get that right?
If so, has *anyone* on your entire team *ever* actually had success with such an approach? AT ALL?
How will you automate usability testing?
Later on, the author asked:
>I think that while we are all very enthusiastic
>about agile development and automation of the acceptance
>tests we are not entirely sure of the practices that
>we need to implement.
I find it helpful to remind myself that our end goal is to produce working software, not cool automation. So one technique I have used is to write up the automation as stories and let the PM prioritize those stories. That means that sometimes, the automation doesn’t get written. That’s ok, because the business has made a decision to not invest in the feature – a fancy word for this is “governance.” On the other hand, the automation that does come out is a true project – not something you, as a tester, are expected to write on your lunch hour despite other deadlines.
Of course, everything I’m writing above is in reference to customer-facing test automation, not developer-facing. Developer-facing unit tests, in my mind, are just part of the discipline of development, and I simply expect them to happen. If they don’t, I expect the devs to either live with the pain or start automating. (If the code released to QA just plain doesn’t work, and the devs have no automated unit tests, then I begin a … collegial conversation among peers …)
Regards,
–heusser
xndev.blogspot.com
What is a professional?
For the past few years now, I have heard countless exhortations for software testing to become a “professional” community.
Generally speaking, those exhortations are a sales pitch – come earn by certificate and be a professional.
Sorry, guys, I don’t buy it. The root word for professional is PROFESS – to say out loud. To stop saying http://www.blogger.com/img/gl.link.gif“testing is what I do” and start saying “a tester is who I am.”
You do not need a certificate on your wall to do that. All you need to do is to *care*.
The next step is to get involved.
Now, not everyone is going to go out in public and speak, and not everyone is going to publish – or even blog. But there are other opportunities.
The fact is, for every guy on stage at a major conference, there are probably two behind the scenes. Printing out ID Badges. Dealing with registration. Putting the website together. For non-profits, the work is generally unpaid, but it has considerable value – you get in the conference free and you get to know other people who share common interests.
Right now, somewhere near you, a conference is calling for volunteers. It might be in West Michigan, it might be PNSQC in Portland, it might be Test2008 in New Delhi, India.
If your company doesn’t have budget to go to a conference, I have very simply advice:
Go Anyway.
What is a software architect, anyway?
There is an interesting thread on JoelOnSoftware on the subject.
Here’s my follow-up question for you: What is a test architect, and do we need them?
Or, perhaps more meaningful: What kind of problems can a ‘test architect’ solve? What kinds of problems would that role introduce? If we had one, what would he do?
Call for Reviewers
I’m working on an article proposal for a respected softare testing publication. If you would like to review (preview!) the work, please drop me an email at matt.heusser@gmail.com. If you’ve never published before and are thinking about it, then learning about the process might have some value for you.
–heusser
Trip Report – San Mateo
I just got back from a conference in San Mateo, California, followed by a short stay in Salem, Oregon. Here’s my version of the skinny, in no particular order.
(A) STPCon was great. I got to run into Michael Bolton, Rob Sabourin, finally meet Doug Hoffman and BJ Rollison, and make a few new friends. My biggest complaints were with myself. I did not plan to stay for the whole conference, and I presented so much material that I didn’t get to see many of my peers present. What I did see, however, was spot-on – particular what I heard of Rob Sabourin’s keynote on Scrum.
(B) Both the Grand Rapids International Airport (GRR) and the Portland International Aiport (PDX) have free wireless. Minnepolis/St. Paul and San Francisco, on the other hand, charge just enough that it isn’t worth it. This is good news if you are flying into GRR for the Great Lakes Software Excellence Conference – www.glsec.org – for which planning is finally getting started for 2008. (Sorry – the Call For Papers is not yet public.)
(C) I bought a copy of “Fast Company” for the plane. Long-time readers know that I enjoy it’s sister magazine, Inc, mostly for the inspiring writing. I’ve always been a bit put-off by Fast Company; I don’t quite “get” the concept of a magazine and it feels weird. No, I mean it literally feels odd. Still, I needed reading material for the plane, so I bought an issue to sample. It was good, but not in the way I expected. First off, the magazine is not printed on cheap paper, but instead on 100% recycled paper, which explains the odd feeling. Second, I got two actual real ideas that I intend to take action on, directly from the magazine. For me, that is both unusual and valuable.
(D) Nothing spells success for lightning talks than a comedy of errors. We had problems with the presentation equipment, with getting a mac to project, with powerpoint templates and getting automatic, timed slide advancement working. Not only was it a total blast for the audience (and interesting for the organizers!) it was a great lesson in the ways automation can go wrong.
My favorite part of the lightning talks: Jason Huggins, creator of the selenium test framework, is showing how he tests a google orb. (A google orb is a continuous integration light that lights up red or green based on weather the build passes or fails.) He explains that he lights the bulb, then uses the macbook’s digitial camera to take a picture of it, then his test software analyzes the pixels in the picture to determine if the bulb lit up the correct color.
Michael Bolton’s Response was priceless:
“Couldn’t you just look at it?”
(E) I really, really don’t like traveling. The enclosed spaces (planes, hotel) leads to air circulation problems and noise pollution. Flying leads to lots of essentially wasted time and tends to hurt our relationships with actual, well … people. At the same time, my old traveling regime will have to give way a little bit so I can focus on testing for Socialtext and, well, being a father to my children.
My current problem is a glut of opportunity. I simply can’t do all the good things that are coming my way. The net result is that I will plan to be a little scarce at conferences for the rest of 2008-2010. That may mean more writing or blogging, or maybe something else, but I’m going to cut down on the traveling. If you want to step up to fill the void in the Continental United States, please let me know; I know of several program chairs looking for speakers.
In other news, The original spot for Google Test Automation Conference 2008 was Hyderabad, Indian, but management just announced a venue change to Seattle, Washington, set for an October conference date. I was hoping to use GTAC to get out to the sub-continent of India, an excuse I am still looking for. From here, it looks to me like a trip to India that might become my one big traveling conference in 2009.
GQM – Yellow Brick Road – II
This is the best response I have read to my earlier GQM Request.
It is from James Bach, Originally posted on the Software-Testing Discussion List.
Enjoy:
I don’t like GQM. Here’s why:
It assumes you have nothing to learn!
GQM is appropriate for situations where you understand the precise workings of the system you wish to control.
Example:
GOAL: I wish to back out of my driveway.
QUESTION: Is anyone behind me?
METRIC: turn around and look behind me.
But this is rarely the case with software development projects. In projects, we are trying to learn how things work. A development project is a social system. We don’t just monitor social systems, we must study them. Is X doing testing well? I’ll have to observe his testing and learn how he is doing it. I must be open to surprises while doing so.
My version is called OIM:
1. Observe: What is happening?
2. Inquire: Why is THIS pattern happening? Let’s study that.
3. Model: Here’s my theory about how this project works.
This is not only a cycle, each task is simultaneous. During any of these tasks, at any time, you take action based on your current model of the project. OIM is consistent with social research methods such as Grounded Theory.
This is how we used metrics at Borland, and everywhere I’ve worked since then.
GQM makes it sound like metrics are easy to use and interpret. OIM is all about coping with murky and changing reality.
STPCon Draws near ….
I’ll be spending the weekend (or, at least, a lot of it) prepping for the Software Test&Performance Conference in San Mateo, California. If you’ll be there, please shoot me a note, we’ll find a few minutes to talk and maybe run through some testing challenges.
If you can’t make it, I still have something for you – my article “Where Do I Go From Here?” On career management software testers appears in the April Issue of Better Software Magazine. While it is a subscription based magazine, I am allowed to give out copies of my article as a PDF.
Want more? You can can follow the link above for a free issue of the magazine, and a subscription link if you want more after that.
Goal Question Metric – The Yellow Brick Road?
I posted this on the software-testing email group yesterday.
The replies have been fascinating; but I’m curious what you think:
Hello Folks.
Many people here have heard my own life stories about programming and testing; how, essentially, I kept getting “patted on the head’ and told that I “Didn’t Get It” because I opposed big extensible designs, rituals, signoffs and handoffs in the development process, and expensive, heavyweight test case programs.
I stopped worrying about it when I realized that my projects were far more successful than my peers. Eventually, I started talking about it openly.
Metrics are currently on that list. I have in my study the handbook of software quality assurance, 3rd edition, that contains a list of about 150 qualities (like scalability, security, etc) that can be measured. Then it tells you that one or two metrics will cause dysfunction, you need a balanced scorcard. And that the easy-to-gather metrics are also easy to game and bad, but that the good metrics are expensive to measure. Oh, and be careful, because your engineering staff will rebel if they have to spend too much time gathering metrics instead of doing work.
To summarize, this is what the book has to say about metrics:
“Good Luck.”
Which brings me to my next sacred cow: Goal Question Metric.
GQM is a framework written by Victor Basili; you can google it. The basic idea is that instead of gathering a bunch of metrics, you actually figure out your goal (like “faster production”), ask a question that will help measure that goal, and turn that into a metric.
I have to grant that this is an intellectually valid framework, and it beats the pants off of mindless gathering of numbers. For software testing, the idea has been endorsed by people I respect like Cem Kaner and Lee Copleand.
Here’s my problem: This idea has been around for a long time. When It comes to software testing, I’ve read a great deal of the literature, been to the conferences, read a lot of blogs.
Except for a few examples from people like Lee Copeland and James Bach here’s what I always see: “If you want metrics, use GQM. Since all contexts are different, I can’t give you an example.”
pschaw. Is it too much for me to ask for a case study before I invest time, energy, and effort into a metrics program? One with positive ROI? Enough positive ROI that I wouldn’t be better off working on other projects, or sticking the money I would have spent in a CD?
It’s been 13 years since the first GQM paper was published. I haven’t seen GQM provide it’s value in a software testing context.(*)
Have you? I would be really interested in success stories, please.
Regards,
–heusser
(*) – Please don’t say NASA. They work under an entirely different set of constraints than commercial software development. And even then, the business case is shaky.
UPDATE: Dr. Kaner replied that he doesn’t really ‘endorse’ GQM as much as he simply mentions it during talks. His overall comments are along the lines of “GQM looks interesting, it’s more grounded than nothing – if it works for you, good for you.”
The Nine Forgettings
“If you measure the wrong thing, and you reward the wrong thing, don’t be surprised if you get the wrong thing.”
— Lee Copeland
Lee’s talk is about 34 minutes and challenges how the typical company does software testing.
It’s a good watch, and it’s free.
