Matt Heusser’s Blog
Testing at the Edge of Chaos
Going to Boston in the Fall?
I went to Software Test and Performance Conference last spring in California and had an absolute blast, met some really interesting people, reconnected with some old friends, and learned a thing or two.
This Fall, Sept. 24-26th, is the Software Test & Performance Conference, Boston. James Bach is keynoting this year, and many, many of the speakers are top talent in the Industry.
I am told that the Extreme Early Bird Rate expires on August 1st – that’s an extra $400 off. In addition to a wonderful and stimulating conference, you’ll be in the Boston/Cambridge area, the home of Harvard and MIT, and yes, ancient pirate-y history.
In fact, you’ll be going to Boston in the fall.
(And for those who suspect this entire post was an excuse to link to a veggie-tales silly song with Larry … I plead no contest.)
June Issue of AST Newsletter is out -
For those who don’t know, I am a paid member of the Association for Software Testing.
Yes, my own pocket. No work reimbursement; I spent real money.
The AST puts on an annual conference, has mailing lists, special interest groups, and even has a newletter – and the June Edition of the Newsletter is available online.
The word on the street is that it is moving to print in the next few months.
Don’t have time to read the whole thing? Skip to page 12 “The Mythology of Software Testing” compiled by David Christiansen – it’s quite a hoot.
If you were a CIO …
… What would your test policy be? Is an interesting question posed by Annie-Marie Charrett on the SoftwaretestingClub.com Site.
It turns out that Annie-Marie is contributing to an IEEE council on the subject, trying to make a standard.
My first thought was:
– What problem does a testing policy (document) solve?
– When would such a document be appropriate?
Here’s my more detailed response:
Look, international standards are great – for products. They are the reason that I can pick up any AA battery and plug it into any AA outlet – they clarify the required form factor, voltage, and tolerances for both.
For _process_, however … not so much. *HOW* the 1.5 volts is delivered is entirely up to the manufacturer. If he *had* a standard for batteries in the early 1970’s, it would have been zinc-carbon. In that case, alkaline would not have been “standards compliant”, we’d never have the duracell bunny and our batteries would run out in about 1/10th the time. Oh, and forget about rechargable (ni-cad and later nimH), that’s just crazy talk.
The next left of abstraction is to define the process by which those products are made. While we do have some process standards (ISO or Gmp), let’s be very careful to not stifle innovation in software testing.
So to be direct: If, as a CIO, I had some *problem* for which I thought a test policy might solve, I suspect I would post the problem on a wiki page and ask my direct reports to solve it.
To quote the one-minute manager – “I don’t make decisions for my people.”
(Credit where it’s due – The seeds of battery analogy come from PeopleWare, 2nd Edition, Pg. 187. Yes, I looked it up.)
Community Support, Love and Beauty
Clay Shirky gave a talk that covered community support at SuperNova 2007. During part of the talk he covers the value of community support – things like this blog, or the agile-testing list, or theSW-IMPROVE list.
About nine minutes (perhaps the best nine minutes) of the talk are available online here in quicktime format.
It’s a great, inspiring talk about using open-source, and open-source tools to build community. Part of my explicit goal is to build the same thing in the software testing community.
I’ve had my moments, but I am no Clay Shirky. Watch the video. It’s worth it.
Interview with Software Quality Engineering
Software Quality Engineering recently interviewed me for it’s Grey Matters podcast, about career management for testers and the use of social software for software engineering. The podcast went up on the podcast page today – but after a month or so it will rotate out. So you can hear the SQE Greymatters interview by direct link.
The Stickyminds Greymatters podcast comes out roughly monthly; you can click and drag this link into iTunes in order to subscribe.
It’s all zero-sum – or – more Peter Drucker
The growth companies of the fifties and sixties promised both more sales and higher profits indefinitely. This alone was reason to distrust them. Every experienced manager should have known that these two objectives are not normally comparable. To produce more sales almost always means to sacrifice immediate profit. To produce higher profit almost always means to sacrifice long-range sales. In almost every case, this irrational promise and the resulting refusal to make balancing decisions between growth and profitability objectives was the direct cause of the large losses and the equally large write-offs of the growth companies in the late sixties and early seventies.
There are few things that distinguish competent from incompetent management quite as sharply as performance in balancing objectives. These is no formula for doing the job. Every business requires it’s own balance – and it may require a different balance at different times. Balancing is not a mechanical job. It is a risk-taking decision.
- Peter Drucker, Management Tasks, Responsibilities, Practices
In Software Development, the objectives are time, staff (or “resources”/money), quality, and, perhaps, technical debt or technical investment.
Sure, there’s a place for exhorting, for motivation and leadership – just like in parenting. But any parent with anyone more than a three-year-old knows that imploring and exhorting fail. Eventually, you have to create and enforce tough consequences.
Likewise in management, a manager has to make tough choices. Sometimes, sadly, the tough choice is “pretend the everything is fine, and when it goes south, blame the techical people to save my job” – and you can’t blame the manager. He’s just doing what the system is incenting him to do.
Yet in a sane workspace, what matters is results, and the manager is going to assemble and direct the troops to reach the objective. How high he sets each objective, is a key decision leading to overall success or failure – a risk management decision.
But don’t take my word for it; quote Drucker. 
On Vacuous Documentation
If you read this blog, you know I’m no fan of vacuous documentation. In my experience, “comprehensive” documentation is often not comprehensible, and likely to get stuck in a drawer and forgotten.
By the way – The technical term for that is “waste.”
On the other hand, sometimes your are mandated, or required, to fill out the paperwork. To this day I remember the huge amount of documentation we had to produce for 10th grade history. It was so intense that I dropped the course.
Then, a month later in Alegebra 2 class, a friend of mine showed me his A+ homework score … and, on the second page, where for the answer to question #4 he wrote the entire lyrics of “lucy in the sky with diamonds” as an answer.
Yup, you got it, the teacher either played favorites or merely judged the first page of homework. And it makes sense – with 6*30 = 180 students, with five pages of homework a night, she simply could not grade all the assignments she was given.
I learned something that day.
Fast-forward to the summer of 1993, where I am a flight commander at the Maryland Wing Summer Encampment for Civil Air Patrol. Basically, the program is very similar to the Boy Scouts, with an Air Force feel.
I am a flight commander, in charge of a sergeant and dozen cadets or so. Each night, I had to fill in an evaluation form. The form had me score my experience for the day, somewhere from 1-10.
The first day, I was a 5. It went downhill from there; I wrote 3, 1, -5, “are you even reading this?” — and never got a reply.
I figured no one was reading the forms; the whole thing was a waste of time. It was just like 10th grade history class.
Fast forward ANOTHER two years, to 1995, where I’m a squadron commander, and have a little more leisure time. I strike up a conversation with the encampment XO, and he says something that really surprises me.
“Oh yeah, Heusser – dude. I remember your eval forms back in ‘93. Those were hilarious. Absolutely hilarious. We passed them around the entire command staff.”
Then I remembered that that year I did not get honor officer or the honor flight award.
There’s two edges to this sword. Yes, the forms might be vacuous and silly – AND – just as important – you never know who might be reading them.
When it comes to vacuous documentation – be sure to tread lightly.
When to automate a test?
Along the lines of the previous post, I’ve just re-discovered a classic – Marick’s Paper on When To Automate A Test.
Oh, it’s from 1998, and it’s showing it’s age, but it lays down some basic dynamics in software testing and gives you tools to determine to automate, to test, or maybe not to test at all.
I think it is great /input/ to help you determine what approach to take.
But what do you think?
Testers Taking Too Long?
We had a question on the Michigan Agile Enthusist’s list – the team was doing Scrum, four devs and a tester, and the testing was taking too long. In some cases, it would take the tester six hours to create the (manual) documentation but only three to run the (manual) tests.
Some people suggested it be a “whole team” and share responsibilities. Others suggested it be a “whole team” with no specialties at all. This is my answer:
Pre-Note: It is a whole team effort, and the devs can certainly help test.
However, if you would prefer to keep *some* amount of specialty, I have a few ideas. (Essentially, I am suggesting a little of both approaches)
//—Begin my ideas
Let’s apply the theory of constraints to find how to increase the throughput of our tester.
We’ll find the bottleneck – what is taking the most time in the process. Then we isolate it, elevate it, increase it’s throughput, throw out what is not adding value, and go on to the next thing.
First off, the amount of time spent documenting is just bizarre. I suspect if you ask ‘how much of this documentation adds value to you personally, the tester-guy?” it might be a lot less than the “the process” or “what I was told to do” implies. Cut it down.
Second, there is a good bit of evidence from cognitive psychology that high-manually-scripted, repetitive testing both covers less lines of code(1) and also is more prone to inattentive blindness(2, 3) than more exploratory testing, or testing with a more loosely-scripted charter.
So we can go faster by having less documentation and a more loosely-defined charter for testing.
Now let’s find the next bottleneck. Commonly, it’s one of two things:
(1) The Tester is spending a lot of time in meetings or supporting other projects, or some other activity that does not add value to the project. Eliminate it!
(2) The Tester is spending a lot of time trying to figure out what made a bug trip, documenting the defect explaining it to people, reporting on the status of the defect, re-testing, etc.
The easiest way to eliminate time spent on #2 is to have the devs deliver better quality code to QA in the first place.
I’ve worked in shops where the code was extremely good quality before it got to test, and test could move along at a good clip. I’ve worked in shops where that was not the case.
Guess what? In those shops, testers spent a lot of time on #2.
How to increase the quality before it gets to test? I would suggest TDD, and, if it’s still buggy before it gets to test, you gotta ask “what’s up with that?” and fix it.
My overall suggestions for test strategy -
(A) On the Dev Side, Test Driven Development (TDD)
(B) On the test side, some amount of high bang for the buck automation. Do automation as an investment where the ROI is clear.(4)
(C) PERHAPS, an as-small-as-possible manual test suite
(D) Warmed over with exploratory testing
Of course, I write, speak, and consult on these topics, but I am focusing my professional energies on Socialtext right now. On the east side of the state, for TDD and dev-facing tests, I would recommend Ron and Chet, possibly the Menlo guys. For exploratory and other forms of testing, some of the best testers on the planet are in Indianapolis, and Louise Tamres is local to the Southfield area.
Regards,
–matt heusser
xndev.blogspot.com
(1) – Ref -”James Bach Minefield Analogy”
(2) – Ref Cem Kaner, Black Box Software Testing Video Series
(3) – http://www.youtube.com/watch?v=mAnKvo-fPs0
(4) – See some of Brian Marick’s recent work for some of the serious ROI issues companies are having with 100% acceptance-test-automation.
And More from Brian Marick
In sum: compared to doing exploratory testing and TDD right, the testing we’re talking about has modest value. Right now, the cost is more than modest, to the point where I question whether a lot of projects are really getting adequate ROI. I see projects pouring resources into functional testing not because they really value it but more because they know they should value it.
This is strikingly similar to, well, the way that automated testing worked in the pre-Agile era: most often a triumph of hope over experience.
– From his latest blog post
Brian is co-organizing the second Agile Functional Test tool workshop in early August, the day before the Agile 2008 . Instead of increasing the value of automated functional testing, he is interested in techniques to decrease the cost.
– And they aren’t full up yet. If you can’t get to tech debt because you’re going to Agile 2008 — you might want to consider going a day early and making this workshop.
