Matt Heusser’s Blog
Testing at the Edge of Chaos
How would you test this?
Last Tuesday I went to XPWestMichigan, where we talked about craft in software development. One of points raised was that developers don’t spend enough time reading code, or working in pairs with someone else to gain experience.
I would argue that the same problem occurs in testing – how many times have you seen a given piece of software, along with a discussion of the strategy used to test it?
“Not often enough” is the first thing that comes to my mind.
So how about a testing challenge?
No, not the Triangle problem. A real test challenge that you might experience on a real web 2.0 application, like, say, Socialtext.
Now I’m going to describe a relatively complex problem, which you could answer in several ways. The simplest way would be overall strategy, that you might do in two paragraphs, on down to a detailed test plan – which you might want to do as your own blog post. (or not; long comments here are okay too.)
The Problem
Take a web 2.0 application, something like twitter or myspace or facebook. The basic application is a wiki; a series of editable web pages. These wikis are organized in Workspaces. Workspaces are distinct and independent; I have can have a ‘hello’ page in one workspace, and a ‘hello’ page in another, and the two could never meet. If two users overlap by a workspace, they can see each other’s activity.
Likewise, Workspaces can belong a Network, which generally line up with companies. (A big company, say GE, might have a GE account and then workspaces for each line of business, a general HR workspace, an executive workspace, and so on.) Belonging to a workspace in another Network implies you are a member of that Network.
Users can edit pages, comment on pages, tag a page, search by text, name or tag. They also have a profile, which they can save or tag, and can search the profiles. The software also has a digital dashboard that allows the user to see summaries, metrics, and other data at a glance. One of those widgets is the activities widget, which summarizes what the users (that you can see) have done in the software:
Activities Widget
The widget has drop-downs to filter what activities you want to see; in the picture, I’ve displayed all of them at the same time. Now let me explain a little more about these activities:
A user can also send a signal, which is a 140-character or smaller message. You send signals by filling out the empty space at the top of the widget. The signal can be replied to or have a name mentioned – reply is the enter-like signal at right. If you are mentioned in the signal, the message appears highlighted. Messages can also be private (the envelope like symbol at right); private messages will only be seen by you and the recipient, and will appear with a great background.
Contributions can be page edits or comments. Each person also has a profile. Profiles can edited (by the user themself, only) or tagged (by anyone). Or tags can be deleted; all three of those create a ‘people event.’
You can also choose to follow people, though who you follow or stop following is not listed as a people event.
If someone edits a page that you have previously edited, that creates a ‘conversation’ event. Likewise, you can watch a page (which is not displayed as an event) and if someone edits or comments on that page, it is also a conversation.
I think that’s enough about the activities widget for now.
The Five minute challenge:
* Describe your ‘test strategy’ at a high level. What kind of activities would you do? How long would those take you? What would your goals be? How long do you think it would take you?
The professional challenge
* What would be the kinds of things you would test for (your ‘test cases’). I don’t need every single test case; some of them can be generated with simple multiplication. Some can not. (You can always do a blog post and link back in a comment.)
And, of course –
* What else do you need to know to answer those questions? (I will answer these, in italics, in the comments)
Keep in mind, this is an agile shop that delivers working software every two weeks. The developers coded four pieces of functionality like this in this two weeks, and they will deliver four more in two more weeks. Your test team is three people, one of whom is a manager.
I’ll post the winning entry later, along with my comments.
In the mean time, Batter up …
Visual Test Challenge
First, the video:
Seriously, please, watch the video. It’s only a minute. Then, and only then, scroll down.
(This space intentionally left blank.)
(This one too.)
(More intentional whitespace/delays for people who’s eyes wander)
(Still more)
(You get it)
Ok. The video demonstrates that when we focus deeply on one subject, our brain has tunnel vision – we miss other things that may be happening in the application. The more scripted the work, the more likely we are to miss failures outside of the specific parameters of the test case “guidance.”
Now for application. Let’s consider a typical “test case” for a simple calculator. Press 4, Press Plus, Press 4, Press equals, expect to see 8.
What if the screen layout is wrong? What if, after we press plus, the cancel button becomes disabled? What if, on the second display, one of the icons loses it’s transparency? What if we run it twice in a row and get “44″ as an answer? What if anything else goes wrong besides the number eight appearing on the screen?
I submit that behind every test case is the hidden concluding sentance “… and nothing else went wrong.”
‘Oh, that’s obvious’, you say – yet every day I talk to people and organizations who certainly act as if this is not true.
The general conclusion for testing is an increasing skepticism on heavily scripted (’manual’) tests, and I agree. We can write that test at many different levels of specificity:
- Click 4, click plus, click 4, expect 8.
- Add four plus four, expect 8
- Add two whole numbers
- Add two numbers
- Experiment with adding
- Perform arithmetic operations
- Test the calculator
I think that’s a good thing, and credit is due: I first heard of this video from Jon Bach at STAREast in 2007, and I understand Eric Peterson used this insight at Agile 2008. Still, I hadn’t seen it on the web and thought it might be new for you.
But there’s more
Consider the two most common kinds of GUI-driving test automation: Record/Playback and Keyword driven.
Record/Playback has to see a successful test run in the first place in order to have something to compare to. After that, it will notice every change in the GUI – every single one. Resize your window, display a different day in the lower-right hand corner, move a button, and it’ll throw an error, regardless of weather or not a real problem occured.
Keyword-Driven test automation has the opposite problem; it will only check to see that the result is eight. It would miss all of the examples above. Literally an automaton, it would concentrate all it’s effort on counting the balls in the example above and totally miss the moon walking bear.
That’s the thing about test automation; it lacks intelligence.
James Bach covers this issue with his use of the term sapient testing, and Michael Bolton with his distinction of testing vs. checking. Yet I haven’t seen this distinction applied much when it comes to browser-driving tests, say with selenium. The discussions tend to imply that driving the GUI is ‘just’ developer-facing testing cranked up a notch. Yet with developer-facing tests, the developer controls the interaction and can examine every little bit that comes back from the function call.
Suddenly, with a GUI, a lot more can go wrong.
Let me ask this question about customer-facing ‘test automation’: How much of it is counting the balls that are passed around by the black team? How much of it is missing the Moon Walking Bear — and how can we compensate for that?
(Someone is going to point out that there are specific branches of test automation literature that recommend adding randomization to tests, or calling them in a random order, or otherwise attempting to introduce something closer to artificial intelligence, and that’s fair and important . It’s a good thing. I can’t help but ask, though – Why is that such ideas have had such narrow adoption and discussion?)
More to come.
Why Chaos?
My old blog “Creative Chaos” has moved to this web address:
http://blogs.stpcollaborative.com/matt/
(Yes, folks, Creative Chaos is no longer hosted by Blogger, but by our friends at STP Collaborative.)
We’ve also changed the name to “Testing at the Edge of Chaos”, which sort of brings up the question “Why Chaos? What’s the deal?”
Let me explain.
Cowboy Culture
Like many North Americans, I grew up watching Westerns. You know, the ones with the hero in the white hat, with the badge, representing good, law, order, and civiliation. Opposed to him were the bad guys – with black hats, bandannas, representing all that was evil and chaos.
The ranchers, trying to grow cattle or farm, would be attacked by the black hats, and the hero would appear, save the day, and ride off into the sunset, sometimes with the girl by his side. We go to sleep, knowing that good has triumphed.
Now, let me ask: What’s wrong with that story? Why, strictly speaking, nothing’s wrong with it. But it does sort of … conflate two things.
Is Law and Order always creative, on the side of the ranchers, building the new frontier?
Is Chaos always destructive, on the side of the black hats, stealing the cattle?
I don’t think so. Consider, for example, the gardener who wants all his carrots to grow “properly”, in rows, and so he pulls up perfectly fine young carrots who are out of the row.
Or consider the story of the opressive, brutal, but legal sheriff of Nottingham, who represented the law, and Robin Hood, who represented Chaos – but was using the Chaos to return ‘taxes’ that were essentially stolen from the people.
A better model
This gives us four settings:
Creative and Ordered
Destructive and Ordered
Destructive and Chaotic
Creative and Chaotic
—-> For another example, let’s look at a typical company in the United States, say a book publisher or a magazine. The actual product is created electronically, and the file is transferred electronically to a printing vendor. Physical delivery of the books or magazines is outsourced to a shipping company. Sales are processed electronically or over the phone. Customers don’t want a walk-up office, they want a website. Thanks to voice over IP phones and call routing, customer service can be done via the phone. There is no actual need for a physical office.
And yet, every day, a thousand people, forced to live in a very small geographic area, get in their cars, drive an hour each way through smog and traffic, in order to get into the office.
Why even have an office?
Because in the late 19th century, when the corporation was born, it was physically pulling coal out of the earth in a specific place, and needed the workers in that place, at the same time, to get the work done.
It had to be in that place, because that’s where the coal mine was; for that matter, the coal mine needed to be near a railroad to ship the coal. (Or the oil well, or the manufacturing plant, take your pick.)
In other words, the reasons we need an office to build digital-only tech products is mostly historical accident.
Or, sure, there are physical and environmental factors. Some people like the office and the watercooler, and that’s fine. There’s room for temperament, but it’s not a requirement.
If you do have an office, most research shows that open-plan spaces are the most productive for knowledge work – or an office with a physical door in some rare cases. Yet the typical corporate office is filled with cubicles.
So the typical North American Corporation (and parts of Europe, I’m told) tell us where to work (in the office), when to work (9-to-5),what to work on (whatever project the portfolio management office picked), how to work (”follow the process”), values efficiency over effectiveness (”you’d better be typing!”) and a stable, predictable process over innovation.
And what we get is bland, boring software, that may meet some interpertation of a document written months or years ago.
I think we can do better.
How then, should we test?
If we look at the history of innovation in technology, it rarely happens in a cubicle. Linux, Unix, Napster, The Web Itself, The iPod, the Mac, Google, Amazon.com, Object-Oriented Computing, the mouse – these were not invented in a cubicle farm by having a specification that produced software that was tested through a traceability matrix. They just weren’t. They were invented by two kids in a dorm room, or a computer lab, or maybe three founders in a garage. (Did you know that IBM’s personal computer was a skunkworks project, run a thousand miles away from corporate headquarters, on purpose, so the bureaucrats wouldn’t have a chance to kill it by “helping” it and insisting it “follow the process”?)
There are plenty of places on the web to go to learn about Templates and Documentation-Driven Processes and governance. This blog will be a little different; my goal is to explore the other side of the software development story – the creative yet chaotic side.
My goal is not to have a stable, predictable, repeatable process. I want to help ‘move the needle’, just a little bit, toward the chaotic side; to work in the edge of chaos, providing just enough process rigor to enable executives to make decisions and be accountable for my behavior. I do want to make a big pile of money for a company, and, hopefully, share a little bit of it with myself – without getting fired.
It’s going to be quite a ride. Will you join me?
Why is QA Always the Bottleneck?
“Why is QA always the bottleneck?” is the second in a series on how to deal with unfair test questions; it is up this week on SearchSoftwareQuality.com. (Free registration required.)
The next in the series will probably be “how long will testing take?”, but i’m curious what you think. What questions do you struggle with, and what interesting answers do you know?
Life is short – live well
I was reading The Secrets of Closing Sales yesterday and was struck by this line:
Nothing in the world can take the place of persistence.
Talent will not; nothing is more common than unsuccessful men with talent.
Genius will not; unrewarded genius is almost a proverb.
Education will not; the world is full of educated derelicts.
Persistence and determination alone are omnipotent.
I could nitpick some of the words of the quote – but the spirit – that consistency and dedication will win in the long run – is something that resounds with my experience.
Then, later, Jason Huggins pointed me to this blog post by the creator of wordpad. In it, Matt points to this blog post by Tim Ferris that is a gentle introduction to the writing of Seneca.
It’s one of the most inspirational things I have read this year.
Go ahead, invest thirty minutes in Senaca. Breathe it in. I believe you’ll find it time well spent.
What am I saying when you cross the initial quote with Seneca’s commentary? Well, yes, persistence matters. Yes, if you try again and again, you may succeed where others will fail. Just be careful that you don’t climb the ladder of success, only to find that it was leaning on the
wrong wall.
Test Management Tools
Allright folks, I’ll admit it.
I’m not excited about test management tools.
Oh, you could argue that I should be. After all, Test Management tools are purchased by test managers and executives. Test managers and executives have money; they control the budget and decide who goes to what training when. Finding someone’s pain point – and taking the pain away – is a perfectly legitimate business strategy. (If they have money to spend, why, that’s even better, right?)
Yet I’m still not excited. Why?
Well, let’s take a frank look at the thinking behind a test management tool, by which I mean something specific: A keeper of ‘test cases’, and a tracker of which test cases have been run against which codebase.
It starts with this thinking:
(A) We can define all our ‘test cases’ up front,
(B) When those test cases pass, our codebase is ‘good’ (Or, alternatively, when some fail but some decision maker desices to ship anyway),
(C) /Recording/ which test cases have run, and which are yet to run, in precise detail, has some value in and of itself
I reject the premise behind all of these arguments.
Here’s an alternative, that we use at Socialtext:
1) Create a single wiki (version, editable web page) page for a release
2) Mark down each type of testing you want to do in every significant combination
3) For example, break the app by major piece of functionality, then further by browser type
4) Add all the automated suites or unit-test results if those matter
5) Have the technical staff ’sign up’ for which pieces they will test
6) When testing on a component is completed, the tester writes ‘ok’ and the bug numbers he found, or, perhaps ’skipped’ and the reason why.
For what it’s worth, we’ve been doing this at Socialtext for nearly two years, since before I was hired. We are constantly tweaking the process.
This one-page overview is a higher-level view than a test management tool might provide. It shows you what matters – the failures – not 5,000 “ok” results. It assumes that the test ideas are located somewhere else that the test can find if needed. It assumes the tester actually did the testing and leaves open the possibility that the tester can explore the functionality. It leaves the tester responsible for what ‘ok’ means, instead of a spreadsheet or document.
This isn’t a brand new idea; James Bach recommended something similar in 1999, called a “low tech testing dashboard“, only he suggested it be done on a whiteboard. Other people have suggested using a spreadsheet, but that has version and read/write problems.
A wiki is just one more step forward; it provides version control, transparency, and creates a permanent artifact that could be audited. In my mind, this provides some of the benefits of test management tools with much less time investment.
So no, I’m not excited about most test management tools on the market. In many cases, I am suggesting they swat a fly with a sledgehammer. Yet I recognize that test managers and executives have legitimate problems. So let’s not rush off to build something to get money; let’s come up with real solutions and see if the money flows from there.
Who’s with me?
Hello world!
Welcome to STP Collaborative. This is your first post. Edit or delete it, then start blogging!
Best New Software Test Writing
Over the summer, I’ve noticed a trend that bothers me just a little.
Cem Kaner hasn’t blogged in months; James Bach hasn’t blogged in weeks. Michael Bolton is blogging sporadically; Elisabeth Hendrickson is blogging very occasionally. Ben Simo hasn’t blogged since February.
Of the people on my blogroll, only Adam Goucher is consistently writing new blog material.
Now, there may be good reason for this. The people on my blog roll are mostly independent consultants; perhaps the economy is picking up, and they are so busy, that blogging is the first thing to go. Perhaps they are focusing on twitter – or focusing writing on a book. I don’t know.
What I do know is that when I click through my blogroll, I’m not seeing a lot that is new.
So I went and asked the Writing-about-testing Yahoo group for some recommendations, are a few we came up with:
Michelle Smith
Pradeep
Catherine Powell
Marlena Compton
Lanette Creamer
Geordie Keitt
Yes, getting to the point where you are known by first name only is a compliment, and yes, that’s the same Lanette Creamer who’s paper “Testing for the User Experience“, won the best paper award at PNSQC 2008. (For those who live near Portland or need and excuse to make the trip: Lanette and Marlena are both speaking at PNSQC this year.)
In addition, all of the students of the Miagi-Do School of testing happen to have a blog. That is no accident. These are people that I personally vouch for as having an interest in, and passion for, software test excellence. While some have English as a second language and are learning to communicate better (as we all are, right?) – they sharpen those skills through blogging. Check them out, please:
Justin Rohrman
Ajay Balamurugadas
Markus Gaertner
Jeroen Rosink
Update: I’ve also been told that David Christiansen is blogging again. I went and checked and his recent posts have been very tester-centric. Yay!
September Software Test&Performance
I just got my copy of the September Issue of Software Test and Performance in the mail yesterday.
Yes, I got a September Magazine on September first. Not August 15th. Not October 5th. The timing is actually right. Amazing.
The theme is on outsourced testing, and yes, Chris McMahon and I have a column on page 8. (And yes, we listed The Boutique Tester as one model of test outsourcing.)
If you register, you can download the PDF – or you can read the article directly on the web.
The new, re-tooled STPMag.com has a comments feature, so please, feel free to put comments up here or on the website.
We’re working on a column on coverage right now; if you send us your thoughts early, you could help make a better column …
BONUS: This week’s informationweek had a back-page editorial on outsourcing; I thought you might like to compare and contrast to what Chris and I did for ST&P.
Music to test by
About a year ago, Danny Faught and I team-authored an article on Music to test by for the Association for Software Testing’s magazine. Sadly, they had a change in editorship, well … from having one to not having one. (It is a volunteer position)
So the article was never published. I just got an iTunes Gift card and find myself listening for music to test by.
So instead of listening to me pontificate, I am curious: Do you listen to music while you test or code? (Or do you have any music playing in the background while you pair or collaborate?)
I’ve found that movie soundtracks often work well because they are /designed/ to be on in the background. But I’m curious what you think …
