Matt Heusser’s Blog
Testing at the Edge of Chaos
How would you test this?
Last Tuesday I went to XPWestMichigan, where we talked about craft in software development. One of points raised was that developers don’t spend enough time reading code, or working in pairs with someone else to gain experience.
I would argue that the same problem occurs in testing – how many times have you seen a given piece of software, along with a discussion of the strategy used to test it?
“Not often enough” is the first thing that comes to my mind.
So how about a testing challenge?
No, not the Triangle problem. A real test challenge that you might experience on a real web 2.0 application, like, say, Socialtext.
Now I’m going to describe a relatively complex problem, which you could answer in several ways. The simplest way would be overall strategy, that you might do in two paragraphs, on down to a detailed test plan – which you might want to do as your own blog post. (or not; long comments here are okay too.)
The Problem
Take a web 2.0 application, something like twitter or myspace or facebook. The basic application is a wiki; a series of editable web pages. These wikis are organized in Workspaces. Workspaces are distinct and independent; I have can have a ‘hello’ page in one workspace, and a ‘hello’ page in another, and the two could never meet. If two users overlap by a workspace, they can see each other’s activity.
Likewise, Workspaces can belong a Network, which generally line up with companies. (A big company, say GE, might have a GE account and then workspaces for each line of business, a general HR workspace, an executive workspace, and so on.) Belonging to a workspace in another Network implies you are a member of that Network.
Users can edit pages, comment on pages, tag a page, search by text, name or tag. They also have a profile, which they can save or tag, and can search the profiles. The software also has a digital dashboard that allows the user to see summaries, metrics, and other data at a glance. One of those widgets is the activities widget, which summarizes what the users (that you can see) have done in the software:
Activities Widget
The widget has drop-downs to filter what activities you want to see; in the picture, I’ve displayed all of them at the same time. Now let me explain a little more about these activities:
A user can also send a signal, which is a 140-character or smaller message. You send signals by filling out the empty space at the top of the widget. The signal can be replied to or have a name mentioned – reply is the enter-like signal at right. If you are mentioned in the signal, the message appears highlighted. Messages can also be private (the envelope like symbol at right); private messages will only be seen by you and the recipient, and will appear with a great background.
Contributions can be page edits or comments. Each person also has a profile. Profiles can edited (by the user themself, only) or tagged (by anyone). Or tags can be deleted; all three of those create a ‘people event.’
You can also choose to follow people, though who you follow or stop following is not listed as a people event.
If someone edits a page that you have previously edited, that creates a ‘conversation’ event. Likewise, you can watch a page (which is not displayed as an event) and if someone edits or comments on that page, it is also a conversation.
I think that’s enough about the activities widget for now.
The Five minute challenge:
* Describe your ‘test strategy’ at a high level. What kind of activities would you do? How long would those take you? What would your goals be? How long do you think it would take you?
The professional challenge
* What would be the kinds of things you would test for (your ‘test cases’). I don’t need every single test case; some of them can be generated with simple multiplication. Some can not. (You can always do a blog post and link back in a comment.)
And, of course –
* What else do you need to know to answer those questions? (I will answer these, in italics, in the comments)
Keep in mind, this is an agile shop that delivers working software every two weeks. The developers coded four pieces of functionality like this in this two weeks, and they will deliver four more in two more weeks. Your test team is three people, one of whom is a manager.
I’ll post the winning entry later, along with my comments.
In the mean time, Batter up …
Comments (17)
6 Trackbacks/Pingbacks
at September 29, 2009, 1:46 pm:
This is not a trivial piece of software to test given your overview of pages, workspaces and networks and activities widgets.
Given that this we are using some flavor of Agile:
What was developed for this iteration?
[Great questions, Justin. In reality, in the first iteration, we didn't have the next/previous links at the bottom, or the reply icons at right, and you couldn't send signals. All that was developed in the second iteration. If you want to exclude that from your testing, that's reasonable. But I thought I would crank it to eleven, ya know?
]
Is acceptance criteria defined for what was developed?
[If you'd like, I can play the role of the product owner and we can collaboratively develop those acceptance criteria. For now, let's say no.]
How much time do(es) the tester(s) have to do their thing?
[I would think you could do a decent 1st pass at testing this, in our four supported browsers, in three hours. Maybe add an extra hour if you want do document your scenarios and do a great job brainstorming. Do you agree?]
What are the consequences of going over that time?
[After about six hours of testing, you're going to have to work a little modest overtime, but basically, it's a whole team concept. Nobody's going to get fired if you want more time to do a through job. The problem is going to be if you want it every iteration.]
Are there any descriptions (business scenarios / use cases) of how what was developed will be used?
I’m sure there are whitepapers out there. You can google around for ‘Sociatext’, here’s a link that might be helpful. If you have your own business or domain name, you can go to http://www.socialtext.com and sign up for a free account for 50 users and try it out – or even run your own test scenarios! This is also a chance for you to try out designing a test strategy for a company developing speculative products for an emerging market.
I hope that helps.
Justin Rohrman
at September 29, 2009, 7:32 pm:
Hi Matt:
Only a bit of 5-min strategy for now, at the Goal level.
Goals:
– Help create a precise, deterministic, executable “definition of Done” per User Story (a Storytest, in other words), before developers begin test-driving the story’s implementation.
– Know enough about what developers are test-driving, and what behaviors they are likeliest to cover, that I know what areas of risk testers can focus effort on (to minimize duplicated effort)
– Carve out enough time per User Story for manual Exploratory Testing
– Learn the domain, from the user’s perspective, thoroughly enough to inform my Exploratory Testing best
– Create just enough through-the-GUI automated tests to smoke-test the system, and to end-to-end test the things that developers cannot easily isolation test and collaboration test
– Know enough about the eventual production platform that I can plan to create a cheap mechanism for iterative deployment to a production-similar platform.
– Understand unspoken load, performance, failover, SLA, and other kinds of requirements as soon as possible, to plan for testing tactics and effort around them.
That’s all I have for now. I took more like 10 minutes.
–Patrick
at September 29, 2009, 7:33 pm:
Nice with some discussions on actual content, and it would be interesting to hear if the comments will have effect on your future testing.
Difficult question though when you don’t know what kind of unit or other automated tests exist; you don’t know what the tester’s preferences are, what the team want to get feedback on, if there are important functionality like customizations or an API.
[Say the software hits a REST API that was developed test-driven. Does that help?]
Anyway, my high level strategy would be to use this widget in as many ways as the real users will.
We would use the activity widget to record test ideas (that probably are executed at once.)
One tester could create (and re-use?) larger and complex scenarios involving many different users, activities, followers, signals, networks etc, and then look carefully if something seemed to go wrong.
The other tester could go into detail about one small part at the time, re-using those generic tricky strings you surely have, run each functionality at least once, and doing this on an error-prone machine (non-English OS, old/new software, German regional settings et.al.)
The manager can look more carefully at quality attributes like usability, accessibility, security, performance, stability (or ask the testers about these issues.)
After some time, they can pair for a while, then switch assignments, or change the plan.
When the activity widget is stable, we will attach it to the internally used application, and see how it does in a real environment.
Since it is a visual tool, we start with manual testing only, and think about automation later.
[Good 5-minute answer; I'd love to hear about your test case ideas. Yes, I realize, that's a bit of a time investment, but you could start with, say what you'd do in the first hour - or at least identify the /possible/ tests at a high level ...]
at September 29, 2009, 8:26 pm:
If you don’t have much time I think you can trust that the developers have tested their REST API.
But you could use the API to assist the scenario testing verification.
Test ideas is too much of a time investment right now…
at September 29, 2009, 11:26 pm:
Hey, Matt. I put together a set of test ideas over on my blog so I could make a long post without dominating your comments section
http://i-am-justin.com/?p=74
Justin Rohrman
at September 30, 2009, 3:37 am:
When I read the problem statement, I despaired. Just too much there and too vague. Which is good, b/c it reminded me of a fundamental truth of testing: Software is effectively infinite, but testing resources are finite.
[I think it's common, and expected, for a tester - to work with time pressure, under conditions of uncertainty, and yet be expected to test with integrity, in a defensible manner, that strands up to scrutiny. That is the level of skill and expertise I claim to provide to my customers, and work with my students to reach toward.
Yes, this particular problem is hard; if it was easy, it would not be a challenge. Perhaps, though, as the audience is not familiar with the underlying app itself, it might help if I provide some product-owner derived 'acceptance test criteria.' If I get more requests along that line, I can do that in a follow-up post.]
It’s always like you’ve got N missiles and the entire Red Army is charging through the Fulda Gap. You gotta pick the N best targets in a target-rich environment.
So, back up and talk to the guy who gave you the work: “What’s important to you? What’s MOST important to you?” Let’s take all your stories and rank them according to importance. Try to do a Pareto analysis on them.
[Well, that would be the product owner, and that's a fair question. We do have PM's at Socialtext, and we thet do provide a first pass at acceptance criteria. But those "happy path tests" usually do pass; it's the nook and cranny bugs that the testers are looking for. I'll put up some happy path tests tomorrow.]
at September 30, 2009, 2:05 pm:
Hi
I would start by making some kind of drawing to visualize the new functions.
[That's neat, and that's where I would start too - with a decision tree.]
I would then ask my team if i missed anything they felt as important.
I would then have them explain the functionality for me and me asking questions to them. Together we would come up with scenarios and characteristics to start our testing with. During our testing i expect us to learn more about the product and will then go back to our drawing and add or remove test ideas as time goes by.
Our testing would be exploratory and sapient since this is a social application with lots of human interactions. Lots of the value is how the users perceive and appeals to the application.
I would ask the developers for some help regarding volume, duration, and performance test on the database. This we could do beneath the Gui.
I would also ask the support organization to have a look at the application to get input from them about installabiliyt, supportability and other support related issues.
Also i would ask if we could release this application to for an example a school class with teenagers to just go nuts with the application for a week or so. Teenagers are used to this kind of tools and have there way of using it and therefor some what different expectations that we might get some valuable input from.
My goal for the testing is to the team and product owner provide as much relevant information about the usage of the application from all different kind of users.
By having this approach we will gain diversity and variation in our testing. Getting valuable information from different types of users about the impressions and feelings not only functional but also usability , performance, stability and others. Also adding some supportive information our database testing.
We will stop our testing when our stakeholders are satisfied with the amount and quality of information we have delivered or when they tell us to stop testing.
Ok my 5 minutes is up so i better stop now!
[And A good five minutes it was, thanks.]
Cheers
Henrik Andersson
http://www.houseoftest.se
http://twitter.com/henkeandersson
at October 1, 2009, 11:37 pm:
5 second answer:
Since it’s a wiki, I’d get the team to eat their own dog food by using it for a project repository: story details, planning notes, discussions, etc. Hell, get the entire company to use it.
Cheers,
/bruce
[and we do! (You're onto me, as they say. Would you have them use the daily build do to it, and update it every day? Or continuous build?]
at October 2, 2009, 1:36 am:
If it’s an application like twitter, myspace, or facebook, you probably want the freedom to update anytime you need to, as soon as you need to, regardless of how many users are currently online. And, as they say, if you want to become good at something, do it all the time. That seems to indicate that you should try to push updates once it’s cleared your continuous build.
Sounds scary, but there are a couple of obvious advantages to doing this. First, if you can go a couple of weeks of pushing CI builds out without breaking something then you can say that you’re ready to handle a live rollout. Plus, nothing like the possibility of busting the company wiki and having the CEO give you the evil eye to encourage people to think twice before committing that change.
Crank it to 11.
at October 2, 2009, 6:57 pm:
There are some questions open for me to make a more informed decision about what might happen in the next two weeks to three months. Anyways I started to draft some basic ideas, that I would origin the approach on: http://blog.shino.de/2009/10/02/how-i-would-test-this-part-i-the-basic-idea/
at October 12, 2009, 3:18 am:
Thanks for the challenge Matt! I posted my response on my blog. I’d love to hear your thoughts (as always)!
- Joe
http://joeharterqa.wordpress.com/2009/10/11/how-id-test-it-re-matt-heussers-challenge/