Software Test & Performance Collaborative

Test This – Finale

By Matthew Heusser on October 28, 2009 | 6 comments.

It’s down to the entries and the grading for the test challenge. I would like to start with some of the actual bugs found in test; then we can ask “would the test plans have found these?”

Activities widget: cannot select Page Tags in My Conversations
Activities Widget: my conversations ignores event filter
IE: JS error during Activities widget testing
Activities widget: Showing menu cut off by widget boundary
IE6: Activities Widget: Cutoff slightly in left col at 1024×768
Activities widget does not maintain “Post Signals to” value
IE: Activities Widget: Vertical padding in comment activities
IE: Activities Widget: Edits show up in my conversations/signals
Private messages are not all grey in activities widget
Activities Widget: Edits and Comments for People I follow are the same thing
Activities Widget: Unable to see follow/unfollow events in ‘all events’
Activities widget does not display edit summary signals
Activities widget does not display My Signals
Humongous stacktrace upon adding Activities widget

—-> Yes, we could do a whole blog series of predicting bugs and test strategy. For now, as you can see, most of these errors are with either internet explorer, the displaying of specific events, or with signals.

The Entries

To grade the entries, I used three measurements: How through was the test plan, how well-written was it, and did I think it could be executed reasonably fast – after all, one of the requirements was that it work in an agile shop. Now, some of the test plans looked like they took longer to write than I actually had to test the widget, so I tried to give credit for thoroughness but not speed.

Two of the entries, Joe Harter and Justin Rohrman were unique, in that they did not actually define test cases – instead, they laid out high-level functionality and how much time they would devote to each element.

Two other entries – Marlena Compton and Joe Harter again, tried to engage me in a discussion of what the functionality was and how it would be used. They used to this both to fill out, and to ‘check’ their mental model of what the functionality was before designing tests around it. This was what I was looking for; in addition, I would have given out bonus points if they had tried to have a negotiation about how long testing should take. (Justin and Joe’s proposals to test were great, but Joe both wanted about twice as long to do the testing as I could reasonably give him, and Justin wanted a more modest, but still too large, percent.)

I was also looking for other ways to mitigate risk, like having the company actually use the software prior to it’s release to production.

There were other ideas that impressed me. Markus Gaertner pointed out that Socialtext offers a ‘free 50′ program, and that he could actually test the software, then work backward to define his plan. (Speaking of which, his plan is pretty good.) That was a novel approach, and I give him a ‘tester reputation point’, a James Bach would say, for the idea.

Ajay Balamurugadas, Parimala Shankaraiah, and Sharath Byregowda used a similar, traditional, test strategy template with details filled in. I was impressed by the thoroughness, especially Ajay – who scored highest of all entries in that category. What struck me as I read he entries is that I believe such an approach would slow down the team.

It’s hard for me to explain why, but I’m left with a feeling that a team attempting such a strategy would become bound by checklists, documents and process. I certainly don’t see anyone creating a twelve-page document, then turning around and executing the tests in the three to six hours I had to do the work.

Competition Results:

First Place: Marlena Compton
Second Place: Joe Harter
Third Place: Ajay Balamurugadas

To the winners: drop me an email (matt.heusser@gmail.com) with your home address, and I’ll get the prizes in the mail.

To everyone else: I tried very hard to grade fairly based on those three categories. You’ll notice, however, that the first and second place awards went to the people who actively engaged me in a dialogue about my expectations. There’s a testing lesson in there; I’ll leave it to you.

This completes my current challenge; what are some of yours? Please feel free to write them up and leave a link in the comments. If I like one, I’ll endorse it …