Software Test & Performance Collaborative

John McConda
at November 18, 2009, 2:48 pm:

I would like to see self-paced online modules that teach basic programming skills tailored to testers. Maybe have a separate module for each of the more popular scripting languages and how to apply those to testing situations.

I’ve also greatly benefited from the online Black Box Software Testing (BBST) courses offered by AST. The biggest barrier to students taking (and finishing) these is time commitment, though it’s certainly less than an online college course.

[What if these were offered during the day and locally? There would still be homework. I wonder if companies would be interested in BBST foundations in two or three days? I know AST tried this a few years ago and couldn't see many seats, but I wonder if this was because people didn't value it, didn't have budget, or just plain didn't know about it?]

Lanette Creamer
at November 18, 2009, 3:44 pm:

Hi Matt,

This topic is near and dear to my heart. If I see one more company go only to web based self-paced tutorials I am going to lose my mind! Yes, that is ONE way to teach and it is cheap, however, usually they are not well done and the problem with them is it is extremely hard to get any time to do them. I’ve started 15 of them not to finish and it isn’t because I’m lazy. It is because my project is very busy and the competition for my time is really tough right now.

[Certainly "run of the mill" online tutorials are horrendous. Do you think they can ever be done well at all?]

So, here are the attributes of useful training right now to me: 1. It includes real live demos. 2. After the demos it includes interactive exercises. I mean often, like every hour! 3. It helps me test faster and better, not just one of those. I need both. 4. It is technical enough. What I mean by that is the old style training is not cutting it anymore with who testers are now. Almost all testers are programmers of some sort. The audience has changed and the training has got to change with it.

[Are they? I wish we could do some sort of survey on this. It seems to me that we are splitting into people who's role is tool-makers and people who's role is tool-users. I think we could use education on frameworks for the former, and a little education on scripting for the latter. In general, the dichotomy bothers me. There are also a number of companies where the testers use the software - doing a kind of very advanced, high skill exploratory testing that many are quick to label "ad hoc" or "manual" - and doing just fine at it. I'm not sure how large that group is.]

I like the idea of an live online web course with interactive exercises and peer review as part of the class. A class that went for one hour at a time for a week would be nice.

[I'm really starting to think there could be a market for a BBST-like course, offered during the business day, coupled with some mechanisms for students to collaborate. The formal BBST course, which is a trademark owned by the Association for Software Testing, is entirely about testing, not dev/testing, though. ]

As you know, I went crazy yesterday over Adam Goucher’s scripting class materials and the reason is because it is at the right level for me and that is tough to find right now. It seems like all I find are either for the non-programmer entirely or for a person who writes code all day every day and not something for me.

Topics-We need more on the new web testing tools and on getting them working together. [There's definately lots of room for improvement in how we talk about testing web apps, I agree]

We need more on scripting (the Cyborg tester idea), we need more on human assisted testing (when the automation is only to help the tester spend more time evaluation and less time on setup, action, and parsing machine data). Also, I need a class on translating tester to managerial speak. ["Aligning testing with business strategy", maybe?]

I look forward to seeing what you [we?] come up with!

Alex
at November 18, 2009, 4:04 pm:

I am skeptical of training in general. Too often companies believe training will solve problems. It doesn’t work if the people that take the course come back and don’t use it. There has to be a commitment on the business side to integrate the concepts and allow time for learning and establishing it as part of ongoing process.

Take automation. We’ve had many classes offsite, onsite, and I teach many classes myself. The majority learn a bunch of cool stuff, then go back to their teams and are never given the time, or never take the time to make those changes and adopt the concepts learned.

I realize I’m not answering the question. I guess what I’m saying is that all these examples of training – on-site, offsite, classroom, example, CBT, whatever. They’re all good. But there also all worthless if there’s no commitment to allow the people being trained to learn, practice, use, and integrate that knowledge.

Personally, I prefer classroom sessions with lots of interactivity. Lecture on concepts, then do hands-on, highlighting the concepts during the exercises. Offsite and onsite both have their advantages and disadvantages.

[Hey man, I hear you. "Drive by training" is one term for that kind of thing. A few years back, I sat through some Oracle PL/SQL training one week, the instructor got great eval sheets. I was disappointed. A senior manager asked me "Matt, everybody else loved it, what's your issue?" and in reply I told him to ask his people "What are you going to do differently on monday?"

Let me ask you this Alex: What if the trainer came to your site, spent a day or two immersing themselves in your business and problems, then led a workshop on concrete ideas for improvement, combined with pulling out training materials that are appropriate to the problem at hand? What would you think of that?]

[Also - I think your comments about test automation training are spot-on, for a variety of reasons. I think it fails a *LOT*. I believe there is a lot of snake oil and broken promises around the subject. I've got a blog post in my hip pocket on the subject. I wonder if a completely new approach to teaching it should be considered, or if a homebrew workshop would have better chance of success, or if, as an industry, we should focus on training on something else, and leave automation training to, well, training of programmers ...]

Michael
at November 18, 2009, 4:48 pm:

A lot of what I have seen in training has been mentioned, it’s either too low or too high level for someone like me (I can hack code but I am not a programmer, I can build a test framework by putting pieces together but I am not going to write one)

[I wonder what percentage of testers in the world fit into this bucket. 10%? 25%? Enough to have a voice, I suspect ...]

plus, as you said, once I get back what am I going to do different? It’s been tough for many companies to pay for QA training, I fought for a year and a half to get someone in our group to go, and I finally got permission to go to STP two years back, Developers had something on schedule every couple of months. Until the companies are going to realize they need to add value to their testers not much is going to get done, I think part of that in some companies comes from the fact that some higher level managers don’t really know how to integrate QA.

Now, what would I like to see? Labs, and more of them, you can lecture me till I am blue in the face but unless I get to use what I am seeing I won’t retain it, training by lecture is nice of you have an abstract concept to discuss and do so but learning how to do X-driven Testing is not going to do much for me if I don’t put it in practice soon after I see it. Web based is fine, as long as you have a presenter who knows how to do it and keep people engaged, some don’t, and with that sort of environment you can get many different groups in. Keeping the classes to people with the same level of experience helps in some respects as well, that way things move along at a good pace, I know its hard to judge that but maybe putting it up front how the class will be structured will make it simpler. I’d love to see more about frameworks, integrating them into existing systems, or even about how to integrate them into the workplace, there is plenty out there about using a Fitnesse or marketing material on many of the big frameworks, but what about going over the basics in what to look for in Open Source frameworks and tools. There are plenty of tools, but a question I see alot in some forums is that someone is new and looking to build a framework and they have questions, this may be a help. Otherwise, anything that hits Unit Testing or Programming for Testers, what to look for in code, or how to be aware of what can be a problem – its a big space I know, so anything can fill it.

James Bach
at November 18, 2009, 6:49 pm:

My Rapid Testing class tries to focus on skills. We have computers in the room and I do several computer-based exercises with them. The class has been in development for many years and I’m pleased with it.

But what I would like to do is an order of magnitude deeper than that. Tait Electronics gave me that opportunity recently: a rapid testing clinic. Six test leads and I tested one of their products for two days in a conference room. The point was to apply all the skills of testing. My job was to lead and document. Worked out great. All the leads had previously taken the RST course, so we were just practicing the skills.

[Well, we only have seven comments here, so I don't want to call it a trend but one category I would like to see more of are interactive workshops where the students walk out with a test strategy or have done a new method, and see value both in 20-random-strangers AND in having a consultant working with a specific company, so the problem domain and technology stack is singular.]

Dave Whalen
at November 18, 2009, 8:09 pm:

I think there is a huge need for software test training!

The real problem is what to teach. Each of the certification bodies has their own methodology and body of knowledge, as do all the “test celebrities”. Sadly their opinion seems to be “my way or nothing”. Then there are the Agile disciples who think Agile is the only process worth caring about.

[I dunno. I find a lot of the context-driven people complementary. I'd rather focus on exercises and skills in general than a specific "pyramid" or "process." I like James's idea of testing lessons as opposed to degrees and certificates that are single-time earned. ]

Seems to be a huge need, but limited training budgets.

[Agreed about the limited travel budgets. So how do we overcome that? Should we try to give more local training (not travel costs) or on-line, or something else? One idea I had was an organize-it-yourself discount; if a local user's group delivers 10+ people for a three day course, they can get a seat free. I know a few trainers who might accept a deal like that.]

Ron Pihlgren
at November 18, 2009, 11:10 pm:

We brought Michael Bolton in about a year and a half ago to do a 1-day version of RST and feedback was 50/50 with the negative 50 mostly comments about how they didn’t see how this would help them do their current job, which aligns with what Alex says above.

Your comment to that post:

What if the trainer came to your site, spent a day or two immersing themselves in your business and problems, then led a workshop on concrete ideas for improvement, combined with pulling out training materials that are appropriate to the problem at hand? What would you think of that?

…and what James Bach describes in his post seems to be the approach that I think has the most value and that might be something I could gear up to sell to my management.

James Bach
at November 19, 2009, 1:28 am:

The Context-Driven guys do not teach “my way or nothing.” We teach our way, sure, but we teach it in a manner designed to help you come to your own way. We certainly challenge ideas. But challenging students about how they think is not the same as saying “you have to do it my way.”

That there are many solutions to problems, and many different flavors of problems is a founding idea of the Context-Driven School. We’ve been preaching this creed since 1999.

A big problem with putting realistic exercises into test training is time. I think we need to get away from the idea of a testing class, and embrace the idea of testing lessons– just like music lessons.

Thomas Ponnet
at November 19, 2009, 11:27 am:

Matt, you wrote: “… but instead teaching courses with practical exercises designed to yes, help us actually test.”
My question is, what do you want to test?

Trying to create a training course for testing is like trying to create one for programming. It’s far too high level and should be broken down I think. How it should be broken down is open for discussion, I’ll throw some ideas on the table.

[Indeed. I agree that one next step might be to create a model of all tester skills and then create a mechanism for different courses to 'plug into' those skills. I wonder, however, if another way to do it would be to create a bunch of 1-hour to 1-day 'modules', which people customize.]

- Experience of tester
- Goal of the training course – is it to be an overview of say test approaches, is it about Selenium and scripting focussing on a specific problem, learning the test vocabulary, how to test in agile projects, etc
- Test approach to focus on, ie manual scripted, exploratory, semi-automated supporting some form of manual, automated, etc
- level of testing required/mission, ie high level smoke tests, detailed – try and break the system, test that requirements can be signed off, etc
- Is the tester working in a group or alone, responsibilities of the tester. A one man shop has other training requirements that someone working in a group of 20 with a rigid hierarchy. Someone looking at UX issues needs a somewhat different skillset than someone regression testing websites against x browsers.
- The way individuals learn

Having that out of the way, I attended the Rapid Software Testing course given by Michael Bolton and left the course knowing exactly what I would do different on Monday. The course is a few months ago now and I still go back to it. For me it came at the right time. It was three days long – good length, had experienced testers there wanting to know more so there were peers at the same level, also very important, we tested a lot, very hands-on and it didn’t hurt that the trainer was excellent.
If I had done the course at the beginning of my career I might have worked differently in the last couple of years. More likely though it would have been too much at the time, I needed the basics before someone could tell me what’s wrong with what I learned and question my approach. My point is that training courses need to be aimed at specific levels as Lannette has pointed out as well.

One point that I believe the community either forgets or doesn’t want to raise is that some people simply are not interested in learning about testing, even if it’s their job. They’re happy to attend a training course, relax from the stress at work, maybe learn a bit or two if it’s not too much hassle and doesn’t mean too much effort. We keep hearing that most testers haven’t opened a book related to their profession in years.

Do we want to create training courses for this group? Or do we want to create training courses for the others who make an effort to read articles, books, etc? I think there’s a huge difference between these two. I’d say that current training companies cater largely for the former while this discussion seems to be around the latter group. I’ll get my jacket but this is something that I feel strongly about.
I don’t mean to be elitist, quite the contrary – everyone has the chance to educate themselves. My preferred choice is something like the RST training course, even better if it’s on site and some day to day projects can be used as an example. This way it’s halfway between a test consultants job and half trainer but with a subject that is relevant to people’s daily work. Unfortunately this is only going to work in bigger companies due to costs involved with that approach.

Cem Kaner
at November 19, 2009, 2:41 pm:

As with anything, if it doesn’t have to work, you can do it really cheaply.

Well-done online courses actually require more instructional work than live ones. It takes much longer to develop the materials, the courses must be much more precisely planned, interaction is more difficult to manage, student participation is more fragile, group work is harder to manage and often harder for students to arrange. In courses that involve grading, giving students feedback is more time-consuming than in live classes and student protests / confusions are harder to manage well.

I’m not speaking only from my own experience (though I will say that online BBST has been enormously harder and slower to develop and teach than live BBST). Every study that I’ve seen on academic costs (development cost, grading cost, etc.) shows that the online model is harder and more time consuming–if you want to do it well.

In addition, not everything teaches well online. The next WTST (workshop on teaching software testing, January 29-31, contact me, kaner@kaner.com, if you are interested in attending or see WTST.org for an official announcement soon) will focus on teaching implementation-level testing (e.g. unit testing). I hope to hear different news this time, but so far, I don’t know anyone who knows how to teach skill-based implementation-level testing online. I am aware of some people who think they can teach skill-developing programming courses online but haven’t yet met/discussed-with them.

(Of course, some people think that learning to memorize definitions or work trivial exercises is skills-based and they then claim that it is easy/cheap to do this. But I don’t see that as skill-based teaching.)

At the moment, I think we have a methodology / model for online teaching (see http://conference.merlot.org/2008/Saturday/kaner_c_Saturday.pdf) that I think handles many black box testing skills well but we are having to write new skill-focuses workbooks to support the techniques courses. I think the model will stretch easily to courses on software metrics, basic math (calculus, discrete math, applied or theoretical probability & statistics), but it doesn’t stretch well to the courses I’d like to teach on requirements analysis, software design, or implementation-level testing.

Putting practical exercises into a live course is very difficult. People work through exercises at very different speeds. Working from student self-reports or watching check-in times across iterations in our source control system (consistent with student self-reports), 4th-year and graduate-level software engineering students can vary by a factor of three (Jane takes 6 hours, Joe takes 18 hours) or more. Half-hour live labs in my classes can take 15 minutes to 2 hours. Designing exercises to minimize this variability creates exercises that feel, too me, too constrained and too artificial. And there is still enormous variation. The asynchronous online format is better for this because people can work on their own time. But managing groups is harder than in lab courses. My experience with practical exercises in live _commercial_ courses is exasperating. I can never tell in advance which groups will tolerate what level of work (what level of difficulty, what proportion of time). Some students get frustrated with what they feel is a slow pace, some students are infuriated (sometimes because they feel humiliated) by examples that are harder for them to solve than they expect given their self-perception of their expertise. I don’t think this is impossible, but it is very hard.

Several companies want interactive courses in short time periods. I think they are setting themselves up to fail. I am not surprised at the comment that the 1-day Bolton class was disappointing. Don’t blame Bolton. If you want a cognitive-skills class, give it enough time to succeed. The more time, the more the instructor can create a series of experiences that evolve students to meaningful results.

[Thank you for the serious and well-considered response, Cem. I will have to digest this.]

Ron Pihlgren
at November 19, 2009, 5:01 pm:

>> I am not surprised at the comment that the 1-day Bolton class was disappointing. Don’t blame Bolton.

Didn’t mean to imply any blame on Bolton. In my email to him after the class I told him that the feedback told me more about my team than about his class which I thought was excellent. Your point about a short, one-time event is spot on. The class was a learning experience for me and not about what was taught in the class.

Marisa Burt
at November 19, 2009, 7:33 pm:

The best software test training I have had has been the interactive workshops such as James Bach’s RST, Gerald Weinberg’s AYE, etc. They add to my testing knowledge base and make me think. I strive to continually learn and thus increase my skill set. Attending these sessions allow me to do that and then figure out how to apply it to my situation back on the job. Just attending training isn’t enough to solve all your problems, you have to learn and go back and apply what you have learned to your individual testing situation. Like in college, they teach you how to think, and give you something to think about. I like the idea of Testing Lessons as James mentions here. It takes time and continued effort.

jon hagar
at November 23, 2009, 2:26 pm:

As several posters have addressed at least in passing, as much as “what to teach”, for me it is “how to teach it”. I have been studying learning and human brain for a while now. I now know how little I know on the “how”. In any class or training, some people will learn and some wont (you get good evals and bad). It is another context dependent “it depends” situation. I think many different methods of “how” must be available to the test world. Several posters above have been working on this. It takes a long time and evolution as much as revolution, though as in biology, maybe it is time for a “leap forward” with some mutants. Borg anyone?

[I agree with your sentiments Jon. Dr. Kaner's comments certainly gave me pause. Coming from someone who has worked in this industry so long, and put so much effort into both mastering testing and mastering how to talk about it, I was more than a little humbled when he expressed some of his challenges. (Knowing a little about you and your background, that makes twice.) Right now, I have hope for a 'better' online model for training, as well as local training that's a little more focused, and the on-site consulting/training engagement. I'm not sure how much of that the industry is ready for. Let's keep talking.]

Selena
at November 25, 2009, 2:33 am:

I agree with other comments about immersive learning experiences. Those offered by Jerry Weinberg and his many associates, including Michael and James, are spot on for allowing people to learn new concepts, and apply and practice them – which is really important to have the new learnings stick. James’ comment on creating learning opportunities as ‘lessons’, similar to music/swimming/dance lessons, is a great perspective. New skills stick when they are practiced again and again and again.

In my experiences in teaching and coaching testers, the experiential workshops with lots of practice time is crucial for having folks try to apply new concepts on the job. I’ve been most successful though when able to use ‘real life’ projects in the workshops, allow participants to review and discuss others’ work in the workshop to learn from each other, and revisit a particular topic in a follow up workshop a ’short’ time afterwards to practice some more. Coaching on the particular concepts by pairing with participants in their day-to-day work after the workshop has helped too.

I would love to see lots of local training crop up everywhere that offered this type of learning, via coaches and consultants as well. I’m personally heading in that direction, and am able to trial a lot of things out in my current organization. It would be great to pair with one or two others locally to create well-rounded workshops and consultancy gigs that offer this (my technical skills are rusty, for example, and I would rather collaborate with someone else to flesh out that portion, then polish my own up at this time).

[Where are you located? I might not be there, but I know a lot of people ...]

I’m not sure how to do this with online training, though haven’t thought about it much.

[Well, "keep it tuned here" as they say. In the arena of web-based training, I think we are all learning and experimenting, which is a really fun place to be.]

Additionally, we need more conferences that offer learning opportunities that are experiential! Let’s have conferences that are purely experiential, focused on different domains and different topics, where testers are immersed in learning and applying concepts. As an aside, I am starting discussions with someone in my area to create such a conference for testers (think AYE but for testing skills) – if you are interested, find me on Twitter (sdelesie).

[AYE for testing skills ... fascinating. This sounds a lot like the peer workshops James, Cem, and Doug Hoffman have been doing for years, but those are more retrospective and experience based, instead of exercise based. I'm @mheusser on twitter, and it sounds like a fun idea to kick around.]

Great discussion Matt, thank you for starting it!

[Thank you for being a part of it.]

Michael Bolton
at November 25, 2009, 2:49 pm:

Over the last little while, I’ve been running some of our exercises over the Web, on GoToMeeting. On a one-on-one basis it’s not too bad, because I can see what the individual tester is doing and have a direct conversation. One-to-many would be very difficult. What I miss most is the direct human interaction, the ability to see faces, note expressions of surprise or puzzlement or enthusiam or disconnection. That’s very, very difficult to detect over a Web connection. Virtual presence isn’t real presence by any means.

Meanwhile, I’m with James on the coaching and collaboration thing. We both now offer a free day of that when anyone books us to teach Rapid Testing in-house. That time can be used for mentoring, hands-on testing work, strategy ideas, talking to senior managers; whatever the client likes. That’s been pretty successful, but I’d prefer to do the class and then have a week of the coaching/consulting work, followed by periodic followup visits.

I do remember the trip to Microsoft that Ron mentioned. My reaction to the team’s reaction was like Ron’s: the feedback revealed a lot about the way those folks had previously been trained to do testing. My sense was that these were people who had been confusing testing with checking, and treating testing as a simple programming problem. That’s an idea which has been rife in certain parts Microsoft from time to time, and I think it comes largely from Microsoft’s exaltation of programmers, often at the expense of non-programmers. If you’re engaged mostly in checking, the cognitive stuff isn’t important and doesn’t make much sense.

The most valuable exercises that I’ve found (and, I immodestly claim, developed) are the ones that fool people (in a safe environment). If you learn how you can be fooled, I think there’s a chance to avoid being fooled quite so often or so badly. Testing is about not being fooled.

Here’s a hint: if someone laughs at some point in your exercise, learning has occurred.

—Michael B.