Software Test & Performance Collaborative

Community, Resources & Knowledge Sharing for Test & QA Professionals

Matt Heusser’s Blog

Testing at the Edge of Chaos

/Performance/ Improvement, not “Process” Improvement

By Matthew Heusser on November 25, 2009 | 5 comments.

Would you be willing to go on a little experiment with me?

Pretend that you are an executive at an American Company in 1965. The story of Henry Ford’s Assembly Line, and the billions he made from it, have moved into the stuff of legend. “Everyone” knows that one way to make big money is to create a great product – one Americans wonder how they used to live without – that is disposable. Then, you break building the product into component tasks, get a cheap labor group of specialists to work on each task, and sell it to every single household, over and over again.

Enter Ray Kroc, businessman extrodinare. He notices that the McDonald’s Brothers have a better product – a burger – that people are willing to stand in lines for. Kroc buys the franchise, standardizes and defines every step in the entire operation, then takes it national. Then global. During his lifetime Kroc amassed a fortune of five hundred million dollars. Along the way, he acquires a major league baseball team.

What lesson do take from the McDonald’s story? Is it that the way to success it through standardization?

For many corporations, the lesson they took was exactly that.

I think they got it wrong.

You see, the McDonald’s Brothers had a great process to start with. They had a burger that was better than everyone else. So if you’ve got it all figured out allready – if you team has a “secret sauce” – it might make sense to do things in the same way. If you don’t; if the process is inefficient and slow and involves a lot of bottlenecks, then standardizing the process is going to prevent individual improvement.

But there’s more.

Part of Kroc’s goal was to produce a hamburger that was the same every time. His process was actually identical. In software development and testing, every project is different. We can try to talk in a language of standardization – “gathering requirements” or “requirements inspection” – but the actual requirements will be different every time.

Oh yes, there are rare cases where we really are doing very similar things; report generation, or just-slightly-different-each-time website development – but both of those are begging for some really strong 22-year-old codejockey to come along and write a code generator to make the problem disappear.

So in many cases in the corporate world, I’m suggesting we don’t need process improvement – at least, not the institutionalized, defined, audited kind. I suspect we need performance improvement. Let me give an example.

This summer, our family took a day off and went up to Michigan’s Adventure in Muskegon. The kids had a blast on the roller coasters and the waterpark, and I had one day to actually not think about technology or process. Until lunch, that is.

So it’s lunchtime, and I’m waiting in the line at the snack shack. A long line. And a long wait. They guy behind me starts to complain “You know, we come here every year. The kids love the park, but I spend the day in line. They need to do something about these waits for food. They really do. Maybe have one teen-ager work the cash register, another cook, and a third assemble the food. I don’t know. But they need some kind of process improvement to improve speed, that’s for sure.”

Man. Try as you might, you just can’t get away from it, can you?

So I start to talk to this gentleman, and I realize something about the teenagers behind the glass. They just aren’t moving very fast. I remember being a teenager myself, and the kind of jobs that paid the same minimum wage if I hustled or If sand-bagged.

A bigger assembly line with stations wouldn’t help the process. The food was pre-cooked, all the workers had to do was assemble it. What the workers had to was move faster.

There are lots of ways to accomplish that. One might be the McDonalds route; if the park was willing to invest the time and effort in hiring real process engineers. It’s easy to spread the cost of process engineers over 2,000 restaurants; over a half-dozen non-standard-sized snack shacks in a west Michigan Amusement Park, not so much.

Another would be simply to count the number of meals served in an hour, or the average wait time of the line. Then hand our raises and bonuses. You can do that with the register itself or with each manager simply using a stopwatch during lunch breaks. Or you could calculate the profitability of each shack and let the workers figure out the best way to make it more profitable. Oh, you might have to watch out to make sure the now-motivated workers aren’t serving undercooked food in order to make the numbers, but if you’re going to organize a group with managers, well, that’s part of the job.

Now, back in the world of technology, things are a little big harder. First of all, it might be safe to say that a dollar is a dollar or a minute wait is a minute wait, so those metrics are valid – but line of code are not creatd equal, and it is relatively easy to abuse them, or ‘test cases’ or most other engineering ‘metrics.’ Which makes management by output a whole lot harder then conformance to process.

So it’s harder. I don’t think that means we should abandon it.

In fact, quite the opposite. I think it means we need to work to understand the dynamics at play. We may not be able to turn the performance of a team into a metric, but I think we can manage for output. And management for performance is one of the things I am excited about right now.

I suspect that an outsider might see that kind of high-performance testing, snot and laugh that the work is the ‘very edge of chaos.’ Fancy that.

UPDATE: I’ve been reflecting on this, and I really have a strong emotional reaction to the standardized process thing. I dislike it. When I’ve worked with ‘process engineers’ who think in this command and control style, that want to separate the worker from the work … well, the reality is, they would not enjoy working in the very system they created. I expect they would hate it. Yes, they have ready-made excuses, like there are different kinds of people, some like to be told what to do, and fit well into the system, and others are more intelligent and independent and could not work in such a system, etc, and so on.

I don’t buy it. That is snobbery of the highest order; it’s nothing less than a new kind of class warfare, that replaces peasants with white-collar workers and nobility with executives.

It is also not treating people the way we would like to be treated. I wonder, as we Americans are about to embark on a two-day binge fest of Turkey and shopping, er, um, I mean, a two-day Holiday to thank our Creator for all the blessings he has bestowed upon us. I wonder if we should also stop to question our cultural assumptions about ‘process improvement’, and if they are an example of the Golden Rule … or not.

Update 2: At least one person read this as “Matt approves management by metrics.” Well, wait, that is a very complex subject. Please consider the post above in light of my other writing on metrics – this article comes to mind. How to do the performance improvement work in a knowledge-worker world, I suspect, deserves it’s own set of blog posts.

Keep it tuned here. More to come.

Comments (5)

Adam Knight
at November 25, 2009, 2:20 pm:

Nice example and a good point about the idea of a ‘one size fits all’ process being impractical.

Seems to me that they key to what the staff in your example really needed was motivation. This could be driven through “raises and bonuses” as you suggest or simply an improved sense of team spirit, shared responsibility and making their work more enjoyable. The staff might then be encouraged to identify their own ways of acheiving performance improvements. They would certainly be in the best position to identify improvements that could be made.

[That's a good point, Adam. A little time away from the cash register for all-staff meetings to talk about the direction and goals of the company, or free passes to the waterpark on your day off, where your team builds a skit for the next all-staff ... people are motivated by all sorts of different things. Somewhere in that park, there is a manager in charge of that shack. Actually figuring out what is taking so long, helping to get it done faster, and figuring out how to help people with their internal motivation, is, very likely, a neglected part of his job.

That manager might have to do something different for each person; I am okay with that.

Sometimes, he just might need to express appreciation and thanks. Speaking of which thank you for your contribution.]

Adam.

Julio Vazquez
at November 25, 2009, 3:00 pm:

The reality is that the key to achieving the ability to increase throughput is process. Without a process, there is no way to improve performance. In fact, most fast-food chains do have target goals for delivery of product to the customer. Obviously, the restaurant that held the concessions at the cited park does not. If that area is contracted out, then the park should look for another vendor; period. Having people wait in line leads to dissatisfaction and decreases your repeat revenue. Not what you want.

How does this apply to code? Simply put, if there are modules or functions that have already been tested and proven to be the most efficient, then those artifacts should be reuse. While it’s great to be creative and write code to solve a problem, if there’s an existing solution you should examine it and determine if you can do it better. If not, you should use the existing code. If you can write it more efficiently, write it and submit it for the library of reusable content.

[No argument so far. A little nuance below, maybe.]

More lines of code does not mean you are a higher performer, less lines of code may be. How quickly the code processes information is the key measurement of performance and a developer should be held to that metric above all others.

[I'm not sure what you mean by "how quickly the code processes the information". If a dev spends a year writing a routine that runs in ten minutes to write checks, and the other dev writes it in a week and figured "it can run overnight", who is the better dev? I suspect you mean something a little deeper, but I'm afraid I don't quite grok it yet.]

Of course, this is only my opinion.

[I do respect your points, and, to some degree, that's what I was trying to get across when I wrote that a lines of code are not created equal. I do think however, there is some nuance we both assume that an uncritical reader might miss: For example, the process of figuring out what code should be reused, and how to reuse it - how, exactly, do you standardize that? I suspect that you can have rules of thumb, but the individual contributor will have to actually think. And I am okay with that.]

Kevin Baribeau
at November 28, 2009, 9:47 pm:

Hi Matt,

I enjoyed this post, thanks for sharing.

I think this kind of thing ties back to “individuals and interactions over processes and tools”.

[Totally man; I have an article coming out shortly on the people side of agile. I suspect you might enjoy it.]

There is a lot of value in a good process, however, if you’ve got someone THERE, who’s actually getting it done, and they have a better idea, you’d better listen to them. I think we’ve got to find ways to look for good processes, but at the same time encourage innovation, and not be afraid change (or failure). It’s not an easy balance.

Markus Gärtner
at November 30, 2009, 5:47 pm:

Hi Matt,

there a two stories I can tell from McDonalds on my own. First of all I was today at a local McDonalds and the personell there was creepingly slow. Some of them. There were clear queues, two counters open at lunch time, come on. This can’t be true. Well, it was. Not that fast for a fast food restaurant at all.

[I suspect that the workers were simply being slow, right? It's hard to "process improvement" that away, isn't it?]

Second some years back they had a one-minute bet at McDonalds Germany. After placing the order until it got served there was a bumper that counted down 60 seconds. Waiting 60 seconds for your meal seem rather short, but there fits a whole bunch in 60 seconds. (Maybe it was 30?) They removed it since it didn’t serve their needs. Maybe the personell at the store I was today ruined the company, I don’t know.

[They certainly ruined that store ...]

Just my 2 cents.

Leave Comment

Name – required
Email – required
Website