Matt Heusser’s Blog
Testing at the Edge of Chaos
Estimates – III (Serious this time)
I’ve re-titled the last Estimates III as “Models – I”, as it really deserves it’s own serious.
Pressing on …
(AKA – Stupid Project Management Tricks; a True Story)
The code is big stinking mess, all over the floor. The team is dependent on a vendor who hasn’t delivered a good build to us yet. We have no idea when anything is going to be ready. My best bet is three months out, but nobody really knows. Senior Management is Antsy. What will the project manager say?
Ok, so I wasn’t there, but I know what happens: He walks into the big office and says “THE PROJECT WILL BE DONE IN FIVE MONTHS.”
Where did that number come from? I have no idea. There was probably a gannt chart somewhere that added up to five months, but nobody really knew, and nobody technical trusted the vendor.
All I do know is that senior management went home that night – and they slept well, because they had a date. A firm commitment. None of this “gee, we don’t really know, how long does it take to find your lost keys?” business. A firm date.
It was bogus, but, whatever. They could sleep.
… time passes …
Four and a half months later, and it becomes obvious that the team won’t deliver in two weeks. I’ve been pulled off to work on another, different death march(*), but I take a general, cynical tester’s interest in the project.
Again, the word comes down from on high “WE’VE HAD ANOTHER SLIP. THE PROJECT WILL SHIP IN FOUR MONTHS …”
Of course, four months goes by …
Eventually, the project was cancelled: Over a year late.
Now, I criticize a lot of things about that project. 99% of the time, I think the project management was irresponsible, incompetant, and (possibly) unethical.
Then again, 99% of the time, senior management was able to sleep at night, because, dang nab it, they had a date. None of this “uncertainty” business.
Which reminds me of an old quote by DeMarco/Lister – all too often, people would rather be certain and wrong than uncertain. After all, it’s better to be a team player than be negative.
So, the basic technique is this: If you have no idea when you’ll be done, make up a date so far in the future that you’ve got a good shot. When it becomes obvious that you cannot hit that date, well … make up a new one.
Or, in other words: When uncertainty is politically impossible, do what gets you rewards.
Like I said before, 99% of the time, I think this is an incompetant and bogus way to “lead” projects. But that 1% of the time, I think: Man, if you have no idea when it’s going to be done, making up a date real far away and slipping if you have to is a decent approach. (It can’t be too far out – otherwise you’ll get push back.)
All of that presupposes that you actually can ship at all. The problem in the story is that we couldn’t, and anyone with decent systems thinking skills could see that, advise management, and would, er, well …
… “find other projects.”
–heusser
(*) – Actually, to be honest, that one was a relief; it was a death march, and I worked massive overtime, and it was painful, AND IT SHIPPED.
Models – I
AKA – Tripping through physical models of the universe(*)
Aristotle proposed a model of the universe with earth at the center and the heavens revolving around it. This model could not explain why days were longer in the summer, or the seasons, but it could predict the behavior of the moon – and the heavenly bodies seemed to support it.
Copernicus and Galileo changed the model, and insisted that the earth rotates the sun, as do Mars, Venus, and the other planets.
The only problem is, well, it didn’t work. Copernicus thought that the earth traveled in a circle, and it actually travels in an ellipse. Gelileo’s observations could confirm the theory in some aspects, but in others, the math wasn’t quite right. It wasn’t until Newton invents the calculus that was get the planets traveling in an ellipse and equations good enough to predict behavior.
That is, er … good enough to predict behavior most of the time. Objects that were very small or very fast tended to be “off” of what Newton’s equations would suggest. Still, most things here on planet earth fit well into Newton’s methods; it wouldn’t be until Einstein that we figure out the equations to calculate space-time distortion for objects as they approach the speed of light.
That is more than the brief outline of scientific history – it is the story of the evolution of a model. At each level, the model becomes more precise, more detailed, more formal, more “correct”, and a better way to predict (or analyze) the behavior of objects(**). Also, that more “correct” models tend to take more time and effort to learn, understand, and master, and that a model can be “good enough” for the person solving it. Third Century British Farmers didn’t need Newton to raise crops, and High School students in Physics I don’t need Einstein to predict the distance a cannonball will fire.
Now, I say that lower-level models take more work and more variables. In a hierarchal organization, the higher levels do not have the time or the attention for those details; they are managing many, many more projects in a stack. The test project that is your life’s work is only a “phase” to the project manager, and one part of one project to the PMO. To the CIO the whole project is a “tactical objective.” To do the detailed model well, you need a lot of variables; the CIO doesn’t know them at that level, and the detailed middle-manager probably doesn’t have the authority to answer them definitively.
So we end up having these conversations where someone “just” wants a single number to put on a bullet-point on a powerpoint slide, and we say “it depends, what about …” and try to get the variables for the model. It can be painful.
Here’s one technique I have used with some success: Ask for the margin of error. When you’re told that it must be very precise (+ or – 5 days, for example), physically roll up your sleeves and say something like “ok, let’s get to work.” Then plan a half-day session to hammer the thing out, with the key players involved.
Building an accurate predictive model is, well, relatively impossible(**); ask anyone who plays the stock market. Far too often, we expect precise, Einstein-like answers when we only have the data of 3rd Century Farmer.
Perhaps, then, we should estimate like a 3rd Century Farmer “We expect to harvest at the end of 3rd quarter; it depends on the weather …”
Of course, there are other tricks – both for estimates and for models. More later.
–heusser
(*) – We use models and prediction every day. When we say that “Bob is a jerk, he won’t help”, we are making a stereotype, or model, of Bob. It may predict his behavior, and may even do so accurately, but it could be that Bob isn’t a jerk at all. He might be over-worked and going through multiple deaths in his family. As we improve our understanding of the world, we can ‘model’ it better – these are informal models, and they are far more common than formal ones with mathematical rigor.
(**) – “predict” is a funny word. Netwon’s equations allow me to predict the speed that an object will hit the ground if I know the height it fell from, but I am not suggesting that I have a crystal ball and can predict the future. Any project will have risks, and if they all come true at the same time, well – forget about it. The point is that more analysis allows us to make better educated guesses.
Estimates – II
As usual, some of my commenter’s have written my article for me …
Seriously, Ben Simo points out that is it always possible to give an estimate, but that all estimates are wrong (if they were right, they would be commitments). Shrini points out that we get lousy requirements for our estimates. For example, when asked:
“When will it be done?”
You probably have a vague and fluffy definition of “IT” and a vague or unrealistic definition of “done.”
For example, let me re-translate:
“When will it be done?”
Could be:
“How long will it take you to do everything I’ve written down here, with scope creep that _I_ deem reasonable, without a single defect or flaw – given that anything over a month is too long?”
Yikes!
Of course, I promised some actual answers. So here goes …
The “GAP” between what the software actually IS and how it was analyzed is a serious pain point in software development. That gap widens with time – so my first suggestion is to never run a project more than about a month. (Yes, require/dev/test/prod in thirty days.) Schedule any “big” project as a series of small ones.
A worst, you might be two weeks late. That’s not the end of the world.
If you have to run longer than a month, then periodically bring your code up to production quality. Develop features end-to-end, in thin slices, and add them to the system.
That’s not really an estimating hint – it’s a project organization hint. Estimates of less than a month are drastically easier than more. Moreover, they are small enough that an employee can feel a real personal commitment to the date, and work a little overtime to make it. (On a ten-month project, once you realize you’re late, overtime will just kill you, not bring the project back on track.) So you organize the project so you never have to estimate more than a month at a time.
Plus, you can prioritize the customer’s feature requests, so you build the most important thing first.
So what if that is not an option?
Ok, next idea. Do two types of estimates – realistic and pessimistic. Make the realistic estimate your goal date, and make the pessimistic a commitment back to the business. Within the team, talk about the goal; with management, talk about the commitment.
A third – Estimates can be done in two different ways. The first is an analysis task where break down the features into chunks and add up the chunks, or you take the spec and do some math to come up with an approximate number of test cases. The second way to do estimates is to actually start doing the work – at a very high level. Go ahead and _DO_ the design, or start to decompose the test cases, or write “Tracer bullet” objects to do the highest levels of disk, screen, and database interaction. Leave comments or “TODO” markers at where the next step will be. When you’ve finished that, come back to the TODO markers, estimate them, and add them up. Finally, add a little extra for risk – the bigger the risk, the bigger the buffer.
The thing is, the further you go into the project, the more you’ll know. I have one colleague who simply avoids making any commitments or estimates at all. He’ll say things like “if you say so” or “that’s certainly our goal” but never make a promise. I’m not advocating that – but certainly the further you get into the project, the more accurate your dates will be.
(In Rapid Development, Steve McConnell actually suggests that you communicate estimates in a range that gets smaller over time, with the larger number first. “Six to four months” sounds strange – but if you say the smaller number first, people tend to forget the larger one.)
Looking back, most of this is not how you should do the estimates – but how to communicate them so they are not abused or mis-understood. This can be very challenging; as my old friend Eric likes to say “Sooner or later your three paragraphs of very specific if-thens and assumptions are going to turn into a PowerPoint bullet with a date on it. Tread lightly.”
If that’s the case, then you have a few options. One is to hit any arbitrary date by structuring the project so that you can go to production every month. Another is to make no commitments and blame someone else. A third is to way is to continually and consistently communicate risk-adjusted dates back to your customers.
My final thought is to express the cost of every change. It’s usually not the project that kills us – it’s the feature changes at the end. And that’s fine – it’s good – it makes us provide software that has better fitness for use. When that happens, though, either make the cost explicit (by adding the feature and insisting on moving the date) or pay for it with your nights and weekends.
The choice is yours ….
I’ve been busy …
Last weekend I went to BarCampGrandRapids2, which turned out to be a wonderful experience. Dave Brondsma and Carlus Henry put together a great little un-conference; jointly organized, show up and present, if people come to your talk, great, if not, don’t worry about it.
The conference was sponsored by Gordon Food Service (free food), Calvin College (free space), and Google, which provided a speaker, a recruiter, and a few give-aways.
The most interesting talk I went to was probably Matt Michielsen’s “Industrial Automation for Techies” – learning about programmable logic controllers and such. Second to that would be the open spaces, where we talked about everything from SOA to the impact of universal GPS/GIS. No Geocaching, but maybe next time.
The weirdest thing for me was interacting with the attendees. We’re all a bunch of computer nerds, so a lot of people were just sort of quiet and passive. I found that I kept falling back on where do you work / what do you do / here’s my card / what’s the hiring situation kind of discussions. Nothing wrong with that, but it’s hard to get deep.
Now BarCampGR2 was on a Saturday, and it drew a totally different crowd than usual. This was no during-the-work-week junket at a posh resort. Attendees were computer enthusiasts; people who do this stuff because we love it – or who love this stuff and wish they could do it for a living. There was a talk on robotics. Instead of asking “what do you do?” a better question for something like BarCamp is probably “What do you care about?”
In the mean time, I am also reviewing a book manuscript for Addison-Wesley on the process of evolutionary software development, learning to play golf (badly), and I owe a few blog posts on estimates.
More to come. Also, if you’d like to preview a couple chapters of the Scott Bain book on evolutionary development, let me know. If it’s popular enough, I have permission to post (a little bit of it).
Estimates – Laws
I’m a big fan of “Laws” of software development – Rules that express complex ideas in very few words. Moore’s Law, for example, that computing power doubles (roughly) every eighteen months.
Or Jerry Weinberg’s advice to not attribute to malice what can be explained by incompetance.
Or Paul Jorgensen’s rule of large numbers in software development: Big Numbers times Big Numbers equals REALLY big numbers.
Now me, I’m working on a theory for software estimates, but here’s another interlude, Heusser’s Rule, which explains a lot of the … odd behavior you’ll see on software projects:
Most people really like to be able to sleep at night
This explains why people would rather be certain and wrong than uncertain and right. It explains why people hold on to bad ideas that should have been discredited long ago. It explains why people can force themselves that they will meet a deadline, despite a mountain of evidence to the contrary.
And it explains why they shoot the messenger. It’s not that you’re wrong; no, you are very much right. But by calling a spade a spade, you’ve robbed the entire room of the ability to sleep at night.
More soon.
Estimates – Interlude
The previous post was on estimates for technologist in general.
Software Testers, things are a little more … challenging. For example –
What quality will be the code be when it is delivered to test?
Most of the time, we don’t know. Vast differences in quality could make testing take a longer – or shorter – length of time.
What is the management expectation for the software when it is released?
Most of the time, these expectations aren’t quantified – if they are articulated at all.
Once identified, how quickly will the developers fix the bugs?
Unless you’ve had the same team for several projects, and the projects are similar, you probably won’t know this. If the answer is “not that fast”, then it’s probably not a testing phase – it’s a fixing phase – and it’s not bound by the testers.
For every one hundred bugs the developers fix, how many new bugs will be injected?
I’ve seen teams where this number is fifty, and I’ve seen team where this number is much less – but I can’t think of a team where this number is zero. Obviously, if this number is bigger, testing will take longer.
I could go on – the point is that we’ve got a whole lot of variables that are beyond the control of the testers. In fact, it’s probably impossible for the testers to even know what those variables are. So how can you provide an estimate?
Offhand, I know of only one career field that has to deal with a similar problem. That is — your friendly 401(k) investment advisor. I always thought that “advisor” was a bit of a misnomer, because providing advice is the one thing that guy doesn’t do. Instead, he carefully explains that he doesn’t know where social security will be in fifty years, and that he doesn’t know the value of your house, how it will appreciate, and weather you will pay it off early or take home equity loans. He doesn’t know what inflation will be, or how you investments will perform.
So he gives you a piece of paper with a big, complex formula, and tells you to figure it out for yourself. After all, he will say “It’s your money.”
Testing is an investment of time by a business owner. It is senior management’s money, yet we couldn’t get away with this, now could we?
Don’t worry. I have a few ideas and solutions for you.
More tomorrow.
Estimates – I
If I had to think of one subject that was not taught in school, and only covered in industry certifications in the most naïve way – it would be estimation.
It seems so simple. You figure out what needs to be done, figure out how to do it, break the how down into tasks, and add them up. This is pure functional decomposition, right? Easy as pie.
Sadly, In the real world, not so much. Do any of these sound familiar?
Forced Due Dates. In this case, nobody really does estimates. Or perhaps the boss does something quick, sloppy, and “aggressive.” In any event, the boss needs the project done in two weeks. Period.
‘Someone else’ does the estimates. Estimation is a skill, and it’s a skill that few junior staffers have. So having a senior staffer (or manager) do the estimates for a junior person might be helpful. The problem is that is separates the estimate from the person responsible. In other words, making someone else’s estimate is like spending someone else’s gift money – it is just detached enough from reality that you won’t be careful enough.
“Four weeks? Really? I could do it in one …” This is a manipulation technique used at the big status meeting, and I hate it. In some cases, the bully really can do it in one week, and it’s a great win for the company to switch. More often, all it takes is an offer to switch to shut the bully down. Junior people tend to buckle, promise the impossible, ‘fail’, and then look bad.
Bring me a rock estimates. “Hmm … three months? Can you shave a little time off that? … two months? We were looking for something a little more aggressive … One Month? OK, that sounds good. Remember, that is YOUR NUMBER, and we’ll hold you to that estimate!”
Slippery-Slope Requirements. Perhaps, by some miracle, you are allowed to estimate the project with your own date. Halfway through, the customer realizes that the requirements won’t meet his needs. This leaves you with two options – (A) Add the features and slip the date, making the project “late” or (B) Deny the features and deliver a product that doesn’t fit it’s purpose.
Why Estimates Are Important
In many cases, the “success” or “failure” of the project depends on weather or not we hit the date. In most of the scenarios above, that’s not a judgment of the technical contributors – it’s an evaluation of the person who made the original SWAG compared to reality.
In my next post in the series, I will discuss what we can do about it.
Something old …
I just got back from the wedding of a collegue.
In that spirit, I am reminded of the talk I gave last year at the Indiana Quality Conference – something James Bach recently refered to as a “Kick-Ass Podcast.” (No really, his words, not mine.)
It’s at the very bottom of the stack for Creative Chaos, so I thought I would let it bubble up -
The title is “So You’re Doomed” – Here’s a link to the PowerPoint (5MB)
And the Audio (45MB).
The audio is forty-five-ish minutes. I look forward to your feedback!
Test Automation – IV
Reads of Creation Chaos has left some amazing comments on the previous post; if you haven’t read them, please take a gander.
First off, I agree with Shrini that “regression” has too many definitions, and we get confused by it’s use. I think that most of the time, today, when people say “regression tests”, they mean what Shrini calls type II regression – “Make sure stuff that works yesterday still works.”
Yet, sadly, even with automated regression tests hooked up to a CI server, you still don’t get that! All you get is “The tests that passed yesterday still passed today.” Or, as I prefer to say it “Automated tests give you *some confidence* that no *big* defects were injected in the code since the last build.”
For the most past, I’ve been writing about scripted test automation. For example, If we had a GetNextDate() function, we could pick two dozen different dates and run them again and again. Of course, if something breaks on a date that is not one of those twenty-four, the automated tests won’t fix it.
That’s where model driven tests can help. For example, instead of twenty-four pre-recorded tests, the software could pick a random date between 100BC and 2500AD, then call GetNextDate(), then pop up Microsoft Excel and ask for the date plus one – then compare the results. This can work as long as you have something like excel to trust. (Cem Kaner calls this “High Volume Test Automation.”)
Another way to do it is to have a separate programmer write his or her own GetNextDate(), then pick random numbers and compare them. A few challenges with this -
1) In this case, you’re literally coding it twice. It will literally take twice as long to develop using this approach.
2) If the two developers make the same mistake (which is likely – think about leap years) the two programs will work “just fine.”
3) If the requirements are vague or wrong (how often does that happen?) the software could do what the developers expect but not what the customers want.
So here’s my conclusions …
A) Automate unit tests for simple bounds and equivalence classes
B) If you produce a single output, then a simple automation regression test is possible. (”If yesterday’s output the same as today’s?”) This will enable refactoring and diffing the two can show new functionality.
C) Documented acceptance tests prevent the “gee, I didn’t mean that, I meant this”
phenomenon and get the customers involved as part of the team. Automating those can help with communication and be a formal specification – but it might not add a lot of value in terms of finding bugs.
D) Model-Driven tests have a lot of promise for finding those quirky odd bugs, especially in the GUI. But …
E) When it comes to the “This just doesn’t look right” kind of bugs, you’ll probably want exploratory testing.
But that’s just me talkin’. What do you think?
Test Automation – III
Charlie Audritsh asked:
“I take you to mean what I’d refer to as a regression test. A test of mostly the old functionality that maybe did not change much.
So yeah. I have to admit there’s a low likelihood of finding bugs with this. What nags at me though about this idea is that I still feel like regression tests are pretty important. We want them *not* to find bugs after all. We want to know we did not break anything inadvertently, indirectly, by what we did change.”
I believe our emphasis on regression testing is mostly historical accident, and here’s why:
Big Projects (and LANs) became popular about ten years before version control became popular. So, at the beginning of my career, when dinosaurs still roamed the earth, while I was working on a bug fix, another developer might be working on new functionality. If I saved my fix before she saves her changes, then she would “step” on my changes and they would be lost.
Thus the software would “regress” – fall back to an earlier state where a bug was re-injected into the code. The longer you delayed integration, the more likely that was to happen.
Today we have version control with automatic merge, automated unit tests, and Continuous Integration Servers. So this huge tar pit of regression just isn’t as bad as it used to be. In fact, I distinctly remember reading a paper that showed that about five percent of the bugs introduced in the wild today are “true” regression bugs – bugs re-introduced by a mistake.
Of course, there’s the second type of regression, which is to make sure that everything that worked yesterday works today. I’m all for using every tool at our disposal to ensure that, but I find that automated customer acceptance tests (FITnesse with fixtures) are very expensive in terms of set-up time, yet don’t offer much value. Sure, the customer can walk by at any time and press a button and see a green light. Cool. But in terms of finding and fixing bugs?
If there is never enough time to do all the testing we would like to on projects, then I believe we are obligated to do the testing that has the most value for the effort involved.
And when it comes to bugs, I believe this is critical, investigative work done by a human. Assuming the devs have good unit tests in place, re-running tests from last week (or creating a framework so you can) probably has a lot less value than critical investigation right now, in the moment.
… but don’t get me wrong. For example, for performance testing, you need to use a tool, and type some stuff in, and then evaluate the results. I’m saying that writing code on top of that, to have a single button, then evaluate the results for you and green or redbar – might not be the best use of your time.
I am very much in favor of model driven testing, but with every release, the model needs to change to test the new functionality.
“Set it and Forget it” customer acceptance testing?
Not So Much …
