What we think of as "Agile methods" are another rediscovery of techniques that go back to the late 1960s. It's easy to despair – why haven't we learned more?! – but we developers go over the waterfall because of real and inherent pressures on projects. Here are things you can do to resist.
Software engineering, the attempt to build computer programs in a rational, repeatable way, is almost 50 years old, and during those 50 years there have been a lot of different approaches to the problem. Now, with Agile methods, we seem to have found some effective techniques: Remember that code, in itself, is cheap. Build pieces toward immediate goals, and refine them or replace them based on experience. Provide a working, demonstrable framework of a system and "grow" it to come closer and closer to a completed system. Build a little, test early and often so that mistakes are caught early and so stakeholders have a chance to see and respond, and so that the inevitable requirement changes are identified and absorbed during the process, not in a terrified rush at an acceptance test.
In other words, do what Harlan Mills proposed in his paper "Top-down Programming in Large Systems."
In fact, the basic ideas of Agile have been with us from the beginning of software development. Craig Larman and Victor Basili's paper, "Incremental and Iterative Development: A Brief History" summarizes this history.
I started this article as a rant about how software engineers and software engineering managers never learn, and how they consistently fall back toward the "waterfall model" W.W. Royce described in 1970. It would have been a fun rant to write, too.
But the truth is that software engineers – and yes, even their pointed-haired bosses – aren't stupid. The reality is that there is an intrinsic tension among opposing forces that every engineering project must solve.
There is the desire to reduce costs; opposing that desire is the need to develop a reliable product with all the features and functionality needed to make it a success.
There is the need for management to predict and plan; opposed to that is the fact that we can't predict to the smallest detail what we will need at the end of the project. (I discussed this at greater length in Weighing the Workload First).
The really confounding part, I believe, is that we software developers ship our schematics. Once we define what we're doing with sufficient detail and rigor – the point at which traditional engineers can turn the plans over to build an example – we're done. There is no fabrication step.
This conflicts in very basic ways with what other engineering disciplines expect. Toyota's stylists might be able to sell their renderings in art galleries, but you can't drive to work in one. In software, you can.
Now, honestly, other engineering disciplines have the same basic problems of requirements that change over time. Consumer demands change, and environments change, whether we're talking cars or Angry Birds games, and the company that can adapt to those changes more quickly gets real advantages. Toyota proved this with the Prius. The demand for higher-mileage cars with reduced carbon emissions became significant, and Toyota was first to the market with a successful hybrid car – and now they have two-thirds of the hybrid market.
So what are we to do? Are we really doomed to repeat the cycle of rediscovery of Agile, incremental, iterative approaches that eventually are overwhelmed by the pressure to make everything predictable and pre-planned?
Golly, I hope not. And I don't think we are. But we must be aware of those pressures and find a way to resist them or avoid them.
Most important, I think, is simply to remember that the pressures exist. It's much easier to treat a condition once you have a good diagnosis. Watch for the signs.
- When a customer insists on a full specification, remind him that the specifications will change because the requirements will change, inevitably, over time.
- When asked for a firm schedule, make sure the customers understand that providing a schedule motivates developers to exaggerate the length of the schedule to give time for changes, or that the actual functionality delivered will be some reasonable subset of what was originally asked for. And then, if they still insist on a firm schedule, do what you've promised to do and give a schedule with a lot of added time.
- If you are forced into a tight schedule, make sure you say, early and often, that the schedule is tight and there will undoubtedly be negotiation on which features are actually included. (An interesting take on this is in "Software Development: How the Traditional Contract Model Increases the Risk of Failure" by Susan Atkinson and Gabrielle Benefield.)
Next, watch out for "cargo cults." It's too easy to adopt some Agile trappings while slipping into a waterfall model. Watch for very long "iterations." If you're going months between deliveries, you're well down the slippery slope to the waterfall.
Third, build code early. In place of a months-long requirements gathering, or as part of one, propose early prototypes to demonstrate key features. Once people are comfortable with the repeated delivery of prototypes during "requirements," they will likely find it easier to accept incremental releases of the "production" code.
Fourth, and most importantly – and yes, I know I have two "most important" points, never mind, I'm on a roll – most importantly don't despair. Yes, the headlong rush down a waterfall keeps re-appearing, over and over. But being aware of it, being aware of its risks, its pitfalls, and its costs, is your sword and armor. The more Agile practices are incorporated into your work, the happier you will be, and the more effective your software development will be.