The Air Force’s $1 Billion Software Flop [Part 1]

Lorinda Brandon

December 18, 2012

When I read the story the other day about the Air Force scrapping a project to overhaul its legacy software system and replace it with a more modern system, I felt myself spinning uncontrollably in the “Way Back Machine.” It took me back to the early 1980’s, when I worked as a civilian in one of the country’s Air Logistics Centers.

My job was to manage the production activities for a series of maintenance patches/upgrades, first for the F-111 aircraft then for the A-10 aircraft. Each tail number was tracked with the date it came in, the work to be done, who did it, who flew it in, when it left, who flew it out, etc. You’re probably imagining a complex rack of servers stored in a temperature-controlled data center. Well, I hate to be the one to burst your bubble … but it was really a drab line of tan metal desks, not one of which had a computer on it, and a line of people like me scribbling in pads and keeping brown folders with handwritten notes to track each aircraft modification to each tail number.

When the work was complete, we gave our folder to Pearl. Pearl was the Oracle of our office, with her enormous cross-tabbed ledger, written in pencil, which consolidated all of the information about all of the work done for all of our nation’s A-10 aircraft. It’s impossible to imagine that workflow in this day and age, but it’s true.

Embracing Change – or At Least Offering it a Cup of Coffee

Of course, that’s ancient history. Eventually, my job evolved to work with the new scary computer lab to design the database and reports that our office was doing by hand. With the notable exception of my rather brave manager and me, nobody in my office welcomed this development, especially Pearl. One data entry clerk could replace the whole office. I was fresh out of college so I was rather breathless with the technology and my role in bringing tomorrow into today. I ignored the glares and grumbling from my colleagues as I sat before the dumb terminal, messaging with the lab about today’s bugs. We had no requirements tracking other than my typed list (typed and printed on a Wang system by the secretarial pool), and there was no bug tracking system. Instead, I would send terminal messages to the developers or walk down to their lab with my handwritten bug list. I know how laughable it all sounds now, but in retrospect we were changing the world, one terminal message at a time.

I distinctly remember the day Pearl was silenced. In those days, when one of our tail numbers crashed, we had 2 minutes to send a report to the Pentagon about all the maintenance our facility had performed on it. After weeks of working on this one report and all the underlying data behind it, we decided to test our new computer system by racing with Pearl to deliver the report – I don’t have to tell you who won. I pressed a button and had the printout in my hand while Pearl was still flipping through her ledger.

But it was a short-lived triumph. While we proved a point, we didn’t effect change. My colleagues continued to ignore the terminal and me, other than to mutter resentful comments about both of us. My manager and I were excited about the forward progress but we didn’t do the right things to get the full commitment of the rest of the team. If we had, we could have moved faster and designed something better.

This is partially what drove the Air Force’s Expeditionary Combat Support System (ECSS) to its knees after six years, $1 billion and a partial roll-out. The idea of creating an end-to-end logistics system that allowed for the kind of oversight and reporting required by the Defense Department is ambitious and worth of respect. It’s the kind of thing that makes you hold your head in your hands with the sheer magnitude of it. Here are three tips for orchestrating change in your organization and avoiding the type of deflating disaster the Air Force just faced.

1.Understand the Personal Impact

Effecting change is about communicating the vision in a way that makes sense to the people you are impacting, and giving them a meaningful way to contribute. And that means first understanding what they do, how they do it, and what they like about it. People will always adopt a change that mitigates the part of their jobs they dislike. But their fear is that they will also lose what they like or what they’re best at. Find out what that is before you design the new system and process. Don’t lose what works in the process of fixing what’s broken.

I wanted to free Pearl from her ledger but didn’t understand what she needed or wanted. Neither my manager nor I stopped to evaluate what was good about her process (one gatekeeper to ensure data integrity, a go-to person to serve as a point of contact for important data requests). One of the main reasons cited for the current disaster was that the change forced organization, process and tools changes all at once across hundreds of thousands of employees. It doesn’t take an aerospace scientist to predict failure if all of those employees weren’t invested in the change.

2.Understand the Requirements, One at a Time

The Air Force readily admits that this project was originally intended to replace the whole system at once, which means that they invested so much time in the definition, selection, and implementation of the replacement system that it was probably obsolete before the first line of code was written.

Big projects are usually a result of big expenditures. Companies usually replace their infrastructure and systems for cost reasons. It’s the old car syndrome – at some point it will cost more to maintain than the car is worth and it’s cheaper to just buy a new one. But outside of most board rooms, discussions about “bottom line” don’t resonate with workers.

So how do you figure out what you need to build and in what order? User stories are a great vehicle for understanding what you need to build because they start with the most important stakeholder of all… the user. Who is the user and what do they need to do? Stating both in each user story will help you create a vivid picture of what each person who uses the system has to do and it does it in such a way that you can group them into portable pieces to start building. “As the Crash Reporter, I need to list all modifications for each tail number in date order.” Cool. Then walk that dependency tree backwards – who enters the tail numbers into the system and what do they need? Who enters the modifications and what do they need? Right, you get the point. It’s about finite statements that express a user’s day-to-day job activities.

3.Communicate, Communicate, Communicate

communication Another point of regret mentioned by people within the ECSS project is the sheer number of meetings. It’s not a surprise to anyone that too many meetings can make a project go over budget and past schedule, is it? So how to avoid it? You want to build the right system, so you need to make sure you’ve heard from all the stakeholders and continue to track and report on the status of the project.

Don’t try to include everyone. Find your key stakeholders and keep that group small. Very small. Use them to understand the business needs and the day-to-day needs of the people on the ground who will use your system. As you design, keep checking back – “did we get that right?” “Did we miss something?”

Then find your influencers. They have to be people the stakeholders trust, but more importantly they are the people others naturally listen to. They won’t fit a mold. They aren’t all managers and they aren’t all young guns. But people know who they are… and you need to find them. If you can tap into their passion, get them to understand the vision and believe in the benefit of what you are doing, you can’t fail.

But back to the poor Air Force…

So, here I sit with one foot in my Way Back Machine, reading today’s story about the next generation of software intended to replace the one I worked on. As that breathless young woman triumphantly waving the Crash Report over her head, I’m sad it was such a failure, even though it was designed to replace the system I helped to build. As a taxpayer, I’m frustrated by the wasted expenditure. And as someone who has made much of my career out of leading organizations through change, I’m baffled. We’ve got so many tools at our fingertips, from process to people to software, all of which are designed for this very scenario. How could this fail?

See also: