Many attempts to implement Agile fail.
When it happens, it’s ugly. People from across a company often make Agile the scapegoat, along with anyone who led the attempt at the new process.
I’ve worked at a number of companies in my years in technical and engineering management. If there’s one thing I know, it’s that when Agile fails, it’s rarely the fault of Agile itself.
Failure happens because of incomplete implementation and incomplete accommodations from the entire company perspective.
Agile in a Laboratory vs Agile in the Real World
Let me tell you the story of the first time I was part of a team that formally adopted an Agile methodology.
The company was a startup made almost entirely of MIT graduates. I’m not an MIT grad myself—my engineering degree is from UMass Amherst. But the CEO knew me and my work, and he invited me to be part of the team.
When I joined, they had decided to implement Kent Beck’s Extreme Programming (XP) model1.
As a team, they were challenged by the notion that MIT Engineers are the thinkers and not the “doers,” meaning they weren’t good at the “execution” and delivery side of software development. Being up for the challenge, they had looked at a number of different methodologies to efficiently and effectively deliver software faster and with better quality to market.
When they chose XP, it wasn’t a decision they took lightly. Once the decision was made, they were zealous about implementing Beck’s workflow to the letter.
Closed Environment vs. Open Environment
Let me say a word or two about Extreme Programming to set some context.
Beck developed XP in what I call a “closed environment.” There was little to no exposure to open-source software, interfacing to other devices, or other outside dependencies. In other words, he didn’t have the constraints we typically find in infrastructure software and data-center applications. He didn’t have to worry about behavior of other programs that were out of his control.
His system was “closed,” relying only on what the team itself could build. It’s a very effective and efficient way to develop self-contained applications. The values he promoted in his method are key, as long as you adapt to your specific environment and outside influences. That is what makes agile work in a real-world project.
Truly closed environments are rare. They might exist in places like academic research labs or within a few Skunk Works-style R&D departments within companies.
But today, most engineering teams are working with at least some exposure to outside influences such as cloud, mobile, web, third party licenses, or “you-name-it-it’s-out-there” environments. And as soon as that happens, Kent Beck’s XP workflows have to be modified.
The teams have to account for these open environments and outside factors. They’re developing their own products, but they’re also working with all kinds of other programs, none of which are under their control.
When the MIT engineers decided to adopt XP at my company, they were enamored with the velocity, the increased knowledge sharing and “to their delight” lack of documentation requirements. They overlooked the “closed” vs. “open” environment.
They expected to adopt all of XP’s workflows to the letter, and they truly expected they would be delivering a final, usable product to the end user at the end of each of their sprints.
Adaptation—The Secret to Implementing Agile in the Real World
I knew this was trouble, having delivered numerous enterprise class products to market myself.
When you’re building products in an open environment, you need a customer-like test cycle after the final sprint (or epoch) is complete, but before you deliver the final product to the user.
In my view, this is the best way to ensure the product is really ready for the customer to use in their production environments.
The Agile Underdog
When I suggested this to the MIT engineers, they didn’t like it. Not one bit!
They wanted to deliver their software to customers at the end of the sprint, the way Kent Beck explains in his book on XP. But eventually—after much discussion—I convinced them to do a six-week acceptance period so the quality engineering folks could do the Customer-like testing.
And sure enough, when we started testing the software, we found all kinds of problems.
There were over 55 “severity one” type issues. That means we found data integrity problems or data wasn’t available when it was needed for certain actions. And there were over 100 “severity two” issues, which meant some part of the program was simply broken.
Modifying Agile Is Not a Crime
Once those MIT engineers saw the value in the testing period, they were glad for the chance to fix the problems. And to do that, they used Beck’s process in the same way they did for their initial development work. They took all the problems that were found, planned out all the tasks over the next couple of sprints, and squashed them all.
When that product did go to market, it ran for over a year with only one severity two issue ever reported from a customer.
We were very proud of the result we achieved. And I gained credibility with the MIT engineers.
Agile Beyond the Engineering Teams
This idea of adjusting Agile goes far beyond making changes within the engineering teams.
In my experience, when Agile fails, it’s usually because the company didn’t make the right adjustments in the departments that surround engineering.
For example, I worked at one company where I had implemented a globally distributed scaled scrum framework (adjusted SAFE) with my team of over 300 people.
This company’s model had always been to supply customers with product development roadmaps for the next 24 months. Users of the product had come to rely on those predictions, and they planned their own internal changes based on these roadmaps.
When we looked at switching to Agile, we faced a dilemma. Agile planning principles recommend that you only plan three sprints ahead. Doing any more than that is wasteful and should be eliminated.
That’s good for the engineering team, but what about the sales teams? How were they going to explain the massive change to customers that pay their commission checks? How will the end users themselves make plans without the help of a roadmap to guide their decisions?
The question came down to this: Should we adjust Agile to accommodate a 24-month roadmap? Or should we make our customers change the way they operate just because we wanted to change to an Agile process?
In this case, we chose to adjust the Agile planning workflows to accommodate the 24-month roadmap.
We knew the roadmap predictions would not be perfect, but we felt they could be good enough to support our end users.
It was just too much to ask of our customers to change their entire process because we were switching to Agile. And besides, we knew that business changes would impact the roadmap far more often than engineering’s commitment.
Some Business Areas to Evaluate When Switching to Agile
This is Agile in the real world.
There are hard decisions for you to make, and you won’t read about them in any book on Agile. This kind of process change will affect your entire organization, including, but not limited to:
- Marketing and Sales Operations
- Supply Management
- Human Resources
You must look at it from the “net results” angle. If your engineering team is talking about switching to Agile, but you haven’t thought about how the change will affect other areas of your business, it’s unlikely your transition to Agile will be successful.
Conclusion: Agile in the Real World Requires Compromise, Trade-offs, and Creativity
Kent Beck, Ken Schwaber, Jeff Sutherland, and so many others have done a terrific job of promoting the Agile values, describing the best implementation practices, and showing why executive support is so critical to promoting an Agile development culture.
And while I have the utmost respect for these great minds who’ve led the Agile movement over the years, we cannot just implement agile within the confines of engineering.
You have to be creative. You have to adjust. You have to figure out how to implement Agile in a way that’s best for your specific company and best for your company’s interdependent processes. You have to be agile with your agile implementation or transformation.
Do so, and you’ll get all the benefits of Agile’s improvements in velocity, quality, flexibility, and above all customer satisfaction applied in the real world.
P.S. What’s been your experience with Agile transitions? Have you seen them fail because the broader organization wasn’t prepared to support the change?
Send me your stories: I can be your sounding board.
Steve Halzel is a technology executive with proven success delivering enterprise software & systems from inception to market. Over 25 years’ experience in software development, quality engineering and engineering leadership with a particular focus on cloud computing and storage virtualization. Significant Start-up, Early Stage, Fortune 500, & International experience. Expert in agile software development across multi-sites and offshore companies.
1. 2004. Extreme Programming Explained: Embrace Change, 2nd Edition. With Cynthia Andres.
Addison-Wesley. Completely rewritten. (ISBN 978-0201616415)