Most Test Automation Framework (TAF) implementations fail before they get started


You know what they say… The best way to learn is to Fail Forward…

Ok, I have a few scars then…  scars

Modernizing or implementing a new Test Automation Framework (TAF) has many of the same challenges as an Agile Transformation.  Whether you are refactoring or starting fresh, TAFs are not considered direct revenue generating.  So, they are treated as that ugly word – Process.   In other words, it gets very little attention.

Has anyone heard their CEO get jazzed up about overhead?  What does your C-Suite think about Process Improvement?  Crickets?

How many times have we heard the following?

“We have smart people, how come our competition can get features to market sooner, more often and with higher quality than us?  We’re the market leader!”

My answer: You have Great Marketing!!  Your competition has a Great Software Development process!

OK, not all C-level management folks care less about process improvement.  Most do care.  Somewhere along their career journey, they have felt the pain… If they haven’t, then good luck trying to get their already sparse attention to process improvement or modernizing of your test automation framework.


Without proper attention, due diligence will not be done.  Without due diligence, the depth of implementation details will not be flushed out.  The test automation strategy will not be fully vetted.  Staffing with the correct skills or experience is unlikely to happen.  The costs & effort will be underestimated.  The risks & contingency plans will be unknown.  The interoperability with other tools and DevOps processes will not be fully considered.  And worst of all, the actual return on the investments (ROI) cannot be calculated.

Welcome to my world

Fast forward… under funding your Test Automation Framework implementation projects could result in any or all the following:

  • Lack of implementation guidelines or rules of engagement
  • Lack of elaborated USE cases & insufficient requirements
  • Lack of integration guidelines with other tools such as:
    • Test Case management software
    • Defect management database
    • Version control software
    • Environment management & Reservation Systems
    • Trace-ability tools
  • Lack of data flow, execution flow & overall topology

Miracles and Heroism

Culture, culture, culture…  More like, “Beetlejuice, Beetlejuice, Beetlejuice”…


Without the proper executive support, only miracles and heroism can succeed. How do you recognize the culture killers before attempting a TAF modernization project?  The signs are there:

  • Unrealistic schedules
  • Management impatience
  • No metrics or data driven decisions
  • Change adverse
  • No accountability
  • The list goes on…

The tell-tale sign:  Look at the product software development life-cycle and if it’s “tribal knowledge” or “cultural enthusiasm” then your TAF project will never be better than that.

What to do… What to do…

What does every QA lead have to constantly do?  Justify everything…

A solid commercial or a modernized Test Automation Framework, coupled with a well thought out strategy and consistent object oriented guidelines will enable your company to significantly increase the speed of delivery and the quality of your software.   That’s… 


Do your homework

Show all stakeholders the ROI or value that resonates with them.  Show Engineering Management & Executives, what it will take to implement, as follows:

  • A high level project plan
  • An appropriate staffing plan with development skills
  • A well thought out strategy & architectural plan
  • The equipment requirements
  • A Cultural Change plan
  • Make the entire plan agile so you can build it in usable increments

Show Engineering Management & Execs what will happen if you do nothing or under staff, under skill, under fund or push unrealistic schedules – Disaster or maintenance nightmare.  

Show Engineering Management & Execs what they MUST DO to support the effort, such as:

  • Cultural support, “promote, promote, promote”, and allow some hard decisions to happen if necessary.

Show Engineering (Dev & QA) the benefits, the risks, what to learn, what to change & what to do (including uplifting attitudes).


 thumbs up

If you don’t get buy-in or enough critical mass, your project will fail. 

Preventing Sabotage: 

There’s always 1 or 2 very strong negative influences that will tank every project. Identify them.  If your Cultural Change plan has no effect on the saboteurs, and you cannot remove them from the bus, or change their seat, the bus will crash. Engaging them beyond their comfort zone is usually futile. You have to make more (positive) noise than they do!  Whatever you do, Do Not Ignore Them!!

“There are no secrets to success. It is the result of preparation, hard work and learning from failure.” – Colin Powell

Hire independent quality consultants

Not sure why this phenomenon exists, but many times, outsiders have more credibility.  Mostly because of their experience and they have no preconceived biases in play.  Regardless, it is worth your while to get an independent view to corroborate what you already know.   And who knows, maybe they’ll point out some things you might have overlooked.  They will provide the needed ROI to get your project funded.  And more than likely, they can help you get over the initial implementation hump and including any backlog items that need refactoring to support the new framework.

Does any of this sound familiar?

If you’ve read this far, thank you.  And I’m sorry.  You have felt the pain.   Contact me, I can help.



Dev Ops: You Can Run But You Cannot Hide… From QA

bull chasing people

In an earlier article, I argued that Quality Assurance (QA) and the revived role of Quality Engineering has largely been ignored or considered an afterthought during companies’ transitions from Waterfall to Agile.

The growing use of the term “DevOps” perfectly illustrates the industry’s continued neglect or, rather, lack of, appreciation that these Quality roles have on this development process.

Check out this definition I copied straight from website explaining DevOps:

“DevOps is the practice of operations and development engineers participating together in the entire service lifecycle, from design through the development process to production support.”

I’m sure the business view of DevOps is more about how to deliver software as fast and as efficient as possible with, of course, the highest quality at the lowest cost. And the sound of DevOpsQA or DevQAOps just doesn’t roll off your tongue quite like DevOps does. Maybe we can call it Q’devOps using a French accent to emphasize: A delicious recipe for perfection!

Regardless, here is my revised definition:

“DevOps is the practice of operations, development, and quality engineers participating together in the entire service lifecycle, from design through the development process to production support.”

What a difference one word makes!!!

DevOps Applied

I see 2 main USE cases for DevOps. I’m sure there are many more.

  1. Continuous Integration & Continuous Deployment
  2. IT Operations developing applications

Both areas are trying to improve their efficiency by integrating best practices from each other while increasing their level of agility.

For DevOps to Work, You Need a Sound Quality Engineering Strategy

Devops circles

My recommendation for DevOps is you need quality engineers embedded on your Agile teams. These professionals should be responsible for the test automated framework, and they should develop and enable development of automated test cases that are combined, integrated, version controlled and deployed alongside your software.

You also need quality assurance folks testing the pre-final deployed version prior to releasing your software to production.


As much as everyone would love to have developers and ops folks testing all their own software, generally speaking, they just don’t think the same way as someone that has the QA DNA.

They can and should create some automated tests, preferably on other team members’ software. But leave the bulk of the testing to the devious, non-trusting (in a good way) quality minded folks. I’m sure the same can be said for the ops folks crossing over the development lines and vice versa, to some degree.

Better call “Zoll”!!

(Pronounced Zaul.  Ok, I know this is a “play” off the AMC hit TV series and only my high school friends know that was my nickname, but I thought it was funny).

So if your DevOps process looks anything like this:

  • Deploy it live and wait for customer feedback.
  • Throw it “over the wall” to a separate QA team for manual testing.

over the wall

I think you could really benefit from integrating a more agile quality engineering strategy into your overall software development life-cycle process.

Here are five ingredients that I recommend you take to ensure your Agile team is testing its software in the right way:

1. Automate Test Deployment to the Test Environment

Automated tests should deploy alongside your software every time it is loaded to your test environment. The embedded quality engineer is usually the one who creates or enables the creation of these tests.

2. Run Test Automatically With Deployment

Tests should run automatically when you deploy the software. No one on your team should have to sit and manually execute a series of tests. Nor should anyone have to manually test features “to see if anything that use to work, breaks.”

3. Build or buy a Test Automation Framework

A test automation framework sets the rules for how your tests will be run. As your developers build the product, your QE staff will be building new test cases for all the new features. Over time, you’ll develop a library of regression tests to run against your software.

A good test automation framework will allow you to reserve your test environment, choose which tests or suites to run, provide error escalation methods, manage test data, log results, and provide execution visibility.

4. Make Sure Your Team is Completely Committed to Testing

Testing must be a baked-in part of what you and your team do. Include it in your definition of “done.” If it’s not, you won’t build up the automated test library you need to ensure quality software. Unless, of course, you use your customers for test purposes.

5. Test Continuously

Building automated tests directly into your deployment process means you’re going to be testing your software all the time.

There will never be a deployment that will not go through your test automated framework. Sorry for the double negatives, but I hope you get my point. You need discipline.

Adding testing to DevOps Makes Everything Work Better

Your goal shouldn’t be to only develop and release “usable software” often.   It should be to develop and release software that’s meets or exceeds customer expectations free of any defects.

Agile is not an excuse to throw away testing. On the contrary, automated testing is even more important in a high-velocity, continuous-release environment.

Your Customers really don’t care about your development process, whether you are agile or waterfall, whether your IT folks have suddenly become QA engineers, or your developers have become experts in operations, etc.

They just want your fully featured product to work—flawlessly.

What’s been your experience with DevOps and QA?

Agile: To Embed QA or Not Embed QA, That Is the RISK!


The focus of Agile developers tends to be delivering “customer-ready” software by the end of every sprint.

I’m all for that of course. Delivering usable software by the end of a sprint is one of the core differences between Agile and waterfall processes.

With that said, bringing usable software to a sprint review is quite different than delivering a complete product to the customer.

How Does QA Fit Within Agile?

Looking at the overall development process, the real question is this: where do the quality folks fit within an Agile framework?

When we think of Quality Assurance (QA), we’re used to “throw it over the wall” testing in a waterfall process. Typically, QA is a separate organization kept in the dark until it gets close to shipping time.  Then, they are in the spotlight and everyone asks, “When’s it going to be done?”

In my experience, waterfall-style QA is inefficient, has longer turnaround times or worse (and quite common), it risks releasing unfinished products just to make the dates. This happens for the same reasons that waterfall is inefficient in general:

  • It’s easy to have miscommunications when a product is handed off between teams.
  • It tends to cause unplanned rework for the development team, who have often moved on to other projects.
  • It causes teams to have longer wait times on dependencies from others.
  • It creates silos and discourages open communication between teams.

Reframing the Problem: Quality Engineering (QE)

The way to resolve these issues is to embed quality engineers on your scrum teams.

Embedded quality engineers can still report to supervisors in an official quality organization, department or community, but their daily work will happen side-by-side with developers and the other scrum team members.

The Benefits of Embedding QE on Your Scrum Teams

The job of an embedded quality engineer is quite different than traditional QA positions.

Quality engineers act as internal police officers, helping the developers stay focused on making software that meets the needs of their customers and will run well in their customers’ environments.

Quality engineers’ most important jobs are:

  1. Guide the team on feature and acceptance test strategies, and proper use of the test automation framework.
  2. Develop and maintain test automation, APIs, framework and infrastructure, which includes coaching developers to write automated test cases that fit within the pre-established guidelines.
  3. Help prevent “scope creep,” and ensure developers stay aware of limits, scale, performance, security, environmental constraints, testability, and all serviceability criteria.

The development of automated testing tools is perhaps the biggest difference between embedded QE and traditional QA roles.

For example, if part of your software includes a field where a user types his or her name, the QE team member’s job is to write a tool that will test every possible variation that might be typed into that box.

The test will need to look at American names, Japanese names, middle names, names with unusual characters or number of characters, hyphenated last names, upper/lower cases, etc.

By doing these tests within the sprints, and re-running these tests on subsequent sprints, the embedded QE team members empower everyone to prevent defects from making their way downstream, significantly reducing 1st and 2nd order defects, especially regressions.

Bonus: When the embedded QE folks are doing their job correctly, they help keep both quality and velocity high.

Embedding QE Doesn’t Mean You Completely Scrap Your Old QA Department

At some point, someone needs to run the finished product through a series of tests in a customer-like environment.

This is the traditional role of QA, and it’s a role that doesn’t go away, even in an Agile framework.

The most successful product releases, with respect to quality and timeliness, are the ones where the quality engineers are embedded on Scrum teams and a separate QA team—using a Kanban approach has thoroughly tested the product in a customer-like environment.

FIG2-TQA-020615 - Software Test Pro

A New Job Description for Embedded Quality Engineers

As you can see by now, the role of “embedded QE professional” is actually a completely new role within our companies. QE professionals may report to a QA manager, but these quality engineers need different skills than traditional QA employees to be successful.

In the past, QA employees worked in their own department, reporting to their own managers, and mostly used “manual” testing to validate software.

Embedded quality engineers must be more collaborative and work hand-in-hand with developers. They also need strong coding skills, since a major part of their job is creating tools for automated testing.

Unfortunately, many QA professionals have never developed these skills. They’ve never needed them before.

In most cases, you’ll need to train your current QA staff, or—where that’s not possible—hire new employees to fill in the gaps.

The Old QA/Developer Ratios Are Meaningless

For years, business consultants have made a living determining ratios that told us how many QA employees we needed for every developer we had on staff. Some advocated for a 2-to-1 ratio of Developers to QA professionals. Others argued for 3-to-1 or 4-to-1.

As I hope you can see, this is antiquated, waterfall thinking, and meaningless now.

It’s not about ratios anymore. It’s about keeping quality and velocity high. If your overall cost of doing business is still too high, then maybe you should look elsewhere for your cost-cutting savings, including possibly rethinking your market or business model.

The End Goal

For whatever reason, QA within Agile is a topic that doesn’t get enough attention until either something painful happens or the Monday morning quarterback shows up with the latest article on why QA teams are not needed. Good intentions, highly idealistic, and too far from reality.

You’ll need strong leadership and good metrics to make changes like the ones I’ve outlined here.

Have an story to tell about QA in an Agile environment? Send me your thoughts and stories.

The HR Role in Agile Development


When an engineering team moves to Agile, it’s a change that will impact every area of a company.

Human Resources (HR)—in particular—is an area that often goes overlooked.

If your team is considering a switch to Agile, have you thought about what that will mean for your HR department?

HR pic 1

The HR Challenge

In a waterfall organization, you have job titles such as:

  • Product manager
  • Program manager
  • Software engineer
  • Software architect
  • Software project leaders

All of that changes in Agile. Instead you’ll have titles such as:

  • Agile coach
  • Scrum master
  • Product owner
  • Developer
  • Agile coach

The team at Seapine Software used this visual on its blog, and I think it captures the difference nicely:

HR pic 2

(Image credit: Seapine Software)

As soon as you start thinking about what this means for HR, all kinds of questions emerge:

  • Can we move our current staff into the new positions? Who should we put in these new roles? Will they know what to do?
  • What new training programs will we need? Who knows enough about these topics to teach them?
  • What about career progression? Should we have “scrum master” and “senior scrum master” titles? What’s the difference between those roles?
  • What are the job descriptions for the various titles? What are other companies using?
  • Will we need to hire from the outside? Are there people in our local job market with these specific skills?
  • What’s the average compensation for these titles? Will we lose people three months from now because we’re not paying them enough?

I could go on, but you get the idea.

Switching to Agile means HR will have to throw out most of what they have on file for your engineering team.

They’ll have to start over from scratch, with a concept that might be completely unfamiliar to them.

The 5 Steps We Took with HR During Our Agile Transition

The first time I was part of an Agile transition, our HR team had never heard of Agile before we decided to make the change.

The Agile Manifesto has nothing in it to help you with a challenge like this. Neither do the books that describe how different versions of Agile are supposed to operate in practice.

Those resources focus almost entirely on how Agile works within an engineering team—not how Agile impacts outside departments such as HR.

Here are the five steps we took with our HR department (and what you should do if you’re facing the same challenge):

1. Clearly define the new roles

This is the foundation of everything you’ll do with your HR team moving forward. It’s also the basis HR will use to help you find new employees for your team when needed.

2. Establish a career progression plan for each role

You’re going to spend a lot of time and money training your people to work in an Agile framework. As you plan, you should also be thinking about how you’ll retain those people for the long term.

We found that having different levels of a role (“senior” product owner, ect.) provided a good way to help people grown within their roles.

3. Find current employees who can fill the new roles

Moving from waterfall job descriptions to Agile job descriptions will be a challenge. The skill sets for waterfall positions are different than the skills needed in most Agile positions.

You can’t just take a project manager and expect her to immediately know how to be a scrum master, for example.

You’ll need to look at every new position along with each employee’s full skill set and aptitude for the skills you need in your new Agile roles.

You’ll almost certainly find you’re missing skills for some areas. That’s where your new training programs will need to fill in the gap.

4. Use training to fill in as many gaps as you can

You’re going to have skill gaps. Some of your people won’t have any experience with Agile. You’ll need to institute training to bring everyone up to the same level of proficiency.

We used an outside consulting team to help us design our training. But we also heavily customized it to match the specific process we were going to use.

Customizing your training is critical to your success. Agile in the real world doesn’t look like the ideal laboratory version of Agile you’ve read about in textbooks. If you send everyone to a generic version of Agile training, it won’t be nearly as good as having a training customized for your specific process.

Remember that HR needs to be trained on Agile too. Don’t leave them out when you’re thinking about who to send to the classes. They are critical to evaluations, hiring, and the new career progression.

5. Hire new employees when necessary

You might find a few roles you can’t fill with internal staff. That’s the time to go looking for help. If you’ve worked with HR to clearly define your company’s new roles, they should be equipped to help you.

In the end, you’ll probably end up with a structure that looks something like this, which Scott Ambler shared on his blog:

HR pic 3

(Image credit: Scott W. Ambler)

Moving Employees from Waterfall Roles to Agile Roles: What Seems to Work

When you transition from waterfall to Agile, you’re going ask this question over and over again:

  • Does this person have the skills for this new Agile job?

This is normal, and you shouldn’t shy away from thinking hard about who’s the best fit for the roles in your new Agile team.  It is critical to have the “right” people in the “right” seats on the bus.

Every team (and every team member) is different, so I can’t tell you exactly who will be a good fit for your new Agile roles.

However, having done this a few times, here are some things to look for:

1. Program managers tend to make good scrum masters

Program managers are usually good at coordinating schedules and shepherding people through a project. We found people with these skills can often make very good scrum masters, especially if they have technical skills.

2. Software project leaders can transition to a variety of Agile roles

Software project leaders usually have a broad set of technical and social skills. They know how to talk to software engineers. They’re good at working with leadership and administration.

This broad skill set seems to help them transition to a variety of Agile roles. I’ve seen several software project leads who’ve become very effective product owners, and I’ve seen others who did just as well as scrum masters.

3. Software architects seem comfortable as product owners

Software architects are typically very good at elaborating requirements in ways that developers can understand and implement. That makes them very good candidates for product owner-type roles within Agile.

4. Product managers sometimes struggle to find a fit

Transitioning from waterfall to agile can be a challenge for current product managers. Many product managers are used to writing requirements and then handing them off to development.

In Agile, there needs to be a constant feedback loop between the end user and the development team, and some product managers struggle to adopt the “sprint” mentality that comes with Agile.

That’s not to say product managers can’t make the transition. It just means you (and your HR team) will need to help them get the training they need for whatever new position they’re going to fill.


If your engineering team is preparing to move to Agile, involve the HR team as soon as you can. You should also be prepared to answer a lot of questions as you help them learn about the Agile approach.

Send this article to your HR director today if it will help. And, reserve some time with your HR team as soon as you can get on their schedule.

In my experience, doing this early will save you lots of headaches down the road.

What About You?

Have you worked with an engineering team that changed to Agile? I’d love to hear how you worked with your HR team (and other outside departments) during the transition.

Leave a note in the comments and let us know.

Is Co-location Necessary for Agile?


Many in the Agile world see co-location as a necessary condition to work in an Agile way.

I understand where they’re coming from.

Locating your team within earshot of each other makes communication much better. It promotes trust and respect between team members. It makes the work easier to manage. In many cases it lowers operational costs too.

The problem is: sometimes it’s just not possible to have your team located in a single place.

The Hidden Costs of Agile in the Real World

When you read books about Agile, they rarely cover the changes an entire company will have to make when its engineering team adopts Agile.

Sometimes your management might give support for Agile without fully understanding the cost or the implications for the rest of the company.

Co-location is one of those costs your management team might not see coming.

A $200,000 Problem with Co-location

In one company I worked at, we ran into this problem in a big way.

We were adopting Scrum, and we had the support of our company leaders for the change. As part of that, we went to the facilities department and asked if they’d help us re-architect the space we were working in.

We asked to relocate everyone’s cubes and reconfigure the layout so we could sit together and communicate with each other easier. We asked for a Scrum room for our planning meetings and stand-ups. We asked for boards where we could hang up all our post-it notes and keep track of everything.

The facilities team looked at what we wanted to do and added up the cost of everything they’d have to change for us.

They told us the cost would easily be over $200,000.

Despite their earlier support for Agile, the company’s management didn’t support spending $200,000 to help us re-configure our work areas.

We got a few things changed, but not before making a lot of adjustments to the plan.

We commandeered a conference room that we shared among multiple teams. We weren’t able to reconfigure cubes, but we did try to move people around to be less disruptive.

We also used teleconferencing where we could.

It was good to see people visually and communicate better with teams not in our same location. But still, it wasn’t the same.

If You Can’t Be Co-located, Can You Still Be Agile?

When you implement Agile in a real-world environment, it’s going to look different than it does in all the books you’ve read.

Those books describe the ideal way to set up an Agile team. If it’s possible to set up your team exactly the way described in the books, by all means, follow their model to the letter.

Personally, I’ve never seen an Agile team that didn’t make significant adjustments to the “book version” of Agile.

Co-location is one of those things many teams simply can’t do in their companies.

That’s especially true today as more and more teams deal with remote employees or teams from different locations.

I have a colleague who uses Scrum with a globally distributed network of engineering teams, including teams in India, the United States, and Canada.

Their process doesn’t follow the book version of Scrum, but if you look at how they work, it’s still a very Agile way of doing things. And they’ve learned how to be very effective at what they do.

Can you be “Agile” even if you don’t have a co-located team?

I say the answer is yes. Unequivocally yes.

5 Tips for When Co-location Isn’t An Option

If you find yourself unable to co-locate your team, it’s not the end of the the world. Here are a few things you can do instead:

1. Commit to Finding Creative Solutions

As engineers and developers, we’re problem solvers. We face challenges every day that require creative, innovative thinking.

Step one of implementing an Agile process is to adopt a mindset that brings that same problem-solving approach to the way you work, in addition to what you’re making every day.

2. Commandeer a Conference Room

If your facilities team can’t make you a dedicated space, try to claim an existing one. If at all possible, use the meeting room that’s closest to the highest number of your teammates.

3. Use Skype, Google Hangouts, or Other Video Conferencing Applications

Technology has come a long way in recent years. If you have team members in other locations, seeing them via video will help help your team members build trust with each other.

If your company has a video-conferencing solution, use it. And if not, Skype and Google Hangouts are effective. Plus you can use them for free in most cases.

4. Consider an Online Scrum Board (And Project It on the Wall if Possible)

Some teams have found success using online programs like Trello to manage their Scrum board. There’s even a plugin that lets you automatically track burndown directly within Trello.

This can be a great solution if you have virtual team members or teams in different locations around the world.

Got a projector? You can even project your scrum board up on the wall in each of your team’s locations.

5. Daily Stand-ups Via Email

Daily stand-ups are sacred to many in the Agile community. I agree meeting in person (or at least by phone or video conference) is the best way to have a stand-up.

But maybe you have workers in different time zones or on different shifts. In some cases, a true daily standup isn’t practical.

If so, remember the point is to keep communication open between team members. If you can’t meet in person, have everyone check-in every day via email or an application like Slack.

It’s not as good as an in-person meeting, but it can still help you keep everyone on the same page and moving in the same direction.


If you want to use Agile in the real world, it’s likely you’ll have to adjust the “rules” to make them fit your specific situations.

Adjusting the Agile guidelines does not mean your team isn’t work in an Agile way. It just means you’re being creative about how you’re getting your work done every day.

And isn’t that the point of Agile in the first place?

How to Implement Agile in the Real World


Many attempts to implement Agile fail.

When it happens, it’s ugly. People from across a company often make Agile the scapegoat, along with anyone who led the attempt at the new process.

I’ve worked at a number of companies in my years in technical and engineering management. If there’s one thing I know, it’s that when Agile fails, it’s rarely the fault of Agile itself.

Failure happens because of incomplete implementation and incomplete accommodations from the entire company perspective.

Agile in a Laboratory vs Agile in the Real World

Let me tell you the story of the first time I was part of a team that formally adopted an Agile methodology.

The company was a startup made almost entirely of MIT graduates. I’m not an MIT grad myself—my engineering degree is from UMass Amherst. But the CEO knew me and my work, and he invited me to be part of the team.

When I joined, they had decided to implement Kent Beck’s Extreme Programming (XP) model1.

As a team, they were challenged by the notion that MIT Engineers are the thinkers and not the “doers,” meaning they weren’t good at the “execution” and delivery side of software development. Being up for the challenge, they had looked at a number of different methodologies to efficiently and effectively deliver software faster and with better quality to market.

When they chose XP, it wasn’t a decision they took lightly. Once the decision was made, they were zealous about implementing Beck’s workflow to the letter.

Closed Environment vs. Open Environment

Let me say a word or two about Extreme Programming to set some context.

Beck developed XP in what I call a “closed environment.” There was little to no exposure to open-source software, interfacing to other devices, or other outside dependencies. In other words, he didn’t have the constraints we typically find in infrastructure software and data-center applications. He didn’t have to worry about behavior of other programs that were out of his control.

His system was “closed,” relying only on what the team itself could build. It’s a very effective and efficient way to develop self-contained applications. The values he promoted in his method are key, as long as you adapt to your specific environment and outside influences. That is what makes agile work in a real-world project.

Truly closed environments are rare. They might exist in places like academic research labs or within a few Skunk Works-style R&D departments within companies.

But today, most engineering teams are working with at least some exposure to outside influences such as cloud, mobile, web, third party licenses, or “you-name-it-it’s-out-there” environments. And as soon as that happens, Kent Beck’s XP workflows have to be modified.

The teams have to account for these open environments and outside factors. They’re developing their own products, but they’re also working with all kinds of other programs, none of which are under their control.

When the MIT engineers decided to adopt XP at my company, they were enamored with the velocity, the increased knowledge sharing and “to their delight” lack of documentation requirements. They overlooked the “closed” vs. “open” environment.

They expected to adopt all of XP’s workflows to the letter, and they truly expected they would be delivering a final, usable product to the end user at the end of each of their sprints.

Adaptation—The Secret to Implementing Agile in the Real World

I knew this was trouble, having delivered numerous enterprise class products to market myself.

When you’re building products in an open environment, you need a customer-like test cycle after the final sprint (or epoch) is complete, but before you deliver the final product to the user.

In my view, this is the best way to ensure the product is really ready for the customer to use in their production environments.

The Agile Underdog

When I suggested this to the MIT engineers, they didn’t like it. Not one bit!

They wanted to deliver their software to customers at the end of the sprint, the way Kent Beck explains in his book on XP. But eventually—after much discussion—I convinced them to do a six-week acceptance period so the quality engineering folks could do the Customer-like testing.

And sure enough, when we started testing the software, we found all kinds of problems.

There were over 55 “severity one” type issues. That means we found data integrity problems or data wasn’t available when it was needed for certain actions. And there were over 100 “severity two” issues, which meant some part of the program was simply broken.

Modifying Agile Is Not a Crime

Once those MIT engineers saw the value in the testing period, they were glad for the chance to fix the problems. And to do that, they used Beck’s process in the same way they did for their initial development work. They took all the problems that were found, planned out all the tasks over the next couple of sprints, and squashed them all.

When that product did go to market, it ran for over a year with only one severity two issue ever reported from a customer.

We were very proud of the result we achieved. And I gained credibility with the MIT engineers.

Agile Beyond the Engineering Teams

This idea of adjusting Agile goes far beyond making changes within the engineering teams.

In my experience, when Agile fails, it’s usually because the company didn’t make the right adjustments in the departments that surround engineering.

For example, I worked at one company where I had implemented a globally distributed scaled scrum framework (adjusted SAFE) with my team of over 300 people.

This company’s model had always been to supply customers with product development roadmaps for the next 24 months. Users of the product had come to rely on those predictions, and they planned their own internal changes based on these roadmaps.

When we looked at switching to Agile, we faced a dilemma. Agile planning principles recommend that you only plan three sprints ahead. Doing any more than that is wasteful and should be eliminated.

That’s good for the engineering team, but what about the sales teams? How were they going to explain the massive change to customers that pay their commission checks? How will the end users themselves make plans without the help of a roadmap to guide their decisions?

The question came down to this: Should we adjust Agile to accommodate a 24-month roadmap? Or should we make our customers change the way they operate just because we wanted to change to an Agile process?

In this case, we chose to adjust the Agile planning workflows to accommodate the 24-month roadmap.

We knew the roadmap predictions would not be perfect, but we felt they could be good enough to support our end users.

It was just too much to ask of our customers to change their entire process because we were switching to Agile. And besides, we knew that business changes would impact the roadmap far more often than engineering’s commitment.

Some Business Areas to Evaluate When Switching to Agile

This is Agile in the real world.

There are hard decisions for you to make, and you won’t read about them in any book on Agile. This kind of process change will affect your entire organization, including, but not limited to:

  1. Marketing and Sales Operations
  2. Supply Management
  3. Human Resources
  4. Facilities
  5. Legal

You must look at it from the “net results” angle. If your engineering team is talking about switching to Agile, but you haven’t thought about how the change will affect other areas of your business, it’s unlikely your transition to Agile will be successful.

Conclusion: Agile in the Real World Requires Compromise, Trade-offs, and Creativity

Kent Beck, Ken Schwaber, Jeff Sutherland, and so many others have done a terrific job of promoting the Agile values, describing the best implementation practices, and showing why executive support is so critical to promoting an Agile development culture.

And while I have the utmost respect for these great minds who’ve led the Agile movement over the years, we cannot just implement agile within the confines of engineering.

You have to be creative. You have to adjust. You have to figure out how to implement Agile in a way that’s best for your specific company and best for your company’s interdependent processes. You have to be agile with your agile implementation or transformation.

Do so, and you’ll get all the benefits of Agile’s improvements in velocity, quality, flexibility, and above all customer satisfaction applied in the real world.

P.S. What’s been your experience with Agile transitions? Have you seen them fail because the broader organization wasn’t prepared to support the change?

Send me your stories: I can be your sounding board.

Steve Halzel is a technology executive with proven success delivering enterprise software & systems from inception to market. Over 25 years’ experience in software development, quality engineering and engineering leadership with a particular focus on cloud computing and storage virtualization. Significant Start-up, Early Stage, Fortune 500, & International experience. Expert in agile software development across multi-sites and offshore companies.

1. 2004. Extreme Programming Explained: Embrace Change, 2nd Edition. With Cynthia Andres. 
Addison-Wesley. Completely rewritten. (ISBN 978-0201616415)