Advisor

Why Does Agile Fail?

Posted July 23, 2020 in Business Agility & Software Engineering Excellence
Why Does Agile Fail?

There are a variety of circumstances that can cause Agile methodologies to struggle to gain a firm foot­hold in many organizations. We believe the most prevalent issue surrounding adoption (or lack thereof) is the misunderstanding of many Agile practices and techniques, a key component of which is the lack of understanding of the underlying reason why it is necessary to perform certain practices.

Several other problems tend to arise due to downward pressure from leadership looking for a “status” or “quicker” delivery. Agile is frequently sold to leadership as a cheaper and faster method of accomplishing tasks, when the reality is often the opposite. In our experience, an Agile practice can do more harm than good when it results in a misunderstanding, specifi­cally around cost or delivery. This misunderstanding ultimately creates friction and, more importantly, a lack of trust between leadership and Agile teams.

In practice, we have observed that Agile teams can misunderstand some practices they have been taught, resulting in an inefficient or even incorrect use of these methods. We believe teams need to ensure they have a consistent understanding of their Agile practices in order to become more effective. This understanding across a team reinforces why a practice is structured the way it is, and even why it is considered to be part of an overarching methodology.

As teams become more consistent in how they deliver successful releases or minimum viable products (MVPs), we believe they will begin to build deeper trust with their leadership teams. Then, as the “window of trust” between development teams and leadership grows larger, teams will become more amenable to experimentation and will be more likely to innovate while continuing to hone their understanding and implementation of Agile practices. Moreover, with deeper trust, leadership will feel more comfortable allowing teams to run with their Agile process and thus be more willing to accept an incremental release of an MVP with the confidence that the “best is yet to come.”

In this Advisor, we address an issue that we call the “circular trust problem,” where teams need to estab­lish trust with their leadership before leadership grants them more trust. To reach that state of trust, we believe there needs to be a better understanding of Agile practices. This understanding is neither about what to do, nor about how to do it; rather, it is about truly understanding why Agile is done.

The Agile Story

Most people who have been working with Agile teams know the story of how the “Manifesto for Agile Soft­ware Development” was con­ceived and documented. In a nutshell, back in 2001, several seminal players in the software industry met in the US town of Snowbird, Utah, to craft a public declaration of how they believed software should be crafted. This declaration signifi­cantly altered the way many have approached soft­ware development and is the foundation for all vari­ants of Agile software development. The ideals of the manifesto are meant to shape how software delivery teams work. The manifesto was based on 12 principles that assist in creating an environment of collaborative productivity. These principles are meant to be a “guid­ing light” for teams as they implement and execute with agility. To provide some structure, several Agile methods (Scrum, XP, Lean, etc.) have gone on to define practices or techniques that emphasize these ideals.

Why would an organization decide to choose Agile methods over a traditional methodology? Some organizations may adopt an Agile method to help their speed to market while attempting to meet ever-changing customer demand, or perhaps they simply want to increase team productivity. To put it simply, organizations want to develop software better, faster, and cheaper.

However, the rapid rise and commercialization of Agile methods has created what some might term a “degradation” of the original philosophy. One of the co-creators of the “Manifesto for Agile Software Development,” Arie van Bennekum, has said “[today] I see people being an Agile coach absolutely not knowing what they’re talking about,” and Jon Kern, another co-creator, has opined, “You get a lot of people, just snake-oil salesmen — folks that say they’re doing Agile when it’s Agile in name only.”

But why would this be so? Organizations are legiti­mately attempting to better themselves and developers are honestly attempting to hone their craft, so why this disillusionment? Perhaps, as we believe, it is the media mantra of “Adopt Agile and deliver better software, cheaper and faster” without really understanding why some Agile methods prescribe certain steps or actions be taken. Perhaps it is because senior leadership sees Agile as a way to quickly lower bottom lines and, therefore, raise greater revenues. Perhaps it is that, in the rush to educate all our teams about Agile, we employ teachers and coaches who don’t fully understand the philosophies that underlie Agile and who pass their misunderstandings on to their students, who then struggle to realize the value and promise of Agile methods.

Adoption Failures

So the overarching question becomes: why do so many teams have issues adopting and intern­alizing Agile methods? A fair number of published papers discuss the issues teams have in moving to an Agile culture. Many of these works agree that three of the most important factors leading to successful adoption of Agile methods are culture, people, and communication. A supportive culture around a team is paramount to successful implementation of any Agile method. Developers need to be given the freedom to make decisions that will lead to frequent work releases of code and should not be doubted during a project. Along with a level of trust in the development team, Agile environments should foster rapid and frequent communication among team members, stakeholders, and senior leadership. These statements may appear obvious and these are things that all Agile methods must take into account. After all, don’t stand-up meetings provide for frequent and rapid communications? They do. But perhaps the type of conversations that are happening aren’t really supporting the team’s drive to deliver value at regular intervals.

For example, one of the strongest detriments to suc­cessful Agile adoption is the lack of trust (or perhaps implied lack of trust) that leadership inadvertently demonstrates to its teams. Such behavior can become a vicious circle. If a team is attempting to hone its skills while understanding the need to fail fast (i.e., make course corrections, understand their impact, and react accordingly) and leadership is second-guessing the team, or “calling it out” for being late and not delivering, the trust between leadership and the development team begins to erode. As that trust erodes, the team feels less secure in following Agile practices and more compelled just to deliver “something” to show progress. The less trust leadership imparts, the less effective the Agile team becomes. Alternately, the more leadership would/could grant the team leeway to continue becoming more effective at how it practices and gains skills with Agile methods, the more the team, knowing that leadership is showing patience and giving it time, will deliver using iterative methods and thus produce the desired results.

Another leading cause for failure to adopt comes from a misunderstanding of what teams should actually be doing. We know that in order to execute work effectively a team needs to be enabled with knowledge. In the case of adopting Agile, some teams simply lack accurate education about, and therefore knowledge of, Agile. According to a survey published in last year’s “13th Annual State of Agile Report,” 36% of teams have insufficient training and education, 35% have inconsistent processes and practices across teams, and 40% have a lack of skills or experience with Agile methods.

Teams are clearly not obtaining the knowledge they need to appropriately implement Agile practices. Why are these teams not getting the education they need? The answer is simple: when teams are making the shift to Agile, they often do so at a very quick pace (perhaps to please their leadership), in the process creating inconsistencies and missing the underlying philosophies of Agile. Instead of the shift to Agile enabling teams to work in an Agile manner and produce better work, they are now limited in their ability to work. This is the point where teams get caught up in the “how” of work, rather than actually producing good work. By rushing straight into a development project, teams are essentially “doing the best with what they’ve got.” The lack of the pivotal component of education leads to frustration and dissatisfaction with Agile.

Leadership bears as much responsibility as teams do when it comes to misunderstanding Agile. When organizations decide to adopt Agile, they are making a commitment — to their clients, teams, and themselves — to change the way they work. However, according to the previously mentioned survey, 52% of teams still believe that there is an organizational culture at odds with Agile values, and 44% think there is inadequate leadership support and sponsorship. With over half of teams confirming that leadership is not playing its part in the Agile adoption that organizations are driving, we can obviously understand why that adoption is failing.

Leadership, like teams, lacks the knowledge necessary to execute Agile. While the leadership may be excited to implement this “new” way of working that is supposedly cheaper, faster, and will produce better work, it is not making the necessary commitment to abide by the practices that would allow Agile to flourish. To enable teams to work in an Agile manner, both leadership and teams need to understand Agile from the top-down.

[For more from the authors on this topic, see “The Speed of Trust: Why Some Agile Teams Succeed and Others Do Not.”]

About The Author
Matt Ganis
Matthew Ganis is an IBM Senior Technical Staff Member in Armonk, NY. He is also an Adjunct Professor of Information Technology and Astronomy at Pace University, where he focuses on the implementation of the Internet of Things. Dr. Ganis is the coauthor of Social Media Analytics: Techniques and Insights for Extracting Business Value Out of Social Media. He can be reached at mganis@pace.edu or via Twitter @mattganis. 
Michael Ackerbauer
Michael Ackerbauer is an IBM Leadership and Innovation Consultant and ICAgile-certified coach and facilitator in Armonk, NY. He is also an Adjunct Professor of Information Systems at Marist College, where he focuses on strategy and policymaking, leadership, and creative problem solving. Dr. Ackerbauer earned a PhD in creative leadership for innovation and change. He can be reached at Michael.Ackerbauer@marist.edu or via Twitter @macker.
Nicholas Cariello
Nicholas Cariello is an IBM Cybersecurity Engineer and Software Developer in Armonk, New York. He earned a bachelor’s degree and master’s degree in computer science, with concentrations in cybersecurity, from New York Tech. In his free time, he is a community theater Production Director. He can be reached at nicholas.cariello@ibm.com.