Matt Ganis, Michael Ackerbauer, and Nicholas Cariello tee up our CBTJ discussion directly from where the “What will it take?” question leaves off. They look at the challenges and missteps associated with Agile, beginning with adoption, which relies on expectation setting. And there’s no expectation setting without education. Can it be that simple? Occam’s razor says, “Probably.”
There are a variety of circumstances that can cause Agile methodologies to struggle to gain a firm foothold in many organizations. We believe the most prevalent issue surrounding adoption (or lack thereof) is the misunderstanding of many Agile practices and techniques, a key component of which is the lack of understanding of the underlying reason why it is necessary to perform certain practices.
Several other problems tend to arise due to downward pressure from leadership looking for a “status” or “quicker” delivery. Agile is frequently sold to leadership as a cheaper and faster method of accomplishing tasks, when the reality is often the opposite. In our experience, an Agile practice can do more harm than good when it results in a misunderstanding, specifically around cost or delivery. This misunderstanding ultimately creates friction and, more importantly, a lack of trust between leadership and Agile teams.
In practice, we have observed that Agile teams can misunderstand some practices they have been taught, resulting in an inefficient or even incorrect use of these methods. We believe teams need to ensure they have a consistent understanding of their Agile practices in order to become more effective. This understanding across a team reinforces why a practice is structured the way it is, and even why it is considered to be part of an overarching methodology.
As teams become more consistent in how they deliver successful releases or minimum viable products (MVPs), we believe they will begin to build deeper trust with their leadership teams. Then, as the “window of trust” between development teams and leadership grows larger, teams will become more amenable to experimentation and will be more likely to innovate while continuing to hone their understanding and implementation of Agile practices. Moreover, with deeper trust, leadership will feel more comfortable allowing teams to run with their Agile process and thus be more willing to accept an incremental release of an MVP with the confidence that the “best is yet to come.”
In this article, we address an issue that we call the “circular trust problem,” where teams need to establish trust with their leadership before leadership grants them more trust. To reach that state of trust, we believe there needs to be a better understanding of Agile practices. This understanding is neither about what to do, nor about how to do it; rather, it is about truly understanding why Agile is done.
The Agile Story
Most people who have been working with Agile teams know the story of how the “Manifesto for Agile Software Development” was conceived and documented. In a nutshell, back in 2001, several seminal players in the software industry met in the US town of Snowbird, Utah, to craft a public declaration of how they believed software should be crafted. This declaration significantly altered the way many have approached software development and is the foundation for all variants of Agile software development. The ideals of the manifesto are relatively simple, and certainly by now everyone in software development has read these words:
We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on
the right, we value the items on the left more.1
These ideals are meant to shape how software delivery teams work. The manifesto was based on 12 principles that assist in creating an environment of collaborative productivity. These principles are meant to be a “guiding light” for teams as they implement and execute with agility. To provide some structure, several Agile methods (Scrum, XP, Lean, etc.) have gone on to define practices or techniques that emphasize these ideals.
Why would an organization decide to choose Agile methods over a traditional methodology? Some organizations may adopt an Agile method to help their speed to market while attempting to meet ever-changing customer demand, or perhaps they simply want to increase team productivity. To put it simply, organizations want to develop software better, faster, and cheaper.2
However, the rapid rise and commercialization of Agile methods has created what some might term a “degradation” of the original philosophy. One of the co-creators of the “Manifesto for Agile Software Development,” Arie van Bennekum, has said “[today] I see people being an Agile coach absolutely not knowing what they’re talking about,” and Jon Kern, another co-creator, has opined, “You get a lot of people, just snake-oil salesmen — folks that say they’re doing Agile when it’s Agile in name only.”3
But why would this be so? Organizations are legitimately attempting to better themselves and developers are honestly attempting to hone their craft, so why this disillusionment? Perhaps, as we believe, it is the media mantra of “Adopt Agile and deliver better software, cheaper and faster” without really understanding why some Agile methods prescribe certain steps or actions be taken. Perhaps it is because senior leadership sees Agile as a way to quickly lower bottom lines and, therefore, raise greater revenues. Perhaps it is that, in the rush to educate all our teams about Agile, we employ teachers and coaches who don’t fully understand the philosophies that underlie Agile and who pass their misunderstandings on to their students, who then struggle to realize the value and promise of Agile methods.
So the overarching question becomes: why do so many teams have issues adopting and internalizing Agile methods?4 A fair number of published papers discuss the issues teams have in moving to an Agile culture.5 Many of these works agree that three of the most important factors leading to successful adoption of Agile methods are culture, people, and communication.6 A supportive culture around a team is paramount to successful implementation of any Agile method. Developers need to be given the freedom to make decisions that will lead to frequent work releases of code and should not be doubted during a project. Along with a level of trust in the development team, Agile environments should foster rapid and frequent communication among team members, stakeholders, and senior leadership.7 These statements may appear obvious and these are things that all Agile methods must take into account. After all, don’t stand-up meetings provide for frequent and rapid communications? They do. But perhaps the type of conversations that are happening aren’t really supporting the team’s drive to deliver value at regular intervals.
For example, one of the strongest detriments to successful Agile adoption is the lack of trust (or perhaps implied lack of trust) that leadership inadvertently demonstrates to its teams. Such behavior can become a vicious circle. If a team is attempting to hone its skills while understanding the need to fail fast (i.e., make course corrections, understand their impact, and react accordingly) and leadership is second-guessing the team, or “calling it out” for being late and not delivering, the trust between leadership and the development team begins to erode. As that trust erodes, the team feels less secure in following Agile practices and more compelled just to deliver “something” to show progress. The less trust leadership imparts, the less effective the Agile team becomes. Alternately, the more leadership would/could grant the team leeway to continue becoming more effective at how it practices and gains skills with Agile methods, the more the team, knowing that leadership is showing patience and giving it time, will deliver using iterative methods and thus produce the desired results.
Another leading cause for failure to adopt comes from a misunderstanding of what teams should actually be doing. We know that in order to execute work effectively a team needs to be enabled with knowledge. In the case of adopting Agile, some teams simply lack accurate education about, and therefore knowledge of, Agile. According to a survey published in last year’s “13th Annual State of Agile Report,” 36% of teams have insufficient training and education, 35% have inconsistent processes and practices across teams, and 40% have a lack of skills or experience with Agile methods.8
Teams are clearly not obtaining the knowledge they need to appropriately implement Agile practices. Why are these teams not getting the education they need? The answer is simple: when teams are making the shift to Agile, they often do so at a very quick pace (perhaps to please their leadership), in the process creating inconsistencies and missing the underlying philosophies of Agile. Instead of the shift to Agile enabling teams to work in an Agile manner and produce better work, they are now limited in their ability to work. This is the point where teams get caught up in the “how” of work, rather than actually producing good work. By rushing straight into a development project, teams are essentially “doing the best with what they’ve got.” The lack of the pivotal component of education leads to frustration and dissatisfaction with Agile.
Leadership bears as much responsibility as teams do when it comes to misunderstanding Agile. When organizations decide to adopt Agile, they are making a commitment — to their clients, teams, and themselves — to change the way they work. However, according to the previously mentioned survey, 52% of teams still believe that there is an organizational culture at odds with Agile values, and 44% think there is inadequate leadership support and sponsorship.9 With over half of teams confirming that leadership is not playing its part in the Agile adoption that organizations are driving, we can obviously understand why that adoption is failing.
Leadership, like teams, lacks the knowledge necessary to execute Agile. While the leadership may be excited to implement this “new” way of working that is supposedly cheaper, faster, and will produce better work, it is not making the necessary commitment to abide by the practices that would allow Agile to flourish. To enable teams to work in an Agile manner, both leadership and teams need to understand Agile from the top-down.
Agile Is Faster (Well, Sort of)
Usually in a top-down decision to adopt Agile methods, a driving force behind that decision is the “promise” that Agile methods enable teams to deliver faster. But does the use of Agile methods really cause a team to deliver faster? The answer is simple: yes (and no). This brings us back to the point we raised earlier: do teams (and, more importantly, their leadership) understand why an Agile project can deliver faster?
Let’s look at the Agile process. When a team starts a project, the assumption is that the customers, or those asking for the work to be done, know they want something but are unclear exactly what that “something” is. Said another way, they have an idea, but perhaps are not ready to enumerate all the details (requirements or features) to the team. Working in two-week iterations, the Agile team commits to delivering something that works (working code), which moves the customers closer to their vision. Along the way, the customers are allowed to change the requirements, permitting the team to pivot in a different direction, or to add requirements to the queue as they gain understanding of the increased value those requirements may add. The team happily chooses the stories from its backlog with the highest value and continues working in two-week iterations.
What about speed in delivery? Based on the scenario above, work queues, called “backlogs,” will continue to grow as the customers continue to refine, until finally all requirements are met, and the final product is delivered. But that process seems far from being “quicker” than the non-Agile process. And it isn’t. An Agile project is bound to take much longer than a traditional development cycle because of all the churn Agile causes. The “speed to market” comes in having the customer see something delivered and declaring that the project, while not having fulfilled all the requirements, is “good enough,” and further development on the release ceases.
As a simplistic example, look at Figure 1. Assume a project has 100 features believed to be needed and each feature takes one week to complete. Based on a simple estimate, the project needs 100 weeks to reach completion. But with an Agile method, using iterations and frequent reviews, if at any point in the project a customer is delivered a working version of the deliverable that incorporates only 60 of those features, and the customer declares that the remaining 40 features aren’t needed to formally release it, then the project was completed in 60 weeks and, therefore, delivered faster.
So it’s not that Agile ways of working are necessarily faster; it’s that Agile provides a framework for allowing customers to decide that:
A certain set of delivered functions is sufficient or better than what they had envisioned
A certain set of not-yet-delivered features aren’t really needed
Incremental releases show tangible value that gives customers and stakeholders time to evaluate a working product. Regular releases of value heighten customer expectations for what might be next. The positive psychology of this is that an organization can potentially release enough incremental value over time that customers and leaders do not try to force-fit feature requests into a backlog that is at capacity.
“You Only Get What You Pay for”
Always on the mind of any senior manager is project cost, and rightly so. Obviously, the lure of a cheaper, faster solution, promoted by various trade journals and media hype, is also always going to pique the interest of business leaders. So is Agile really cheaper? Well again, yes (and no).
Referring again to the example in Figure 1, were we to implement all 100 features of the software in question using Agile methods, it would undoubtedly take longer and cost more than it would using a traditional development cycle. An Agile team does not focus only on development activities; it allocates time for planning (long term for releases and short term for their iterations) and for viewing the most recent build at the end of each iteration. In addition, Agile teams are continuously testing and, using various DevOps tools/techniques, are constantly deploying and debugging. These are all desirable activities — they are how we achieve a higher level of quality — but that quality comes at the “cost” of time.
How, then, is Agile cheaper? Recall that customers are able to declare the project complete when they see a working version that is “good enough” to meet their needs. By doing so, they’ve chosen to discontinue work on items now deemed unnecessary, which means that less money is spent on that project. So we might say Agile practices are more cost-effective, in that they allow customers to identify areas of improvement or to declare completion sooner than would occur with traditional development and delivery methodologies.
Communication with a Purpose
Often, teams look to Agile to help with their development process as well as the communication issues within their existing environments. After all, much of the marketing material around Agile espouses rapid communication, a characteristic that is also implicit in the various methodologies. One of the more well-known communication channels for the team is a daily stand-up.
A daily stand-up is a short organizational meeting, usually limited to 15 minutes or so. The purpose of the meeting is to allow all team members to quickly provide their current status and to understand the status of their fellow developers. When learning about stand-ups, teams learn about the “three questions” to address when they participate (“What did you do yesterday?” “What will you do today?” and “What’s blocking you?”). These questions provide a wonderful framework for the stand-up, but often team members forget the reason for holding the stand-up in the first place.
The stand-up is not meant to be a place to solve problems; its purpose is to make other members of the team aware of progress toward the iteration’s goals or potential blockers to those goals. The stand-up should provide the status of the current iteration, nothing more. Agile teams work in two-week “chunks” of time, with each chunk delivering value, in an attempt to incrementally deliver on the larger project. Any communication outside the scope of the current iteration could cause issues and a delay in delivering. Many teams find themselves simply “going through the motions” of stand-ups, providing information that isn’t potentially useful, and thereby missing out on the value of a stand-up, which is to focus on their near-term goals.
Organizational “walls of work” help track the flow of work and where there may be bottlenecks. Walls of work are a physical representation of the organization work that is currently underway and provide a visible status for all to see. The more eyes on a wall of work, determining what is going well and what isn’t, the more opportunity for organizations to consider improvements to their process.
Increased communication among team members leads to a culture of transparency. Transparency is about making shared assumptions explicit. The more transparent teams and stakeholders are with one another, the more the organization understands priorities, and the sooner critical feedback loops get closed. Teams can build trust by turning shadow projects into known work and updating leaders when circumstances change that help or hinder their delivery capability. Leaders can build trust by participating in playbacks and showcases, giving context to the customer landscape, and providing the rationale behind project prioritization decisions.
Agile Allows for Flexibility
Agility is about being responsive to marketplace changes and customer needs. While there may be apprehension when adapting to a new reality, there is also opportunity for discovery. Inspection before and after an iteration allows teams not only to see what everyone is doing but to evaluate how projects are progressing. If the process isn’t working, teams can either ask “What’s wrong?” or move on to “What is a more optimal outcome?”
Frequently, teams are told that working within an Agile framework allows them flexibility in time, budget, and scope. This is often a false promise. Use of Agile methods does not automatically grant teams the flexibility to choose the work they do, but it does allow them to adapt and improve how they work. When unplanned priority shifts take place, Agile ways of working can enable teams to more effectively embrace the change and feedback they receive from stakeholders.
The biggest challenge for leaders moving to an Agile environment is throttling the funnel of work that is currently in progress. Effective governance means limiting the number of projects and reprioritizing them so that the teams can focus on one project at a time. This ensures teams can deliver more value sooner, from one iteration or sprint to the next. Although this may seem like an oxymoron, it is true. The secret sauce is allowing teams to be able to focus.
The leader’s job is to ensure teams are doing the right work, while it is the teams’ job to do the work right. Once a team is empowered to make process decisions, it will do the work right and find ways to improve. As teams develop confidence in their throughput capability, leaders grow confident in being able to manage the funnel of work to focus on a customer’s most essential priorities.
A popular YouTube video shows the evolution of Formula 1 racing over a 60-year span of time.10 The dramatic shift from 1960s’ pitstops taking well over a minute to under three seconds in 2013 highlights the essence of continuous improvement. In an era of hypercompetition, such improvement may involve changing or redefining rules. Making small changes over time reduces team resistance and simultaneously builds organizational resilience. We believe each of these incremental outcomes enlarges the window of trust between development teams and leaders.
We began this article with the opinion that for teams to truly realize the value of an Agile methodology, they need not blindly follow a prescriptive set of steps but rather understand why those steps are useful and how they affect an outcome. While there are many teams that have internalized these Agile values and practices, we believe there are many more that have not. We don’t mean to imply that all those operating with Agile methods are mere lemmings following a specific method, “word for word,” but we do believe there are teams that just don’t understand the principles undergirding the practices.
Perhaps we need a better educational model. As we stated earlier, roughly a third of teams practicing Agile cite issues with education or the adherence to a common set of practices within their organization. (We are hopeful that perhaps the more persistent teams will be able to stay the course and realize the value of these Agile methods.) And 40% of teams attribute their issues with Agile adoption as stemming from lack of experience. This calls to mind something Mahatma Gandhi once said: “Knowledge gained through experience is far superior and many times more useful than bookish knowledge.”
The potential for success with Agile exists. Teams know it, stakeholders know it, leaders know it. The easily removed “blockers,” such as miscommunication, speed of execution, and adoption failure, directly contribute to why success is out of reach for some teams. Resolution is possible only when there is trust and communication among all those involved. Agile methodology at its very inception was designed to focus on the things that really matter. If organizations rally behind this idea — to focus on what really matters — projects following Agile will find success.
1Beck, Kent, et al. “Manifesto for Agile Software Development.” Agilemanifesto.org, 2001.
2McDonald, Kent. “Agile Q&A: Why Do Organizations Adopt Agile?” Agile Alliance, 2020.
3Nyce, Caroline Mimbs. “The Winter Getaway That Turned the Software World Upside Down.” The Atlantic, 8 December 2017.
4McAvoy, John, and Tom Butler. “A Failure to Learn in a Software Development Team: The Unsuccessful Introduction of an Agile Method.” In Information Systems Development, edited by W. Gregory Wojtkowski, et al. Springer, 14 September 2008.
5Tanner, Maureen, and Ulrich von Willingh. “Factors Leading to the Success and Failure of Agile Projects Implemented in Traditionally Waterfall Environments.” Proceedings from the Management, Knowledge, and Learning International Conference (Human Capital Without Borders: Knowledge and Learning for Quality of Life), Portoroz, Slovenia, 24-27 June 2014.
6Samia, Abdalhamid, and Alok Mishra. “Factors in Agile Methods Adoption.” TEM Journal, Vol. 6, No. 2, May 2017.
7Basili, Victor R., et al. “Empirical Findings in Agile Methods.” Proceedings of Extreme Programming and Agile Methods — XP/Agile Universe, Springer 2002.
8“13th Annual State of Agile Report.” CollabNet VersionOne, 7 May 2019.
9“13th Annual State of Agile Report” (see 8).
10CpatainCanuck. “Formula 1 Pit Stops 1950 & Today.” YouTube, 2019.