How Agile Are Organizations Today?
by Jim Highsmith, Director, Cutter Consortium's Agile Product and Project Management Practice; and Dr. Robert K. Wysocki, Senior Consultant, Cutter Consortium
OVERVIEW
The agile movement is now more than five years old, measured from the authoring of the Agile Manifesto. In this time frame, many organizations have implemented agile methods, with many more planning agile transitions. Previous Cutter (and other) surveys have addressed questions about how organizations are using agile methods, what particular flavor of agile is being used, or whether agile methods result in higher-quality software, but we thought it was time to ask a different question: how agile are organizations today? There are many organizations that profess to be agile, but when we dig down into the details, we find that they are not using very many of the practices, or that only a few teams in the organization are agile, or that project teams are agile but management practices have yet to change. So we wanted to test these aspects of agility.
It is difficult in a short survey to get a complete picture of whether an organization is agile, but the level of implementation of certain practices can provide a good indication. Thus, we asked respondents to think about practices across their entire organization, not just for a project team or two. Instructions for the survey were as follows:
This survey assesses organizations' software development process and organizational agility. We do not assume your organization has implemented a specific agile methodology; we are looking for your use of general traits or practices that are "agile." For example, an organization may use very collaborative practices (which are characteristic of an agile organization) without using the specific practices of a particular agile approach. The focus of your answers should be the entire software/application environment and not just a single project. There may be a very agile team in your organization, but what is the "average" across all teams?
Each statement should be evaluated on a scale of 1-5: 1 -- strongly disagree (few teams operate this way); 2 -- somewhat disagree; 3 -- neither disagree nor agree; 4 -- somewhat agree; 5 -- strongly agree (nearly all teams operate like this).
The five topic areas we included in the survey are: (1) customer involvement and collaboration; (2) software development process; (3) quality and testing; (4) management of development; and (5) feedback and learning. We did not include a section on agile principles due to the subjectivity of the topic; instead, we structured the practice questions in a way that made them good indicators of how principles had been implemented.
What we found in the survey was actually surprising -- as we discuss in the conclusion (and other portions of the report). We think the results of this survey will provide interesting information on how a wide cross-section of organizations are progressing in their agile implementations.
SURVEY PROFILE
The survey respondents represent a diverse set of industry, job, and geographic distribution. The makeup of survey respondents includes a wide range of organizations: software companies (24%); consulting companies (24%); financial institutions (10%); government agencies (10%); manufacturing firms (4%); colleges and universities (5%); publishing/media firms (4%); healthcare (4%); and other industries.
IT organization size ranges from very small (fewer than 50 employees) to very large (more than 1,000). In addition, 45% of respondents have fewer than 50 IT professionals, while 38% have between 50 and 500, and 17% have more than 500 IT professionals. Annual company revenues exceed US $1 billion for 20% of the respondents.
Respondents come from a wide geographic distribution: 49% from North America; 28% from Europe; 11% from Australia/Pacific; 4% from South America; and 7% from Asia/India.
The respondents are also predominately from management ranks: 37% from IT or senior management; 15% from project management; 15% from consulting; 14% from software engineering and programming; among others.
The size of the respondent organizations (55% with more than 50 professionals) and the high level of respondents in management ranks are good indicators of the growing interest in agile methods in organizations.
CUSTOMER INVOLVEMENT AND COLLABORATION
Customer involvement and collaboration among all team members are key indicators of an agile environment. Customer involvement should be of high intensity and frequent (almost daily). There are potentially two types of customers, internal and external, depending upon whether the development group is an internal IT one or is developing products for the market. Internal customers may be the end user, in the case of IT development, or a proxy user, in the case where a product manager/team translates external customer requirements. In the latter case, external customers should be brought into the development process periodically (depending upon factors such as protecting IP) but are probably not available daily.
As the data in Figure 1 indicates, the involvement of internal customers from project chartering in the beginning of the project to end-of-iteration reviews is high (strongly or somewhat agree) in 71% of the responses. In Figure 2, dealing with external customers, the high involvement decreases somewhat to 56%. These appear to be a very positive response and show that the agile emphasis on customer involvement has had a very positive impact on organizations using agile methods.
Figure 1 -- Internal customers, including the product manager, are active participants in the project team for project chartering, iteration planning, end-of-iteration reviews, and on a daily or very frequent basis.
Figure 2 -- External customers are brought into the development process on a regular basis, including to provide feedback on iterative versions of the product.
The next survey questions address the agile issues of cross-functional teams and collaboration. The first and second questions address project staff composition, and the next two questions address collaboration. Figure 3 asks whether teams are composed of cross-functional skill sets -- development, product management, QA, and so forth; 76% of respondents agree with the statement (strongly and somewhat). However, it's easier to create a cross-functional team than it is to get individuals working together. Hence, the second question addresses one of the critical team maturity issues: do team members have a common commitment to the team's goals? In Figure 4, the agreement to the question of common goals is 67%.
Figure 3 -- Project teams are composed of cross-functional disciplines (customers, developers, architects, testers, etc.) required to deliver the product.
Figure 4 -- Teams have a collective commitment to the end product of their efforts rather than just a commitment to getting their own tasks accomplished.
Since a significant percentage of respondents are from management ranks, we hypothesize that the response to the question illustrated in Figure 3 is overly influenced by this fact, but when we look at responses by engineers, 71% agree! It appears that organizations employing agile methods have a much higher degree of interaction and team cohesion than non-agile teams.
The next two survey questions (not illustrated here) address the issue of collaboration among the cross-functional team members. In other words, they look at whether these teams are just a collection of individuals or are "jelled" teams. The two statements we made about these issues were: (1) "Distributed project teams have adequate collaboration tools that enable frequent interaction during the day"; and (2) "Teams conduct frequent, short coordination meetings (i.e., daily standup meetings)."
The first of these questions addresses the most difficult issue for some companies -- distributed teams -- and whether or not the teams are adequately supported with tools. If distributed teams are not supported by tools, collaboration probably suffers. Some 40% of the respondents agree that they have supportive tools. The most basic agile tool for collaboration is the daily standup meeting. Whether a team is using XP, Scrum, Adaptive Development, or other agile approaches, the daily standup is a key practice. We are very pleased to see that 62% of respondents appear to use daily standup meetings.
We also analyzed the data in other dimensions. As might be expected, in general, size tends to adversely affect software development agility. This is especially noticed when size is measured by number of employees, number of IT employees, and by IT budget.
Other specific conclusions we ascertain from the data:
-
Internal and external active involvement does not differ between technology-based companies and non-technology-based companies.
-
Technology-based companies utilize project teams that are more cross-functional than do non-technology-based companies.
-
Technology-based companies have better collaboration tools than do non-technology-based companies. This is not an unexpected finding.
Company location has an effect on both internal and external customer involvement. Ordered by decreasing customer involvement, the surveyed company locations are Australia/Pacific Rim, Europe, and finally North America. This ordering suggests that North American companies have lessons to learn about customer involvement from both Europe and Australia.
SOFTWARE DEVELOPMENT PROCESS
The software development process, from iteration planning to technical practices such as refactoring, generates much of the appeal of agile methods within the technical community. For example, XP, one of the earliest and best-known of the agile methodologies, had (and still has) great appeal within the technical community.
The survey questions in this area cover a range of project management and technical practices, such as iterative project plans; agile technical practices, such as simple design, refactoring, test-driven development, coding standards, continuous integration, and evolutionary design; lean documentation; shippable software; and risk identification and mitigation. Specific questions about quality and testing are covered in the next section.
The responses to the question in Figure 5 dealing with iterative project plans are somewhat perplexing in that iterative development is probably the most fundamental practice in agile methods; yet only 56% of respondents somewhat or strongly agree that they use short iterations and feature/story-driven planning. Nearly 40% of all respondents are from organizations of over 100 IT professionals, and their 47% "agree" response is nearly 10 percentage points lower than the average. This indicates that, in large organizations, there is definitely a mix of teams using and not using iterative development.
Figure 5 -- Project plans (release plans) are iterative (short, one month or less, timeboxed) and consist of allocating user-facing chunks of functionality (features, stories) to these timeboxes.
A second significant technical indicator of "agility" is whether or not the focal point for iterations is producing shippable software. Shippability has a number of ramifications, from project management to the organization's commitment to automated testing. Some 50% of respondents agree that they strive to deliver shippable software, and this is the primary indicator of progress (see Figure 6). This response seems very high in comparison with the previous question in which only 56% agree about iterative plans in general. We have worked with a number of companies that used iterative development but were far from shippable software at the end of each iteration -- usually related to a lack of automated testing. Based on our experience, we would guess that the iterative planning responses are a little low, and the shippable software responses are a little high (in the agree responses) for some reason.
Figure 6 -- Project teams strive to deliver shippable software by the end of each timeboxed iteration, and this shippable software is the primary measure of progress.
Somewhat unexpected is the small difference in responses between development engineers and other respondents. For example, there is a lack of significant difference between management and engineering responses in the perceived use of agile technical practices: among all respondents, 43% indicate they either somewhat or strongly agree that teams fully utilize them (see Figure 7), whereas 35% of the software engineers/programmers feel similarly. We would have expected a greater difference in this area between management and engineering. Since a number of important technical practices were included in this question, the agree levels are encouraging (for example, one respondent's comment indicates that they utilize most of the technical practices except test-driven development). Further study would be necessary to see the levels of usage for individual practices.
Figure 7 -- Development teams fully utilize agile technical practices: simple design, refactoring, test-driven development, coding standards, continuous integration, evolutionary design.
Conversely, in the response to utilizing short, iterative project plans, all respondents agree at a 56% level, whereas software engineers agree at a 64% level. We expected the opposite response -- more engineers disagreeing -- but this may indicate more iterative development projects being undertaken than management realizes.
There is a popular misconception that agile practices discourage, or even eliminate, documentation. In reality, the agile principles advocate "lean" documentation -- that is, just enough for the purpose at hand (and maintenance). This concept of "just enough" documentation seems to be understood by respondents, as 54% agree with the statement (see Figure 8). The misconceptions about documentation usually surface from people who are evaluating agile development or have not practiced it very long. Those with experience seem to be adjusting in a practical way.
Figure 8 -- Documentation from key activities (architecture, plans, system designs, user-interface designs, etc.) utilize lean ("just enough") principles and evolve over the life of the project.
Finally, risk management is an important part of any software development effort. Agile development, by its very nature of using short iterations, helps reduce certain project risks, but there is still a need to employ risk analysis and mitigation practices within an agile project. Some 58% of our respondents agree that ongoing risk management is an integral part of their development process (see Figure 9).
Figure 9 -- Risk identification and mitigation are ongoing and integral to the development process.
In the profile of organizations taking this survey, nearly 45% are software publishers or consultants. We broke out this group to see whether they are more or less agile than the others, which are internal IT organizations. Not surprisingly, the software publishers and consultants are significantly more agile. In iterative project planning, this group agrees at a 63% level, while others are at 51%; as for shippable software, the ratio is 59% to 43%. Technical practices and lean documentation follow the same pattern, as do responses in other areas. This data tracks with our experience that software product companies and consulting firms embrace agile methods earlier than internal IT groups in industry and government. However, the level of response to these questions shows that while IT groups are lagging in their agile implementations, there are still significant efforts underway.
QUALITY AND TESTING
Delivering high-quality software that is shippable every iteration is key to a good agile process. We tried to address this area by focusing on three key areas: (1) Is quality important to the organization?; (2) Is testing automated?; and (3) Is comprehensive testing an integral part of every iteration? Of course, the answer to the first question is always yes, so we added another piece to see how important: are there quality metrics in place? The second question indicates whether or not developers are actively doing unit testing (which shows organizational importance is placed on testing) and whether or not that testing is automated. The third question addresses comprehensiveness of testing each iteration -- for example, an organization that does unit testing for each iteration but waits to do integration testing is not doing comprehensive testing for each iteration -- as well as the degree of automation.
In Figure 10, 57% of respondents agree that there is a significant emphasis on high quality and that quality metrics are being used. And while this is an encouraging number, less encouraging is the fact that only 35% agree that developers are doing high levels of automated unit testing (see Figure 11) and 42% agree that comprehensive automated testing is a part of each iteration (see Figure 12). These numbers seem to indicate that even though the first question is qualified with a metrics component, more people may still be espousing high quality while not stepping up with sometimes difficult practices.
Figure 10 -- There is an organizational emphasis on high-quality development, including the use of appropriate quality measures.
Figure 11 -- Developers spend a significant amount of their time writing automated unit tests.
Figure 12 -- Automated QA testing (regression, system, integration) and automated customer acceptance testing are an integral part of every timeboxed iteration.
In this category, there is a significant difference between management and engineering responses -- not unexpected. To the question about the organization's emphasis on high-quality development, 24% of all respondents strongly or somewhat disagree with the premise, while 39% of the engineers disagree. To the question of developers spending a high percentage of their time writing automated unit tests, 35% of all respondents strongly or somewhat disagree, while 41% of software engineers feel similarly. Finally, to the question of comprehensive automated testing for each iteration, overall, respondents disagree 38% of the time, while engineers disagree 59% of the time.
When the difference in response between management and engineers in the areas of software development process and quality and testing are examined, it reinforces the idea that these groups often have very different perspectives on the same issue. However, the magnitude of the differences indicates that the overall positive responses to the survey -- that organizations are generally doing very well in transitioning to agile methods -- may not be as positive as the overall response indicates.
However, in general, responses to these quality questions appear positive. Although the numbers might seem low, the questions were combinations -- comprehensive and automated testing. It appears that agile development practices are having a positive impact on quality -- a major focal point for all agile methods (which has been confirmed by other studies).
MANAGEMENT OF DEVELOPMENT
The management of development section of the survey was purposely longer because the emphasis in this survey was to focus on organizational agility rather than project agility. Often, agile methods are seen as a "development" issue when in fact the principles of agile development impact management and project management as well as development. It is difficult to sustain agile methods at the project level without changing management as well.
The question surveyed in Figure 13 gets at this issue of management of agility directly by asking about the hopefully changing perspective of customer value and schedule. Agile development focuses on delivering customer value, at a macro level and at a detail story/feature level. Schedule is still important and, in fact, may even be fixed by a project timebox, but truly agile organizations should begin shifting their emphasis to value delivered. As Figure 13 shows, progress is being made in this area, but there is still a significant emphasis on schedule in many organizations. While 51% of respondents somewhat or strongly agree with the value emphasis (actually, a surprisingly high number), 26% strongly or somewhat disagree. We are a little intrigued by the high percentage of neutral responses, at 23%. This may indicate, hopefully, that organizations are in a transition mode from schedule to value.
Figure 13 -- Management (all management stakeholders) and customer stakeholders measure success as much by the effectiveness of capabilities delivered (features and functionality) as by schedule. In other words, effectiveness is as important or more so than efficiency.
Another telling question in this area is shown in Figure 14. In our experience, one of the later transition issues to be addressed by organizations is that of performance measurement. Upper management and project managers have been measuring progress and performance in certain ways for a long time, and changing those perspectives, as indicated by the results in Figure 14, can take time. Only 31% of the respondents strongly agree or somewhat agree with the statement that measurements have been adjusted to reflect an agile approach, while 37% somewhat or strongly disagree. Again, the large number of respondents in the middle, 32%, indicate a transition may be underway.
Figure 14 -- Measurements related to project performance and success have been modified to reflect the differences in an agile approach.
Again, we are actually encouraged by the results from these two question areas. To have 51% of respondents say that value is more important than schedule in their organizations and that, in 31% of organizations, their performance measurement systems are changing to reflect the different emphasis of agile methods reflects a very positive trend.
Another question that addresses measurement is shown in Figure 15 -- whether or not project teams are encouraged to be realistic in their status reporting. This may seem like an odd question, but one of the difficulties we have seen in some organizations implementing agile methods is that early, realistic reporting based on working software is a problem. There are some organizations that only want to hear good news, and they certainly don't want to hear bad news early in a project. One of our consulting clients, for example, had a "green-yellow-red" stoplight system of indicating project status. However, project managers were effectively told, "Only green is acceptable." Many organizations operate in this "state of denial," and this is clearly not an agile perspective. Agile methods encourage early, continuous, and reality-based status reporting so this question gets at that issue from a management perspective.
Figure 15 -- Project teams are encouraged to be as realistic as possible in reporting project status.
Again, the positive response to this question is beyond what we expected, with 70% either strongly or somewhat agreeing that teams are encouraged to be realistic in their reporting. This shows that within organizations that consider themselves to be agile, reality-based reporting is indeed valued.
As we mentioned in the introduction, the survey questions didn't address agile principles directly, but attempted to get at the principles by asking about specific practices that would indicate whether or not principles are being applied. This is particularly true in these next two questions about decision making. Because decision making is at the core of self-organizing teams, we chose to ask about decision making rather than use a less tangible question about self-organizing teams. Figure 16 asks to what extent is there a clear definition of what decisions teams are empowered to make, while Figure 17 addresses the issue of whether most decisions impacting individual teams are made at the team level. The positive responses to the first question, that 52% of respondents feel there is a clear definition of what decisions project teams are empowered to make (something we call decision framing) seems very high based on our experience, which shows that few organizations have thought nearly enough about how decisions are made and who gets to make what decisions. In light of that experience, this response would indicate that agile organizations are doing a much, much better job of decision framing than the typical software development organization.
Figure 16 -- There is a clear definition of what decisions project teams are empowered to make.
Figure 17 -- Teams make the majority of decisions related to delivery of their product.
Similarly, Figure 17, which shows 48% of respondents indicating that project teams get to make the majority of decisions related to their delivery, is a positive agile indicator. For two questions on empowerment (and ultimately self-organization), the assumption might be that engineers would have a different response to the questions than management, but that assumption would be wrong. On both questions, engineers agree at a 47% level, making the difference between engineering and management responses very small. When management thinks it is empowering staff, and staff thinks it is empowered, agile principles seem to be well at work.
Again, the significant number of responses in the neutral category (23% for Figure 16 and 24% for Figure 17) indicates that transitions are taking place in many organizations (some teams may be operating this way, but not all). Hopefully, these transitions are going in the right direction of empowering teams to the best extent possible.
Another critical aspect of becoming agile, as a team or as an organization, is the responsiveness to change. One of the four Agile Manifesto value statements is "Responding to change over following a plan." The statement in Figure 18 deals with this aspect of responding to change, asking about an organization's comfort with evolving plans that respond to business conditions over a conformance to original plan mentality. It is interesting that the responses to this question are not as positive as some others in this category of management of development. Only 51% strongly or somewhat agree with this premise, while 32% strongly or somewhat disagree. Going back to Figure 13, which dealt with value over schedule (51% strongly or somewhat agree that value is more important), responses to these questions correlate well. However, embracing change is such a central and powerful theme in agile development, we would have expected more positive responses here. Conformance to plan, rather than conformance to customer value, still has a significant hold on organizations.
Figure 18 -- The organization actively embraces change: it is comfortable with evolving plans that respond to changing business conditions rather than a conformance to original plan mentality.
While another section of this report discusses the customer involvement and collaboration aspects of being agile, it is also important to gauge management's support of these activities. One way to do this is to measure project chartering efforts. Effective project chartering involves a serious commitment on the part of development and customer management to a joint, multiday, facilitated session attended by development and customer team members. These sessions should include discussions of product vision, business objectives, release and iteration planning, and more. However, even more importantly, these sessions provide a forum for individuals to begin jelling into effective teams. A commitment to chartering shows this commitment to collaboration as a critical piece of the agile initiative.
The responses to the question in Figure 19 are a little lower than others in this management of development section -- 39% somewhat or strongly agree, while 27% are in the neither category. The response to this question may be muted by the fact that many projects today, especially in larger organizations, are geographically distributed, making a face-to-face chartering session more difficult and expensive. However, given our experience with organizations, having nearly 40% agreeing that they do chartering sessions is an encouraging response.
Figure 19 -- Project teams employ a chartering process that involves the entire project community and that includes customers and the product manager in a multiday (depending on project size), collaborative chartering effort.
In Kent Beck's original XP book [1], one of his 12 practices is the 40-hour rule, which states that work beyond 40 hours tends to be unproductive, and, further, working intensely for a 40-hour week is highly productive and, therefore, additional overtime hours are unnecessary and counterproductive. When the Agile Manifesto was written, it was written to say "sustainable pace" rather than mention a specific number of hours -- that is, a pace at which the team could work sustainably over a long period of time. Responses to this question again show management's commitment and understanding to an agile process.
Surprisingly, 61% of the responses in Figure 20 show that working at a sustainable pace is important to organizations that consider themselves agile. Even more interesting is that we predicted that "overtime" would be more prevalent in software product organizations. But when we examine the responses from consulting, software publishing, and outsourcing/Web services respondents, the number only goes up to 62%.
Figure 20 -- Teams are expected, on average, to work at a sustainable pace without a large amount of overtime.
Finally in the management of development category, we wanted to see whether agile methods are impacting areas beyond the delivery of individual projects -- that is, to portfolio management. Our contention is that if short iterations and delivering small chunks of functionality (stories/features) are good for projects, then a similar idea would be good for portfolio management. In fact, in their book on portfolio management, Connecting the Dots: Aligning Projects with Objectives in Unpredictable Times, Cathleen Benko and Warren McFarlan make that very point without ever mentioning agile:
Simply put, Project Chunking involves taking larger projects and breaking them down into smaller bundles that reduce risk, realize benefits sooner, and increase flexibility by providing more choice points.... [Chunking helps] respond to uncertainty and velocity by building choice points an delivering value more quickly. [2]
Evidently, many agile organizations agree with this perspective, as 42% either strongly or somewhat agree that they are using an agile approach to portfolio management (see Figure 21).
Figure 21 -- The organization has an agile approach to the portfolio management process that includes decomposing large projects into smaller projects with discrete valuable customer deliverables and using this "chunking" of projects to increase portfolio flexibility.
FEEDBACK AND LEARNING
Feedback and continuous learning are critical to agile development from a variety of perspectives. Customer feedback helps teams learn about the product and keep it on track, while regular retrospectives help the team improve its performance over time. We included three questions in the survey to evaluate the degree of feedback and learning being practiced in organizations. The first is somewhat subjective in that it asks about management's support for learning, even when it comes from making occasional mistakes. The other two questions deal with specific practices -- regular retrospectives (team learning) and sharing lessons learned with other teams (organizational learning).
In Figure 22, 57% of respondents answered positively to the question about understanding of the learning concepts, and, interestingly enough, managers only gave themselves a slightly higher score on this question at 63%. Both other questions (retrospectives, Figure 23; and sharing, Figure 24) came in lower -- 43% and 50%, respectively, either strongly or somewhat agreeing -- indicating that more people support learning than have concrete practices to implement that learning. While we would expect the "learning" responses in this agile survey to be higher than the wider population of non-agile organizations, having only 43% conducting regular retrospectives indicates regular learning practices haven't been emphasized enough.
Figure 22 -- Leaders and managers at all levels understand and support the concepts of continuous learning, even when that learning comes from making occasional mistakes.
Figure 23 -- Regular retrospectives at the end of each iteration address people, process, and product issues, and issues are regularly followed up.
Figure 24 -- Project teams share lessons learned, both good and bad, with other teams.
As was the case with customer involvement, we found that, in general, size tends to adversely affect the frequency of retrospectives and sharing of lessons learned. This is especially noticed when size is measured by number of employees, number of IT employees, annual revenues, and by IT budget. We also found that learning and feedback are not seriously affected by company type or job title and that the concept of continuous learning seems to receive more support outside of North America.
CONCLUSIONS
The objective of this survey was to answer questions about the level of agility in organizations. While there has been a tremendous interest in agile methods over the last few years, we wanted to understand more about the depth of that interest, whether it is in name only or is grounded in significant use of agile practices.
The overall results of the survey are encouraging -- in fact, more encouraging than we expected when we started. Our expectations were that there was a significant amount of agile development that was agile in name only, but we were pleasantly surprised. Table 1 presents a recap of the five topic areas and all 26 survey questions. We summarized the responses into agree (either somewhat or strongly), neutral (neither agree or disagree), and disagree (either somewhat or strongly). Agree responses correspond to a reasonable implementation of an agile practice, while the disagree indicates a lack of implementing an agile practice.
Table 1 -- A Summary of All Topic Areas
| Agree | Neutral | Disagree | |
| Customer Involvement/Collaboration | |||
| Active internal involved | 71% | 12% | 17% |
| External customers involved | 56% | 17% | 27% |
| Cross-functional teams | 76% | 12% | 12% |
| Collaborative commitment | 67% | 16% | 17% |
| Distributed collaboration tools | 40% | 30% | 30% |
| Daily standup meetings | 62% | 13% | 25% |
| Software Development Process | |||
| Iterative project plans | 56% | 12% | 32% |
| Shippable software | 50% | 22% | 28% |
| Technical practices | 43% | 23% | 34% |
| Lean documentation | 54% | 21% | 25% |
| Risk management | 58% | 24% | 18% |
| Quality and Testing | |||
| Emphasis on testing | 57% | 19% | 24% |
| Developer unit testing | 35% | 30% | 35% |
| Comprehensive, automated testing | 42% | 20% | 38% |
| Management of Development | |||
| Effectiveness (value) over efficiency | 51% | 23% | 26% |
| Agile measurements | 31% | 32% | 37% |
| Realistic reporting | 70% | 15% | 15% |
| Clear empowerment guidelines | 52% | 23% | 25% |
| Teams make decisions | 48% | 24% | 28% |
| Embrace change | 51% | 17% | 32% |
| Chartering process | 39% | 27% | 34% |
| Sustainable pace | 61% | 19% | 20% |
| Portfolio management | 42% | 23% | 35% |
| Feedback and Learning | |||
| Support learning | 57% | 17% | 26% |
| Retrospectives each iteration | 43% | 25% | 32% |
| Share lessons learned | 50% | 15% | 35% |
In interpreting the results, we must also remember that respondents were answering for their entire organization, not just a few teams. We think this is one reason that 21% of all responses are neutral, as some teams implement agile practices within an organization, while others don't. Given the overall trends -- 52% agreement; 27% disagreement; 21% neutral -- we would project that many responses in the neutral category would move in the agreement direction (that is, more agile) over time.
Purposely, many of the survey questions were very demanding and multifaceted. For example, the technical practice statement -- "Development teams fully utilize agile technical practices: simple design, refactoring, test-driven development, coding standards, continuous integration, evolutionary design" -- used the term "fully utilize" and listed six specific practices. For a respondent to answer "strongly agree" to this question, which 14% did (and another 29% somewhat agreed), they would be indicating a very strong commitment to multiple practices. Similarly, in the statement about change -- "The organization actively embraces change: it is comfortable with evolving plans that respond to changing business conditions rather than a conformance to original plan mentality" -- the word "actively" and the phrasing regarding supporting evolving plans rather than original plans indicates a substantial commitment to agility.
Given the above, the overall 52% "agreement" by respondents across all topic areas indicates a very real and in-depth commitment to agile methods. The breakdown by topic area is also illustrative. Fully 62% of the respondents say they employ agile customer involvement practices. Given the state of the industry in this area, agile methods seem to be making a substantial contribution to improving customer involvement in the development process.
In the software development process area, 52% of the respondents indicate they have implemented project management and technical practices. We are a little surprised that customer involvement (62%) is higher than development (52%), given our experience working with companies and how difficult it is at times to get enough customer involvement. As mentioned earlier, the low positive response to the question about iterative development (56%) is also surprising and somewhat tempers our overall enthusiasm for the survey results.
The "agree" responses to the questions in the quality and testing area (45%) are lower than for other areas, but the results are in line with our experience. Transitioning QA from a waterfall environment to being able to automatically test products in short iterations can be very difficult as legacy code, and cumbersome testing tools take time to transition.
In thinking about each of the topic areas as well as our expectations, the management of development area is also a pleasant surprise. Our assumption was that the technical agile practices would be further along than the management practices, but that assumption was not borne out by the data. In this area, there is 49% agreement in questions about agile practices, 28% disagreement, and 23% neutral. Again, even the positive numbers were lowered by difficult and cutting-edge questions about portfolio management and a chartering process.
Finally, 50% of the respondents say they are using agile feedback and learning practices -- again, a reasonably positive number.
So the answer to the question, "How agile are organizations today?" seems to be "significantly agile." Overall, we are encouraged by the level of agile practices being used in organizations; it is higher than we anticipated. In the introduction to this report, we posed some questions about agility in organizations: first, we wondered whether agility was more than a surface phenomena; second, were only a few teams agile; and third, whether project teams were agile but management wasn't.
To the first question, the answer seems to be that agility is definitely more than a surface phenomena -- in fact, the data shows significant usage of agile principles and practices within organizations.
Second, given the size of organizations surveyed, it appears that multiple project teams in large organizations are agile, but there are still a number that are not. We also ran a comparison of organizations with under and over $1 billion in revenue. As might be expected, the larger organizations scored lower in overall agility (but in certain areas, such as risk management, they scored higher). To have these large organizations (over $1 billion) respond that 48% agree they do iterative project planning and 24% agree they use all the agile technical practices shows that even larger organizations are rapidly moving to more maturity in their agile methods usage.
Finally, and possibly the most surprising result, is that management of development practices are as widespread and in-depth as technical and testing practices.
We want to thank all the people who responded to our survey. The data contributes to a growing body of evidence that agile development practices are expanding not only widely across organizations but also in depth of practice implementation.
REFERENCES
1. Beck, Kent. Extreme Programming Explained: Embrace Change. 1st ed. Addison-Wesley Professional, 1999.
2. Benko, Cathleen, and F. Warren McFarlan. Connecting the Dots: Aligning Projects with Objectives in Unpredictable Times. Harvard Business School Press, 2003.
ABOUT THE AUTHORS
About Jim
Highsmith
About Dr. Robert K.
Wysocki
The agile movement is now more than five years old, and many organizations have implemented agile methods, with many more planning agile transitions. So we thought it was time to ask the question "How agile are organizations today?" Thus, Cutter Consortium conducted a survey to dig deeper into the issues of agile implementation. This Executive Report by Jim Highsmith and Dr. Robert K. Wysocki provides the results of that survey, which includes the topics of customer involvement and collaboration, software development process, quality and testing, management of development, and feedback and learning.
