E-Mail
Understanding Perceptions of IT Agility
by Lou Mazzucchelli and Tim Lister, Fellows, Cutter Business Technology Council
The term "agile," which has gained increasing popularity in today's IT and larger business world, is applied as an adjective to a variety of activities, ranging from computer programming to organizational behavior. The word agile stems from the Latin agere, an imperative form of the verb ago, meaning do or drive (or sometimes "drive back," a meaning that might resonate with harried IT practitioners). Common definitions of agile include phrases like "moving quickly and lightly" and "with quick and easy grace." One dictionary even adds the notion of "quick motion in the limbs," helping to dissuade using the word to describe, say, whales.
But here we are talking about agility applied to the closest organizational construct we have to whales -- the business enterprise. One could argue that "enterprise agility" is an oxymoron. But, if we take the "at the limbs" idea to heart, it seems to make sense that we could talk about, for example, the agility of an elephant's trunk; likewise, in the business world we can talk about the agility of some smaller but attached part of a larger enterprise. It remains an open topic in our minds whether agile components can be aggregated, scaled, and controlled in ways that preserve agility. The purpose, however, of this Cutter survey on enterprise agility and IT infrastructure was to collect data to help us better understand how organizations view themselves with regard to IT agility and to understand the factors driving perceived (as opposed to measured) success or failure in the quest for agile IT development.
We received a strong response to the survey, which is perhaps indicative of the sample audience's curiosity surrounding the topic. Our first task in analyzing the survey results is to see if the data leads us to any obvious conclusions. We then look at the IT agility perception versus reality. Finally, we take a step further and examine the relationship between agile methods and problem reports.
SOME FUNDAMENTAL ASSUMPTIONS ABOUT AGILITY
We think of agile methods as a family of approaches aimed at dealing with uncertainty. If there is a high level of uncertainty about what exactly you will end up building, then the agile approaches (all of which call for multiple short iterations with ongoing customer feedback) make enormous sense. With the agile methods, we incrementally steer our way to a useful system. On the other hand, if there is a clear and explicit definition of what needs to be built, then the iterative approach significantly decreases in value, and a preplanned, classic, phased approach may be sensible.
In examining the Cutter survey data, we separated the respondents into two camps: those who responded "yes" and those who responded "no" to the question, "Does your organization explicitly consider the need for IT agility in its IT strategy and plans?" (see Graph 1 in the Survey Data section). Basically, two-thirds of the survey respondents are in the "yes" camp, and one-third are in the "no" group.
We then looked at predictability of requirements as a key indicator that an organization would be more likely to report agile behavior. The assumption is the more uncertainty you have in your requirements, the more likely you are to use an agile method. We focused on the question, "How easy is it to predict the requirements for your organization's IT systems looking three years ahead?" (see Graph 6). For our analysis, we chose to divide the responses to this question into two main groups: "predictable" and "uncertain." The predictable group is made up of the combined respondents in Graph 6 who chose "very predictable" (5%) or "mostly predictable" (36%); the uncertain group represents those combined respondents who chose "mostly uncertain" (23%) or "very uncertain" (1%). We then further segregated these survey results and the responses to the question mentioned earlier regarding IT agility being considered in the IT strategy. Table 1 shows the relationship between those who have and do not have agility as part of their strategy and the level of predictability in requirements.
Table 1 -- Survey Data Segregated by IT Agility in Strategy and Predictability of Requirements
Table 2 -- Survey Data Segregated by IT Agility in Strategy and Percentage of Outsourcing
Table 3 -- Survey Data Segregated by IT Agility in Strategy, Adoption of Uniform Agile Methodology,and Percentage of Projects that Use an Agile Process
Yikes! According to this data, higher predictability leads us to agility, while higher uncertainty leads us away from it. As it is sung in The King and I musical, this is "a puzzlement."
So then we thought that maybe those who are actually doing their own application development work report agile behavior, while those who are outsourcing have little need for agility since they are willing to be at an arm's length with the actual construction process.
Again, we separated the same "yes/no to agility in strategy" groups and then looked at the question, "What percentage of your organization's application development work is outsourced?" We looked specifically at those who had significant outsourcing; for this, we set the bar at 50% outsourced or higher (see Table 2). We even made a category for those 100% outsourced.
Yowzers! (Lou disclaims this exclamation, but agrees with its intent.) Again, this shows the opposite of what we thought we'd find. Heavier numbers reported outsourcing in the agile population, and even four organizations that explicitly consider the need for IT agility have sold the ranch: they are 100% outsourced. Did they explicitly consider the need for IT agility and reject it?
Desperate to leave this "Mad Hatter's Tea Party," we then looked at our yes/no populations in terms of the question, "In projects that use an agile process, does your organization adopt a uniform agile methodology?" (see Graph 7). Responses could either be "yes," "no," or "none of our projects use an agile process." (In the sample as a whole, of those reporting use of an agile methodology, about half reported a uniform methodology and half did not. A bit over a quarter of all respondents used no agile process.)
We assumed that the truly agile would not be lockstepped into a single uniform process; their process itself would be agile. So we also looked at those who reported "75% or more of their projects" in response to the question, "What percentage of application development projects in your organization use an agile process?" (see Table 3).
So much for our assumption: there is a clear bias toward a uniform process for those heavily involved in agile development (when we looked inside that group, Scrum seems the hands-down winner of all the named processes). This apparent process uniformity might spring from immature agile teams who have not yet learned to tweak their process. But we are at a loss to explain the results from 12 respondents in the "yes" population who explicitly consider the need for IT agility but respond that they have no agile projects. One possible explanation for this would be an assumption that an outsourced project was not agile by definition. The overall IT strategy might be considered agile, but the projects are not.
IT AGILITY PERCEPTION VS. REALITY
Perhaps the most important question about agile approaches is "Do they work?" Problem reports per month might be a good indication of development efficacy. The data reported in our survey across all respondents was as follows:
- Problem/bug reports per month
- Average: 65
- Median: 20
In the aim of full disclosure, we did a tiny bit of smoothing here as one respondent reported 7,000 problem/bug reports per month, and it was adjusted down by an order of magnitude (Tim and I also considered sending an emergency team to this site).
If we take the results above as "reality" for our sample, it is interesting to compare this against the perception of IT agility level. Graph 3 shows the data reported by the entire group.
We note a slight negativity among some of the group, in that the "somewhat worse" response is slightly higher than the normal distribution would predict. But, generally, there is a skew toward "better" in the population's self-assessment of IT agility compared to its competition.
We also asked respondents to qualify the size of the group to which their answers to this survey applied: department, division, or an entire company. This response allows us to segment the data by size, as shown in Figure 1.
Figure 1 -- Self-rated IT agility level vs. competition by group size.
Responders for companies seem to have a much higher opinion of their IT agility than their division or department counterparts. Perhaps this is because of a more global awareness of agility efforts being undertaken throughout the company. Or, it could be a result of what Tim and his colleagues at the Atlantic Systems Guild call "news improvement" 1 in action: someone high enough in an organization chart to report for an entire company might be the recipient of skewed data from the troops below.
We see that perception of IT agility varies with organization size, but how does that perception match the reported reality? We separated out responses by size and compared their self-ratings of IT agility against the reported actual problem report frequency in Table 4.
Table 4 -- Problem Reports per Month by Agility Rating
When we compare the reported problems per month by rating with the overall average problem reports per group, we can get a sense of the gap between perception and reported reality (see Figure 2).
As shown in Figure 2, department respondents rating their IT agility as "somewhat worse," "about the same," and "much better" submitted problem report rates well below the average for their group (we ignore "much worse" here since there was only one data point). One could speculate that a smaller group might assume, due to limited resources, that it must be worse off than others (the grass is always greener on the other side of the fence). However, this does not explain why those rating themselves "somewhat better" than their competition also reported problem rates above their peer averages.
Figure 2 -- Agility perception/reality gap using problem-report rate proxy.
Our division respondents fared somewhat better in this gap analysis. We see that those who rated themselves "somewhat worse" actually had poorer performance than their peer group average. The group reporting "somewhat better" had better performance than the group average, and the "about the same" responders were close to the average. The apparently more accurate self-assessment may be because the larger group has more experience or more resources.
This suspicion is borne out by the company-wide data. Again we see that those reporting poor IT agility also have higher problem-report rates, and those reporting better agility are experiencing lower problem-report rates.
USE OF AGILE METHODS VS. PROBLEM REPORTS
One survey question asked for the percentage of projects using agile methods. We compared the reported problem-report frequency with reported agile use. In a world of perfect methods, perfect teams, and perfect organizations, we would expect to find a strong inverse relationship between this data (i.e., the higher the percentage of agile projects, the lower the problem-report frequency; we know this might not account for a singular flaming project disaster, but, hey, we're not statisticians).
The responses, as you might expect, do not paint such a definitive picture. The first set of graphs in Figure 3 shows the results for respondents answering "about the same as my competition" for their IT agility level, grouped by response scope.
Figure 3 -- Problem-report frequency vs. agile use: "same" responders.
For department responders, there appears to be no relationship between agile methods use and problem-report frequency. The company responders reported more agile method use, but the data does not indicate the kind of inverse relationship we might expect. The closest result to our assumption is for division responders, but we would still call the data "lumpy" at best. One possible explanation for this is that the problem-report frequency question related to all projects, not just those developed with agile methods, and responders in the "same" category would not be expected to be doing so much agile development to have an influence on overall development performance.
Next we looked at the data for "somewhat better" respondents; here we would expect a higher percentage of agile method projects and therefore a more pronounced global impact (see Figure 4).
Figure 4 -- Problem-report frequency vs. agile use: "somewhat better" responders.
Amusingly, in this data set, we begin to see the relationship we're looking for in department and company responders but not in division responders. In fact, the division data suggests that there is a direct relationship between agile usage and increased problem-report frequency -- not a claim we're prepared to make. The best result is in the department data (but with a very small sample size). The company data reflects higher agile method use overall, and an encouraging suggestion of reduced problem-report frequency with higher agile method use.
ONE MAN'S CEILING IS ANOTHER MAN'S FLOOR
It appears that agility takes on very different meanings in different organizations. We need to be extremely careful about making any assumptions when someone tells us they are in an agile organization. As Alan points out in his article, many organizations identify IT agility as an explicit objective, but few (less than 25%) have developed rigorous measures of this attribute.
We think this is the primary source of the inconsistencies in the survey results: organizations are using wildly different metrics and assumptions about what constitutes "agility" in both strategy and tactics.
We looked at the problem reports as a proxy for agile method efficacy and were not able to draw strong conclusions based on the data. However, Alan's analysis of new development and problem resolution times seems to show a promising correlation between agile method use and reduced delivery times on both tasks. Our survey did not explicitly call out problem-report rates by project; this means that large shops with less than 100% agile projects would report a blended number that makes analysis difficult.
Responses to one specific question -- "Has your organization committed to a service-oriented architecture (SOA)?" -- revealed a significant difference between our yes/no populations (see Graph 4 and Table 5).
Table 5 -- Survey Data Segregated by IT Agility in Strategy and SOA Commitment
This is a work in progress, as only 12 of the 64 organizations committed to SOA reported that the number of implemented services equaled the number of published services. But it is also heartening to see that real investment in architecture is occurring. In our opinion, the limiting factor to advances in higher velocity development is most often the architecture, or lack thereof. Building an architecture on to large numbers of legacy applications (or a software shantytown, as dubbed by Cutter Fellow Lynne Ellyn) cannot be done cleanly. The urban renewal of SOA has to take place to liberate the potential of increased productivity and quality through agile development.
SOME OVERALL OBSERVATIONS AND UNFINISHED BUSINESS
In trying to establish some baseline measures of agile behavior, we did not focus the survey on organizational issues that inhibit agility. Identifying and mitigating these issues has been a recent focus of our consulting activity, and we have collected a preliminary though not exhaustive list, which includes in no particular order:
- Conflicting demands of fixed-price contracting and agile development
- Preserving agility across business unit/department/outsourcing boundaries
- Glacial or pathological decision-making processes
- Poor architectures and interfaces that impede progress on legacy systems
- Employee incentive programs that are not aligned with business strategy
- Inadequate funding to support test environments
In summary, we are encouraged by the results of this survey, which show some reasonable awareness and practice of agile behavior. We are particularly encouraged by less than unanimous reports of agile behavior. As observers of trends past, we note that this one has not yet moved to the "gratuitous acknowledgement" stage; debate means that people are taking the ideas seriously enough to vote "no."
ENDNOTE
1 DeMarco, Tom et al. Adrenaline Junkies and Template Zombies. Dorset House, 2008.
ABOUT THE AUTHORS
This issue focuses on the management of the agile enterprise and on understanding how organizations can facilitate and foster agile practices through investments in IT infrastructure and technology practices. As such, the survey our contributors crafted tackles issues of strategy, relative positioning and competition, as well as technology infrastructure, software development methodologies, and IT architecture.