In this Advisor series, I have been sharing practical lessons from leading Agile transformations as a coach. Enterprise coaching combines in-depth knowledge of contextual Agile with cultural change techniques. Here in Part VIII, I look at metrics to focus the activities creating enterprise agility.
“If you can’t measure it, you can’t manage it” is frequently misquoted as a W. Edwards Deming statement. Technically, it is true that Deming wrote these words in his book The New Economics. However, the actual full quotation reads, “It is wrong to suppose that if you can’t measure it, you can’t manage it.” Deming goes on to say that this is a costly myth, and many critical factors cannot be measured, such as artistic quality or pleasure.
Deming stressed the importance of using data to confirm beliefs about which organizational strategies and practices are working and which are not. In making this recommendation, Deming amplifies Shewhart’s work with the Plan-Do-Study-Act improvement cycle and the use of control charts. So Deming, a statistician, approved numbers to understand why something turned out the way it did. In The New Economics, Deming calls this variation. Variation is the difference between what we predict will happen and what was actually observed. He encouraged his readers to look deeply at the data to avoid making superficial decisions. Shewhart suggested looking for assignable causes of variation instead of chance causes.
Both individuals suggest looking beyond the obvious to learn the true cause of the difference between what we expect and the real outcome. To learn the true cause, it is often necessary to consider if appropriate data sets are being used for the judgement to be made. For example, it is pointless looking in detail at a train timetable if you are planning a journey by road!
So when embarking on transformational activities, are pointless measures used too frequently? When things don’t turn out the way we planned, do we jump to superficial decisions regarding the cause, or do we dig deep, as Shewhart and Deming suggest?
For example, why was the statistic that 70% of all projects fail from John Kotter in Leading Change allowed to be true for so long? (A number no longer valid, I am pleased to say!) Did we forget that projects that failed in the past were attempts at business improvements or changes in strategy and were therefore deserving a detailed root-cause analysis? Or did we jump at the chance causes of failure rather than digging deeper? Did we pursue significant change with a disengaged workforce? Finally, did we depersonalize transformational activities such that individuals were no longer engaged or committed to their outcome, and failure was the inevitable result?
How We Lost the Outcome Focus
The engagement of individuals with the outcome of their activity is so important in Agile. The final point in Deming’s list of 14 from Out of the Crisis reads, “Make transformation everybody’s job.” Yet, when coaching Agile teams, I often find people who have forgotten their role in producing solutions that will alter the direction or fortunes of their organization.
I blame in part the project custom and practice over the last 30-40 years as a potential cause of this disengagement. By disengagement, I mean people not understanding the connection between their activity and the business outcome. In traditional development approaches, organizations create a business case to justify their investment. That business case was explicit regarding costs, scope, and potential benefits. That business case was also subjected to significant scrutiny both from the financial and strategic perspectives. If this request was of significance, it might also be subject to an executive board approval. Yet, in reality, these administrative rituals did little to improve the outcome for the organization; hence the “70% of projects fail” statistic.
In effect, the business case was often used merely as a justification for the expense! Once approved, the business case was placed on a shelf while the team created requirements or functional specifications. From this point onward, the focus of the team effort is to fulfil the requirements specifications and not to deliver the business outcome. The business realization activities were often left outside the project management envelope, with the final step in the project lifecycle being something like deployment. In many organizations, the most difficult and most critical part of the project was left to chance. I speak of realization or making the business change happen.
In 2010, I had an assignment for a South African bank to compare benefits identified in the business case of completed projects to the actual results identified by the bank’s management information systems. I found that less than 28% of the targeted benefits could be identified. Some of the causes of this low number were due to a lack of management information. However, much of the cause was due to functionality being removed from the activity to achieve vanity targets. For example, all of the projects had completed on time, and the average project overspend was less than 10%.
I concluded that these impressive vanity metrics were achieved by reducing the purpose of the investment! (What generally at the time was called descoping.) As traditional techniques did not prioritize items of high value, important functional items or difficult items, they were often left to the end of the development. Consequently, if items needed to be dropped to meet budgets or timelines, this was done randomly without considering the impact on the business outcome. In many of the projects studied, we found that the organization didn’t get the desired result from the investment. The South African bank had lost the outcome focus of the business case because the data used to evaluate or predict project success, time, and budget had no relationship with the sought business outcome.
Contrast this situation with the Kennedy visit to the NASA Space Center in 1962. According to the popular story, President John F. Kennedy noticed a janitor carrying a broom. He interrupted his tour, walked over to the man, and said, “Hi, I’m Jack Kennedy; what are you doing?” The janitor responded, “Well, Mr. President, I’m helping put a man on the moon.” This janitor went to work each day with the outcome clearly in sight.
My observation is that by changing the focus of the team’s attention from the business outcome to fulfilling a requirement or functional specification, we were guilty of creating that “70% of projects fail” statistic. In my view, this failure rate is almost entirely because many traditional project techniques, such as the V model, or the waterfall SDLC referred to the specification and not the business case. Thus, to many developers, testers, and business managers, the rationale for the investment was hidden. This detachment from the business outcome also meant that people, unlike the NASA janitor, could not get emotionally attached to the activity and investment of their life.
Agile’s Different Paradigm
The statement people and interactions over processes and tools in the Agile Manifesto is more potent than many recognize. This statement is not just about fewer documents; it is also about communicating the vision. The efficacy of Agile is to align emotional intelligence with the creativity and combined effort of the team members in pursuit of the desired outcome defined in their business case. It is all about “I’m putting a man on the moon”!
In its various conversations, a team is made aware of the strategic impact of its solution and, therefore, the importance of its collective efforts. Business value is the primary measure used while prioritizing the backlog. The value of the outcome or how a particular work item contributes to that value is constantly being assessed. Furthermore, rather than dropping critical functions to remain in the budget as with my South African bank, the team chooses to complete the high-value or high-risk items as early as possible. This behavior has two significant impacts. First, just as with Kennedy’s janitor, the team is reminded of their role in the mission. It knows how vital is its contribution. Second, when the team experiences Deming’s variation and the plan is not turning out the way it had predicted, the work items that remain to be completed on the backlog will have a lower business value than those work items that have been completed.
In contrast to the fixed-requirement approach of traditional techniques, the Agile principle of welcoming changes to requirements, even late in the development cycle, results in a retention of business purpose throughout. Agile processes harness change for competitive advantage at their foundation and focus on the desired business outcome even if that goal has recently been altered. So it is with giving the team focus on their outcome that progression metrics play an essential part.
Why Progression Metrics Are Important
You may immediately think that progression metrics refer to velocity, timelines, defects, and the other metrics used to measure project progress; this is not the case! Progression metrics refer instead to the measures surrounding the desired business outcome. So if an Agile activity were established to reduce the time it takes to process an order, then order cycle time could be a critical indicator of project success. Likewise, if an online retailer sought to reduce abandoned shopping baskets, then monitoring the number and/or the value of abandoned baskets would be critical.
Establishing an appropriate measure for the outcome is sensible, and the objective and key result approach is often used. However, the critical part of the progression metric concept is that the team should regularly be provided with these measures. One of my colleagues called this “reverse project reporting”; data from the organization or market goes to the team for consideration and action.
As many know, Agile uses incremental delivery as a means to realize business value earlier. Therefore, a team may deliver several improvements designed to reduce the number of abandoned baskets as per the above example. By monitoring the impact of the deployment on the abandoned basket metric, the team can see if it is making a difference or not. If it did not have the desired result, the team can alter elements of the solution to reach its goal.
Progression metrics are more critical if the targeted improvement is subject to external forces. For example, if an organization seeks to improve its market share against a key competitor, they may make some operational and product improvements designed to increase market appetite. However, if a new competitor enters the market or introduces a new product that makes our product redundant, these situations could harm the business metric. Being aware that the overall situation has altered could cause the team to redefine their objective or redesign their solution.
Agile harnesses these late changes, and the techniques create adaptivity for the team. There is an Agile proverb: “pivot without mercy or guilt.” In essence, this proverb captures a remedy for an outcome that was not predicted. Agile suggests changing the backlog and making another plan, although there would be some governance implications in most organizations.
Picture a pilot trying to land an aircraft in fog without instruments. If we are honest, traditional project delivery methods were just like this analogy. The most common traditional project metrics were time and budget orientated, so in many ways, we were trying to land an aircraft in fog using tide tables and a depth gauge! Progression metrics at least provide the tools for an autopilot landing in fog. However, just because that data is ready to hand doesn’t mean its analysis will produce the appropriate result. Progression metrics may mean creating new sources of information; however, they help a team steer their activities to realize the desired business outcome.
So when Deming talked of looking for common causes of variation, of understanding that human schemes rarely go to plan, he recommended that we look carefully at the data. We should avoid drawing quick conclusions that a different outcome is simply the result of a special case. Instead, Deming was talking about using meaningful data — progression metrics — that point clearly to the business outcome. Progression metrics provide the information that indicates the decisions to be made and whether a strategy or management plan is working or not.
This Advisor has outlined how Agile keeps the business objectives in the team crosshairs. It explained that a team can see if their actions are moving toward success by monitoring progression metrics. Additionally, as Agile uses value throughout the definition, backlog refinement, and activity prioritization processes combined with incremental delivery, the team should receive early feedback if its activities are having the desired result. If not, then it can apply the Agile proverb “pivot without mercy or guilt.”
In the next Advisor in this series, I will outline how Lean Change Management reduces resistance to change and creates enterprize agility. Inspired by ideas from Agile, Lean Change Management is a feedback-driven approach to change management based on Lean startup and design thinking principles.
Series Recap: In Part I, I outlined the transformational change necessary to create enterprise agility, while Part II discussed using a plan to create structure and alignment for the transition. In Part III, I outlined the leadership team’s role in transformation, and in Part IV, I suggested how to create and maintain transformational momentum. Part V described the role of the Agile coach in supporting and enabling enterprise agility. Part VI examined Agile certifications and whether they add more value to the organization seeking agility or the certifying body itself, while Part VII looked at the importance of value streams in creating enterprise agility.