A Prediction for 2021: The End of Predictions
The world would be a much simpler and easier place to live in if we could predict the future. Of course, we cannot do this, but it doesn't stop us from trying. The market for predictions is huge, and people desperate for some certainty will take it from whatever source they can find. There are many approaches to writing prediction pieces. Some will predict the shifts in the technology market. Others will make predictions that favor the clients who pay the highest fees to the analyst's firm. Some will predict cultural or social trends and their impact on the markets. Last year, I predicted that 2020 would be the year that Agile got found out. Well, we all know that 2020 took an unexpected turn. So, in this Executive Update, let's look deeper into the outcome of my prediction and explore how Agile and agility, especially in the face of a global pandemic, has truly panned out since my "before the world changed" assertion.
2020, of course, turned out very differently than anyone expected. Agile was found out, but not in the way I had believed, which would have seen the growing rumblings in the corporate world amplified and a negative feedback loop, leading to the eventual rejection of Agile shamanism sometime in the next few years. Instead, COVID-19 made a dramatic showcase of the dangers of believing in the ability and capacity to simply react quickly to problems as they occur. Relying on the ability to react to events as they occurred was shown to be a very limiting strategy, as overwhelmed hospitals with limited access to personal protective equipment (PPE) quickly found out.
Another Agile trope — the tendency to defer decision making until the last possible moment when data or requirements will supposedly be richer — led to criticism of those who embraced that approach, especially when this was embraced as strategy (e.g., when the British government chose not to cancel the Cheltenham Festival or other sporting events or when the World Health Organization hesitated on confirmation of human-to-human transmission of COVID-19, both of which have been linked to wider spread of the virus). Elsewhere, it very quickly became apparent that globalized just-in-time (JIT) supply chains and the illusion of “agility” as a business capacity began to disappear as flows of goods were interrupted all over the world, leading to shortages and price hikes (understandably on hand sanitizer and masks and less understandably on toilet paper). When these serious problems continually presented themselves, agility was not enough, or sometimes not even possible, because decisions made in the past severely limited the choices available.
Even in cases where we could choose to see the positive impact of agility, we see the dependence on past decision making to make this possible. Teachers and students showed the world that behavioral agility is a natural human feature, flipping their entire existence over the course of days and still getting things done in the face of severe challenges, without the coaches, frameworks, belief systems, or management consultants that Agile methodologies would have us believe are necessary.
The relative successes of teachers and students wasn’t just a factor of their “can do” attitude or some inherent natural understanding of agility; it was made possible by affordances that came into being that were never designed to solve this particular problem (e.g., existence of the Internet, widespread access to technology, various collaborative tools and the collective societal knowledge to use them). These all existed in most developed countries, and the steps required to use them were not especially difficult.
From an individual perspective, surviving the pandemic has also been partly influenced by decisions made in the distant past. Factors such as weight, fitness, savings, and the ability to work from home have all impacted individuals’ ability to navigate the pandemic. The ability to be agile was shown to be entirely dependent on residue — what’s left over after a stressful event impacts us. Societies that discovered their care homes for the elderly were left exposed through financial neglect (i.e., due to underpaid forms of employment, leading to sick people going to work in old peoples’ homes) were experiencing the results of what happened when former chair of the US Federal Reserve Alan Greenspan started having dinner at writer/philosopher Ayn Rand’s house in the 1960s — the birth of neoliberalism and the consequent reduction in public services. Agility has been proven to be fairly useless when the decisions made on a societal level many years before an event are the things that determine whether agility is actually an option. The residues left after the impact of a stressful event determine to what degree an organization will be able to act with agility.
Agile as an Act of Prediction
So much focus has been placed on the ability to react to changing circumstances; many have become enthralled to the idea that design or forethought is not necessary and even dangerous and that reactive capacity is the most important skill. This has certainly been a theme in the field of software engineering.
Yet, the negative experiences of hospitals and the many positive experiences in education both show that what had already happened and decisions already made were more important than the reaction and actually constrained and shaped the reaction. This evidence points to the fact that agility is a feature of design and not something that exists in opposition to design. This means that agility, like design, is an act of prediction.
Agile organizations engage in design when making decisions to consciously enable reactive capacity in areas where they believe it will be needed. If they don’t do this, residual causality (i.e., decisions made in the past) will do the design work for them and shape the organization’s ability to respond. The pandemic has shown that seemingly unrelated decisions made long ago will be the factors that decide whether we will even be able to react at all. In some cases, decisions made in the name of agility — JIT supply chains for PPE being the obvious example — actually caused a huge reduction in reactive capacity. Thus, the act of introducing agility actually reduces the ability to be agile because it is an act of prediction for a future we cannot predict.
That prediction turns out to play a central role in Agile approaches, and the conclusion we have reached here — that the past constrains reactive capacity — are anathemas to the entire Agile premise, which sought to end detailed planning and predictive control. Organizational theorist Ralph Stacey has pointed out that new paradigms have a tendency to enable old behavior to continue with new vocabulary, so this is not surprising.
Agility is ideally the ability to let diverse approaches play out under uncertain circumstances. Counterintuitively, sometimes by introducing Agile programs that make predictions about exactly where to enable agility or by trying to enforce uniformity of approach, we constrain the ability to engage in diverse approaches. Moreover, the act of prediction is often used as a political tool, meant to persuade, and as such has been attempted to be used as a way to reduce diversity of approaches, which means less information and fewer future paths to probe as we move forward.
We have argued here that Agile is an act of prediction and that prediction in turn reduces agility. We could also argue that the pandemic has highlighted this, and as such, Agile truly got found out in 2020.
After All That, a Prediction!
All that happened in 2020 provides a careful lesson for those of us invested in predicting. The alluring concept of superforecasting, embraced by the recent British government and shown to be utterly ineffective by that very same government’s response to the pandemic, shows that we still hold out for the certainty that predictive ability brings and act as if it exists even when the evidence clearly points in a different direction.
The net result of observing the failure of reactive capacity in 2020 leads ironically to a prediction for 2021: this year will be the year that people lose faith in predictions. It won’t be the year that people stop making predictions because the temptation is just too great to be the next lucky superforecaster, but it will be the year many of us stop listening.
If we are lucky, more designers of systems and organizations will realize the importance of residue, of introducing greater optionality, of diversity of approaches to problems, instead of simplistic beliefs in easy solutions to complex problems.
For software engineers, the idea of residue has become important because we need to design systems for increasingly complex environments in which reduction is impossible, where the residue becomes more and more important for system quality. Without the diversity and optionality provided by design that is residual, systems that we build have very little chance of surviving. Every other method of engineering software involves projecting our belief, be that a process, a component structure, a requirement, or a product, onto a rapidly changing environment in a way that convinces us that our belief is indeed a solution.
2021 will be the year of the residue, in which we stop trying to predict the unpredictable and, most importantly, stop fooling ourselves that we have some innate ability to see the unseeable. We will realize that there are things we can do in the here and now to protect ourselves from unseen risk, today and for future tomorrows, none of which involve an infantile obsession with predicting the unpredictable and asking people to bet their lives on our ability to react quickly when these predictions fail.