Applying Residuality Theory for Better Software Design

Posted August 19, 2020 | Technology | Amplify

While the software industry is currently grappling with ideas of complexity and resilience, there has been very little in the way of concrete actions or activities that software engineers can use to actually design systems. Residuality theory answers this need and draws on complexity science and the history of software engineering to propose a new set of design techniques that make it possible to integrate these two fields. It does this at the expense of two of the most important concepts in software design: processes and com­ponents. Moreover, the embracing of complexity science quickly points out that the process-component mapping that forms the backbone of conventional thinking in software engineering is, in fact, the reason behind systemic failure in enterprise software.

Identifying processes, eliciting requirements, and the rapid mapping of these two components are akin to designing cars based on tire tracks in a muddy field. Processes and components are what we see on the surface, but they are a byproduct of the business system execution. The problem is that designing systems has focused on trying to replicate the appearance of other established systems — much like the infamous cargo cults building airplanes of straw — mimicking what was seen but without any real understanding. Componentization can thus be categorized as sympathetic magic.

Residuality theory, conversely, introduces the residue as the alternative building block of software systems. A residue is a collection of people, software functions, and the flows of information between them. It is what we imagine to be left of the system when it is impacted by a particular stressor — an event such as a fire, market crash, or product failure. For every stressor, a residue is created and augmented to be better able to survive the stressor. Therefore, designing an entire business system involves the integration of many, many residues. Processes and components previously believed to be first-order citizens of any model emerge from the integration of these residues. This completely changes how one should think about the design of systems. A software architecture can now be seen as a multidimensional structure of interrelated residues, rather than as a two-dimensional diagram of component relationships. Furthermore, traditional, linear risk management is superseded by early analysis of stress on the system with a focus on vulnerability rather than prediction. The residue forces the designer to work consistently with the software and the environment at the same time. This is a drastic change to the design process.

Residuality theory does the following:

  • Models systems as collections of residues. Considering the limitations with components, we need a new model. The residue embraces complexity science, viewing software components as agents in a system of people, external organizations, and information flows. The residue in a complex environment is equivalent to the component in a complicated environment.

  • Builds on complexity science. Residuality theory is built on the idea that business environments are inherently complex and, therefore, unpredictable, resistant to best practices or pattern-based approaches, and are not static in their nature.

  • Assumes fat-tailed distributions and non-predictability. The external stressors that impact a business environment are too numerous to list, and the probabilities so intertwined that they are impossible to establish; therefore, risk management as practiced by most organizations will fail to identify the risks that will impact the system and does not contribute well to the design effort.

  • Assumes complex business environments and complicated software systems. In complex busi­ness environments and markets, the behavior of a complicated software system is defined by events in the surrounding, complex business system. This is where complexity science has much to add to the software industry’s understanding of the world. The vast majority of software solutions are complicated; they can be understood, modeled, and mapped and are constrained by design. However, these software systems exist inside complex environments, the business system, which cannot be predicted, modeled, or mapped, as the variations are simply too many. The fluctuations in the wider, complex business system are what determines whether component choices are wise or not. Too often, it is believed that complexity is in the software, or that this complexity can be simplified by simplifying the software. But this complexity actually forms the shape of the complicated solution and will do so naturally over time, patch by patch, if the software solution survives the stress it is exposed to in its naive form. There are so many stressors that can cause a program to change that it is impossible to identify and describe all of them. The programmer quickly becomes overwhelmed and retreats to the shamanistic rituals of component divination. Residuality theory recognizes that software involves complicated systems in complex environments, and the difficulties that this causes when expertise in one area tries to diminish the importance of the other, and overcomes this issue by using residues, collections of elements that span the divide and encourage analysis that consistently amplifies the risks in treating complex systems as complicated in order to quickly identify solutions.

  • Uses stress as the driver of design decisions. Huge problems in enterprise software are often caused by ignoring nonfunctional requirements until the functional design is complete. Residuality theory quickly identifies these requirements by analyzing stress and vulnerability rather than probability. Each stressor hits the system in a particular way. Flooding destroys the basement, but the upper floors are OK. Fire destroys the entire building, but the fireproof safes are OK. Each stressor has a related residue — the bits that are still working afterward. Residual analysis examines each residue in turn and asks, “What is needed here to make sure that the system is still working, or that the largest possible part of the system is still working?” The result of analysis is the augmentation of each residue in turn. Eventually, there are dozens of augmented residues, each one surviving a particular form of stress. There is no need to establish the probabilities for these stressors, or identify all of them, or even identify which are more likely. The design effort requires just enough stressors to arrive at a resilient design, not the mitigation of individual risks. Software functions exist inside these residues, and the residual augmentation will cause redrawing of boundaries between these functions to protect the software from the contagion, limiting the impact of a stressor as far as possible so that the flooding in the software basement doesn’t destroy the fireproof safes. These residues will seem unconventionally inefficient from the factory perspective. There is a great deal of repetition. Residues can be very similar to each other. The work of designers of software systems is now to integrate the residues to produce the final design. Here, architecturally significant decisions are made about which functions should be general and which should remain isolated inside the residue to prevent contagion.

  • Analyzes contagion as residues are integrated. Linear risk management is dangerous in complex environments as it reduces risk to a number of singular impacts based on bias-fueled probabilities and impact assessments. In truth, stressors can impact a system more than one at a time and in any order; contagion analysis forces the analysis of interaction between residues in terms of the interplay between stressors. This involves investigating how stress impacts other residues and how it influences decisions about shared logic across residues. Using simple matrices to investigate contagion and dependency drives decisions about the structure of the software system based on the reality of the business envi­ronment and the stress it may suffer, not based on dividing along functional or organizational lines.

  • Allows processes and components to emerge rather than defining them straight away. Rather than matching problems to patterns or using best practices intended for different business environments, components and process emerge during the process of residual analysis. They become products of the stress the system will be exposed to, as they would naturally over time.

  • Shows results directly. Using residuality theory instead of standard methods of componentization would see a massive increase in quality in software architecture. It turns out that systems built like this can have abilities to withstand unknown unknowns — stressors that they have not been built to withstand. This potential property is essential for systems that will spend their existence in complex domains. Once a design is established, the concept of stressors can be used to continue to test the design, showing that the system performs better when exposed to unknown stressors, so the process provides immediate, quantitative feedback that the technique has worked.

The result is a stack of residues that have relationships to each other, and a new model, or view, of the system emerges. Residual analysis arrives at groupings of functions, components that allow for the execution of business processes. We haven’t partaken in any of the rituals of componentization, yet we have designed something that is responsive to the environment around it.

[For more from the author on this topic, see “'There Is No Spoon’: Residuality Theory & Rethinking Software Engineering.”]

About The Author
Barry M O'Reilly
Barry M. O’Reilly is a Senior Consultant with Cutter Consortium and a member of Arthur D. Little's AMP open consulting network. He is the founder of Black Tulip Technology and creator of Antifragile System Design. Previously, he held positions as Chief Architect for Microsoft's Western Europe practice and IDesign, IOT TAP Lead for Microsoft’s Western Europe practice, Worldwide Lead for Microsoft’s Solution Architecture Community, and startup CTO… Read More