CUTTER BUSINESS TECHNOLOGY JOURNAL VOL. 32, NO. 7
Michael Papadopoulos and Olivier Pilot examine how a limited view of digital transformation impedes organizations from fully benefiting from the new, Agile ways of working. They attribute this failure, fundamentally, to reliance on traditional architectural stacks where multiple teams and products rely on large, shared layers, and a change in a layer to meet the needs of one product may inadvertently break other products. To support a feature team–based organization, each team must have full end-to-end ownership of its stack, which consists of smaller, decoupled parts — microservices — that are loosely bound together. The authors advocate domain-driven design and the atomic design principle as the basis for enabling reuse. A managed, messy architecture is the key to an organization structured around feature teams, which enable digital transformation.
The mantra of “disrupt or be disrupted” has created an increasing demand for organizations to be more adaptable and responsive than their competition. Over the last five to 10 years, the idea of “digital transformation” has led to an increased prevalence of Agile ways of working in today’s enterprises. Agile promises to make teams more adaptable and responsive and to reduce products’ time to value. As organizations “go Agile” to reap the benefits of these new ways of working, they restructure themselves to have end-to-end feature teams. Such feature teams (e.g., Large-Scale Scrum [LeSS] or Spotify squads) theoretically are capable of taking responsibility for the design, implementation, and evolution of end-to-end, customer-centric features in digital products or platforms.
However, after trying this new organizational structure for a few months or years, organizations eventually stumble upon challenges and realize that the end-to-end feature teams aren’t really working for them. Though this approach to digital transformation usually succeeds in positively addressing a number of mindset issues, a fundamental problem arises when organizations try to make feature teams work with traditional architecture (or architectural patterns). Organizations quickly realize that fundamental limitations in their technology assets — systems, infrastructure, and tooling — prevent them from reaping some of the most valuable benefits of the new, Agile ways of working. The weakest link is to be found somewhere else.
Digital transformation has hit a wall. The need for reinventing how we think about and approach architecture is becoming ever more prevalent, especially if an organization is to truly become Agile.
The Need for a Managed “Messy” Architecture
It is important to understand that software development has changed significantly in the last few years. Software is no longer built totally from the ground up without any dependency on some form of existing software solutions, nor does software come purely from all-encompassing, off-the-shelf packages configured to one’s needs, spanning multiple business or technical domains. Software is now essentially built by “smart stitching” together already existing pieces of legacy code, new code, open source software and frameworks, and software-as-a-service (SaaS)/platform-as-a-service (PaaS) solutions, while leveraging core infrastructure-as-a-service (IaaS) components that also significantly affect functionality.
Thus, today’s best architecture is essentially minimalist, messy, and inconsistent. Architecture should no longer aim to provide robust, detailed frameworks for mandated solutions and componentry. Rather, architecture should focus on strong general design principles and ensure that it provides guardrails for security, scalability, availability, elasticity, and maintainability (all the typical “-ity” nonfunctional requirements). Moreover, architecture needs to enable solutions that allow for rapid evolution of product and platform features, as well as experimentation with new technologies, by ensuring that all integrations are open as well as API- and event-driven. Architecture should also ensure that all technology decisions support safe, error-free, low-latency, continuous delivery to facilitate a rapid pace of change in a trusted manner.
Fundamentally, we need to recognize that time to value is the most important decision factor in today’s world. Optimizing time to value through the ability to sustain a rapid pace of quality software delivery is what architecture’s contribution to digital disruption needs to be all about. As long as the solution’s reliability is assured, we should no longer care about duplication, solution “inconsistency,” or anything that conflicts with the more traditional approaches of the past that have emphasized consistency and reuse (see sidebar, “What Use Is Reuse?”). In today’s context, it is increasingly as relevant to design for (unexpected) change as to design the change itself.
What Use Is Reuse?
Consistency and reuse were a key requirement in even the near past mainly due to the operational burden of running multiple solutions — you needed people to manage the infrastructure, the databases, and the networks. However, with the advent of more automation, DevOps, and technological breakthroughs such as serverless moving us into low-impact operations, more solutions can be supported without the need for ever-larger operational teams.
Creating reusable software isn’t free, however. To support planned reuse, teams need to expend more effort creating the software — more documentation, more planning, more tests. This leads to an increased time to value.
With one of the key benefits of reuse fading away and the key challenge to creating reusable software still in play, should we really continue to strive for it?
Feature Team Limitations in Agile Transformations
Feature teams working in a traditional architecture cannot meet the agility expectations that Agile transformation promises. With a traditional architectural view, “feature teams” aren’t actually feature teams. They focus on the top layers of the architecture, the user experience layer, and, in the best case, the API layer. They rely on large, shared medium- to high-complexity layers underneath, which are difficult to safely and quickly adapt and change. Not only does this mean that feature teams cannot truly own their “stack,” it also means that the traditional promise of reusability and consistency is typically not kept, because only a limited separation of concerns by business domain exists in these shared layers.
Conway’s law famously says that “organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations.”1 When an organization decides to structure the teams that look after its digital products and platforms into feature teams, the technology stack must closely follow that structure. Most organizations, however, do not structure their technology stacks in this way, and the challenges that we observe today when trying to make changes at speed are usually attributable to that misalignment.
In the traditional architecture stack shown on the left side of Figure 1, products A, B, C, and D are all dependent on a few stacks, and there is high reuse and high consistency. The right side of the figure shows that changing one of the layers to meet the needs of product A may introduce changes that break B, C, and/or D. This is why Band-Aid solutions (e.g., increased dependency management, big room–planning events, Scrum of Scrums–style meetings, and giant dependency boards) are in vogue in some companies. Not only do dependencies introduce significant challenges regarding time to value, but they also usually introduce hidden costs, since nobody factors in the cost of additional project planning time and of time wasted in meetings devoted to ensuring that nothing gets broken.
Thus, we need an architecture that is capable of supporting a feature team–based organization at the deepest level. To achieve that type of architecture, products should effectively be microservices — smaller, decoupled parts that are loosely bound together to create a whole application or product. Importantly, it is acceptable to have duplication in these microservices.
When each team has full end-to-end ownership of the stack, as shown in Figure 2, changes in one stack do not affect any other stack. There is no need for increased dependency management, big room–planning events, Scrum of Scrums–style meetings, and/or giant dependency boards. This simplifies management, allows for rapid change, effectively reduces time to value (the most significant effect), and eventually decreases costs (through reduced management overhead).
A significant percentage of reuse is still possible, however, if you use domain-driven design and the atomic design principle (i.e., everything begins with the smallest element of the interface: the atom).2 We would argue that this is the correct approach to reuse as opposed to the monolithic layers frequently observed today. You can reuse and recombine your “atoms” to create entirely new elements (modules). These atoms on their own are not useful; in fact, they are not even deployed on their own; rather, they are shared building blocks. If you design your elements correctly, your atoms can become “shared libraries” across the duplicated components, as shown in Figure 3.
A level of orchestration still needs to happen on top of the atom, but that is where the specialization and decoupling from other services come in. Your atoms essentially become your underlying stack: your identity and access management service, your API gateway, your load balancer technology, and so on. Essentially, this way of thinking also helps you move toward “function as a service” and SaaS, enabling the rapid implementation and evolution of the digital products and platforms from which you decide to derive a competitive advantage.
For example, unless your business is in identity and access management, you probably should not spend time and effort creating an identity and access management stack; instead, you should just integrate an existing solution. The same criterion applies for all such technical component types (e.g., API gateway, database stores, data streaming, analytical engines).
Pay Attention to Architecture for Successful, Sustainable Digital Transformation
Digital transformation has focused on evolving organizational structures and processes to move closer to what digitally native organizations have defined as end-to-end feature teams. These evolutions are clearly going in the right direction for any organization wishing to fulfill the promises of digital transformation. But they are not sufficient.
Once the top layers of an existing technology architecture have been transformed to support more Agile digital product and platform implementation, much work still needs to be done at a deeper level to align architecture and feature teams. To achieve this, one needs to accept that the speed to market promised by end-to-end feature teams can be achieved only if such teams have maximum autonomy regarding the technology stack on which they build. In a traditional architecture, this is usually difficult since the technology stack is organized in large, shared layers with the — often missed — objective of enabling maximum reuse of technology components. Such traditional architectures make the lives of feature teams difficult at best, since changes to a feature can easily break another team’s feature.
Beyond Agile and Scaled Agile, architecture needs to be on the priority list of any organization wishing to go further in deriving competitive advantage from technology. Architecture’s focus should change from simply driving consistency and reusability to allowing a safely managed, messy, occasionally inconsistent architecture that focuses on minimizing time to value. Indeed, modern architecture needs to enforce strong principles for the design and safe delivery of software at pace, while reusability is managed at the atomic, rather than the component or service, level. This modern architecture, we believe, is a key area that will differentiate the winners and losers of the next wave of digital transformation.
References
1“Conway’s law.” Wikipedia.
2Hacq, Audrey. “Atomic Design: How to Design Systems of Components.” UX Collective, 28 June 2017.
