CUTTER BUSINESS TECHNOLOGY JOURNAL  VOL. 32, NO. 7
  

Software has become the main differentiator by which organizations compete in today’s economy. Through software we achieve faster delivery, higher efficiency, more accurate manufacturing, and higher customer satisfaction. To become and remain competitive, it is imperative that your business master software evolution. This does not only mean being able to create a winning application once; it means always being able to immediately adapt your applications to the unique needs of your users and your market and to continu­ously swap out those software components that no longer provide you with a competitive edge. If your business is not able to relentlessly keep evolving its software portfolio, you will fall behind.

Software Has Eaten the World

It has come to pass: software has eaten the world.

In 2011, Marc Andreessen used this imagery to describe his personal vision of what was then still to come: “My own theory is that we are in the middle of a dramatic and broad technological and economic shift in which software companies are poised to take over large swathes of the economy.”¹ Today, this is accepted lore and is not limited to Silicon Valley:

[T]he digital economy is worth US $11.5 trillion globally, equivalent to 15.5 percent of global GDP and that has grown two and a half times faster than global GDP over the past 15 years.²

The world has woken up to the increasing importance of software, and the question now for businesses every­where is not whether to embark on a digital transformation of their business, but how to trans­form suc­cessfully — and as soon as possible. For new businesses, digital is not a question but a given.

Software Needs to Evolve

But digital transformation is not an end point; it is just a beginning. By going digital, your organization is only entering the game. To actually play and win, it is not sufficient to build a great digital product or service just once. Rather, it is imperative to continuously evolve your digital solutions to keep meeting the needs of your users and your market. After successful transformation, you need to master software evolution.

Starting in 1974, Professor Manny Lehman and his col­leagues famously sought to capture important insights regarding software evolution by formulating a series of Laws of Software Evolution.³ Though these laws predate the dawn of our digital economy by several decades, let’s review three of them for clues they hold to help us evolve our digital assets:

• First law: law of continuing change. This law states that any software system in an organizational context “must be continually adapted, else it becomes pro­gressively less satisfactory in use.” In other words, software that is in actual use has the peculiar charac­teristic of being indefinitely unfinished. Once a soft­ware component has been created and first released, it becomes the subject of a stream of subsequent changes: bug fixes, updates, enhancements. Software that does not evolve at the right speed and in the right direction quickly falls out of use.

• Second law: law of increasing complexity. This law states that “as a [software system] is changed its complexity increases and becomes more difficult to evolve unless work is done to maintain or reduce the complexity.” Thus, with each evolutionary step, the code tends to become messier and the architecture more entangled, making it harder and harder to apply further changes. Software that evolves becomes difficult to evolve. An explicit effort is needed to counteract this tendency.

• Fifth law: law of conservation of familiarity. This law states that “the incremental growth (growth rate trend) of [software] systems is constrained by the need to maintain familiarity.” In other words, the speed by which an organization can enhance and grow its software systems is limited by its collective intellectual capacity to understand the structure and behavior of these systems. Knowledge dissipation and software growth conspire to kill software development productivity.

While these laws tell us that evolution is necessary (first law), potentially self-defeating (second law), and knowledge-bound (fifth law), they do not identify the actual sources of evolutionary pressure. How is it, exactly, that software “becomes progressively less satisfactory in use” as time passes?

Evolutionary Pressure

As in biology, changes in the environment fuel evolution. For business software, the environment is the marketplace. As Figure 1 illustrates, we can readily identify five types of market forces:⁴

1. Innovation. Businesses compete by bringing new or improved products and services to the market. Software supports the production of products and the delivery of services. Sometimes, software is an integral part of the product. Other times, software is the product. Business innovation drives software change.

2. Cost reduction. Services and products that were once innovative lose their differentiating power when competitors start offering the same for less. In markets where similar products or services compete on price, the operational costs of the software systems that support them become a critical factor. Reduction of operational costs drives software change.

3. Growth. A successful software business attracts new users and retains existing ones. This leads to a growth in interactions and in the volume of data processed, stored, and served. Unless storage, algorithms, and interfaces are optimized, the system performance will degrade and hurt usability. Growth drives software change.

4. Regulation. Governments are constantly at work to change laws and regulations, be it for the betterment of society or for propping up the financial system. Such changes in the rules require modifications not only to the governmental soft­ware systems that enforce the rules, but also to the software systems of banks, airlines, and other businesses that must comply with these rules. Laws and regulations drive software change.

5. Coevolution. Each software system is dependent on others. For example, a Web store depends on a payment system, a database system, an Internet browser, several operating systems, and so on. Apart from those execution-time dependencies, software systems have development-time dependencies on libraries, frameworks, and development tools. All those systems and components are like­wise under evolutionary pressure. Changes in any of these induce the need for updates in the system that depends on them. Thus, changes in one system drive changes in other systems by propagating through the network of dependencies among them.

While these environmental factors fuel software evo­lution directly, they also have a significant indirect effect. Each change inevitably introduces bugs. As a result, the initial change indirectly leads to the need for further changes (bug fixes) down the line.

These (direct and indirect) evolutionary pressures from the marketplace explain Lehman’s first law of software evolution (continuing change).⁵ But, according to laws two (increasing complexity) and five (conservation of familiarity), such evolution rapidly runs into problems if the size and complexity of the system are allowed to increase unchecked.

Survival of the Fittest

These problems are not imaginary. When seemingly small feature requests take ages to complete, when stability problems appear impossible to stamp out, when developers are afraid to break the system when making a change — these are the symptoms that your digital asset has evolved into a liability.

Unable to relentlessly keep evolving your software portfolio, your business will lose competitive power and will fall behind. Your team will waste its time and focus on software that does not make a competitive difference. The experience of your users will become unremarkable, and you’ll find yourself making excuses rather than delivering on your promise.

Only the fittest survive in a world that is being eaten by software. Those that do not evolve at the right pace in the right direction will see their software eaten by the world.

Making Evolution Sustainable

Sustainable evolution requires two more types of change, not fueled by the marketplace, that must be initiated internally (refer back to Figure 1):

1. Refactoring. The complexity of the program code of a software system can be reduced through a series of small changes that improve its structure but preserve its behavior. Performing such incremental code improvements is called “refactoring.”⁶ While refactoring takes effort, it does not provide any direct value in terms of new functionality. For this reason, developers may find it difficult to justify their refactoring efforts to their colleagues and managers. Nonetheless, timely and judicious refactoring is generally considered a best practice to keep code complexity in check and, hence, prevent the second law of software evolution (increasing complexity) from kicking in.

2. Commoditization. Modern software is not built in a vacuum. Rather a range of generally available functionality from external software libraries, frameworks, and services is used as a platform on top of which new, innovative functionality is created. However, the boundary between “innovative” and “generally available” shifts over time. Pieces of functionality that you developed inhouse several years or perhaps just a few months ago may have given you a competitive edge then. But others have taken notice and developed similar components, perhaps in a more cost-effective, smarter way.⁷ And recognizing the general utility of this functionality, they may have made it available as a reusable component. Your once-unique functionality has become a commodity, and the smart thing to do is to exchange one for the other. Doing so will relieve your team from continuing to evolve that functionality, and you can instead focus on functionality that makes you unique and competitive. While swapping homebrew against commod­ity may involve substantial effort, it can prevent the fifth law of software evolution (conservation of familiarity) from killing the productivity of your developers.

Since refactoring and commoditization are not sparked by external pressures, they are easily forgotten or de-prioritized in favor of other types of change. They are motivated by long-term sustainability and require a degree of foresight that may feel antithetic to our fast-paced digital age. The judicious application of refactoring and commoditization is a critical success factor, nonetheless.

Mastering Software Evolution

So what are the essential capabilities for any organization to master software evolution, both on the engineering and on the leadership level?

Mastering software evolution means that leadership, the business, and engineers need to work together to balance the various evolutionary pressure factors in such a way as to sustainably outpace the competition. Sustainability means bringing new functionality to the market while keeping your underlying software assets healthy and nimble.

To achieve such balance is easier said than done. It requires a form of data-driven design and decision making that balances commercial and technical con­siderations. Let’s review what is needed in terms of data, design, and decisions.

Data

To feed your design and decision-making processes, your organization needs to look for data in at least three directions:

1. Look around. At any moment, your organization needs to know what is available in terms of reusable libraries, frameworks, services, tools, and technologies. And you need to have an informed opinion on which of those may be useful to you. To get this information, you can tap into external sources, such as the ThoughtWorks Technology Radar.⁸ For a selected shortlist, you should consider a more experiential approach such as a hackathon.

2. Look inside. Your organization needs to have an up-to-date inventory of its software assets. This is not just a list of systems, but also their inter­dependencies and their properties, including volume, quality, rate of change, functional focus, and technical health. To get this information, your software development infrastructure needs to include measurement instruments, such as code scanners, architecture analysis tools, and metric dashboards.

3. Look ahead. Your organization needs to know which market needs you will likely need to satisfy in six to 12 months. To get this information, you need to have product owners on board that curate a high-level backlog of product ideas that can be detailed out into feature descriptions. To collect these product ideas, your product owners need to interact intensively with users, business represen­tatives, thought leaders, and (whenever possible) competitors.

These three data streams provide the basis for design and decision making regarding software evolution.

Design

Your organization’s software design process must go beyond the new functionalities you want to add. Design must also deal with restructuring and eliminating existing software components:

• Refactor. The purpose of refactoring is to improve the design of the system to counteract complexity increases (as described above), as well as to prepare for upcoming additions and deletions. Your software teams must master the art of refactoring and must have the mandate to invest the necessary effort.

• Enhance. An optimally designed enhancement not only takes into account what should be added in terms of functionality, but also how and where in the software the changes are to be implemented, given what is known about the current state of software components, their interdependencies, and their likely commoditization trajectory. (For example, if your next feature gives you a unique competitive advantage, its implementation should not be mixed in with low-level generic functionality; it should be designed to remain confined to higher-level, product-specific components.) Your product owners, architects, and developers must optimize enhancement designs together.

• Eliminate. The design activity should include determining what functionality to remove (or replace by commodity components), when, and how. Functionality that is no longer desired may have been inextricably woven into the code, such that significant refactoring is needed before it can be replaced. Good software design takes future replaceability of functionality into account.

Decisions

Only a small part of software development is about editing code, while a very large part is about fast, effective, and coherent decision making at all levels of the organization. The sheer number of decisions to be taken implies that not all decisions can be made at the top. A large degree of decentralization is needed, while coherence across the organization is safeguarded by shared goals that are set centrally:

• At the engineering level, teams and individuals are empowered to make fast, informed decisions about as much as possible. These decisions are constrained by shared goals.

• At the leadership level, constraints for decentralized decision making are set in terms of goals (what and when, not how). These goals must be continuously reviewed and any changes in these goals must be communicated clearly and broadly within the organization.

Litmus Test

So what are the critical questions you should ask if you want to determine whether your software evolution capability is ready for what lies ahead? After transformation comes evolution. Is your organi­zation ready for the long haul? Here are some critical questions to ask yourself:

• Does everybody know and agree on which of our software components give us competitive advantage? Are the teams assigned to these components aware of what new functionalities likely need to be added in the next six months? Are required changes to these components made diligently and without compromising their structural quality?

• How about the components that do not give us competitive advantage? Are they few and well isolated? Do we have a clear plan and timeline for removing them or replacing them with commodity components? Is that timeline shorter than three months?

• Do our engineers enjoy and master the art of continuous removal of the nonessential elements of the portfolio? Are we refactoring the codebase continuously, and in small increments, to prepare for removal of noncore components? Are we removing code at the same pace as adding code? Do we celebrate decommissioning components even when we (recently) invested blood, sweat, and tears to create them?

If your answers are affirmative, you are able to keep your software healthy, nimble, and focused on what you need to compete. If not, you will gradually lose the ability to adapt quickly to new market needs, regu­lations, and technological opportunities. The world moves on while your software falls behind.

Conclusion

In this article, we discussed the inevitability of software evolution, pressured by market factors, such as the need to innovate, reduce costs, grow, comply with new regulations, and coevolve with other systems. Under these pressures, software evolution risks being self-defeating, leading to increasing complexity and diminishing competitive power.

To evolve software sustainably, organizations must balance these external pressures with an internal drive to continuously improve the software’s internal struc­ture (refactoring) and regularly swap out functionality developed in-house for commodity components supplied by others. To achieve this balance requires decentralized design and decision-making capabilities that feed off a steady stream of data about upcoming functional needs, external technology developments, and the current condition of your internal digital assets.

In our digital economy, the capability to evolve software, fast and continuously, is key. Either your software eats the world, or the world eats your software.

References

¹Andreessen, Mark. “Why Software Is Eating the World.” The Wall Street Journal, 20 August 2011.

²Xu, William, and Adrian Cooper. “Digital Spillover — Measuring the True Impact of the Digital Economy.” Huawei and Oxford Economics, 5 September 2017.

³Lehman, Meir M. “Laws of Software Evolution Revisited.” Proceedings of the 5th European Workshop on Software Technology, Springer, 1996.

⁴Visser, Joost. “Change is the Constant.” ERCIM News, 2 January 2012.

⁵Lehman (see 3).

⁶Fowler, Martin. Refactoring: Improving the Design of Existing Code. Addison-Wesley Professional, 1999.

⁷Boulding, William, and Markus Christen. “First-Mover Disadvantage.” Harvard Business Review, October 2001.

⁸ThoughtWorks Technology Radar (https://www.thoughtworks.com/radar).