Technical Debt Assessment: A Case of Simultaneous Improvement at Three Levels

by Israel Gat, Senior Consultant, Cutter Consortium

The resignation of Intrigue's CTO six weeks before v1.0 of its JavaJoe product was scheduled to ship caused quite a few eyebrows to be raised. ¹ While the resigning CTO indicated he was leaving to pursue an opportunity he could not afford to miss, the timing of his resignation gave rise to concerns about the readiness (or lack of it) of JavaJoe to be released as an enterprise application. Neither the CEO nor the venture capitalist (VC) of Intrigue had the visibility and the data they needed to assess the quality of the JavaJoe code. Various lagging indicators, such as the number of pending bugs, were available, but no one at Intrigue was able to provide "x-rays" of the code and interpret them in terms that would be meaningful to technical and nontechnical people.

This case study sheds light on two sides of the coin -- Intrigue's needs on the one hand and the way they were addressed through the technical debt assessment on the other. In the course of describing this specific engagement, the case study outlines some of the very many benefits to be had by reining in technical debt.

A MULTILEVEL PROBLEM

The JavaJoe code had been acquired by Intrigue two years before its CTO resigned. Following a period of continuing development through an outsourcing company in another country, Intrigue built the capability to develop the code in its US headquarters. At the time the confidence crisis evolved, the US development team was running to the wire to release about 200,000 lines of Java code.

While the lines of code to be released did not represent an overwhelming undertaking in terms of size, both short-term and long-term readiness of the code were hard to assess. The development team indicated it might be able to deal with the most critical bugs before the target release date, but it was not able to provide satisfactory answers to two sets of key questions:

1. Do the defects captured in the bug database represent the iceberg or just its tip? Do the captured bugs constitute 80%, 50%, 20%, or 5% of the overall number of bugs one could expect to find in this amount of code?

2. How adaptable will the code to be released be? Will it lend itself to the constantly evolving user needs that were characteristic of the market segment Intrigue was targeting?

From the point of view of Intrigue's VC, he was staring at an investment question for which he had no good data. His paramount concern was the possibility of throwing good money after bad. This concern led to his wondering whether the company should start afresh instead of fixing the code.

For the CEO, it was primarily a governance issue. He did not have adequate metrics by which to assess the quality of the software and through which to convey it to the board. As the CEO was not a software engineer, many of the quite technical answers to the questions he asked were not too meaningful for the business decisions he needed to make.

Finally, candidate CTOs that Intrigue was interviewing were not able to get a handle on the career risk involved in joining the company. The CTO who eventually came aboard made his decision largely on the basis of his strong relationship with one of the board members. On starting with Intrigue, he was quite eager to find the true state of the code at the earliest possible time.

Whether realized at the time or not, the fundamental issue for all three -- VC, CEO, and incoming CTO -- was transforming subjective opinions to objective assessment based on hard data. The resigning CTO vouched for the code. But he was not able to support that with hard data, let alone quantitative data.

PLAN FOR THE ENGAGEMENT

The recommendation of the two Cutter consultants who held preliminary discussions with Intrigue's CEO was to carry out a technical debt assessment. This assessment views deficits and defects in the code as debt. It expresses myriad complex technical details, from excessive complexity to violation of good coding practices, in dollar terms. It gives the client a simple-to-understand metric -- the dollar amount it will take to reduce/eliminate the technical debt accrued in the course of years of development. ²

Intrigue needed a quick answer to the question of how significant the technical debt was, as the code was expected to be released in six weeks. However, none of the Cutter consultants specializing in technical debt was available for an immediate onsite engagement. To respond to Intrigue 's need to get solid answers before releasing the code, the consultants devised a three-phase engagement, which interleaved on-premises work with off-premises analysis, as follows:

• Week 1 -- a day on client premises to understand business priorities, grasp its software engineering process, and collect the data needed for off-premises analysis

• Week 2 -- a day on client premises to share preliminary findings with the client and get answers to questions that the off-premises analysis surfaced

• Week 3 -- two days on client premises to present final findings, install the analytical tools the consultants were applying to the code snapshot, and train Intrigue's team to conduct technical debt analysis on its own

CARRYING OUT THE ENGAGEMENT

First Day on Premises: Week 1

Uneventful that the first on-premises day was, it raised two alarm bells:

1. The Intrigue team was not doing any automated unit testing.

2. For the capacity of the team, the number of unresolved severity 1 and severity 2 bugs was worrisome.

Given these alarm bells, the Cutter consultants were skeptical about the ability of Intrigue to release the code on the target date. However, they decided to wait to make a judgment after their intuition was supported by preliminary source code analysis.

Second Day on Premises: Week 2

The static code analysis conducted between the first and the second day on premises identified various deficits with respect to complexity, duplication, and violation of coding rules. The dynamic code analysis reconfirmed the finding from Day 1 -- Intrigue did not have any automated unit test coverage.

In addition, examination of the bug database indicated most of the reported bugs were functional in nature. The significance of this assessment was in shedding light on the intrinsic quality ³ of the code versus its extrinsic quality. ⁴ Even if the Intrigue team was able to resolve all pending bugs before releasing the code, it would not necessarily mean that the code was of acceptable quality. ⁵

Between preliminary source code analysis and examination of the bug database, the Cutter consultants recommended to the new CTO to postpone the release date.

Third and Fourth Days on Premises: Week 3

The Cutter consultants shared their final assessment with the VC, CEO and CTO. The overall technical debt was assessed at US $400,000-$500,000. While the consultants did not think the entire technical debt needed to be paid off before releasing the code, they again recommended postponing the release date.

Working with the folks "in the trenches," the Cutter consultants installed the analytical tools they were using on the customer system. In the course of so doing, the consultants made various recommendations for improving the client's development, build, and test infrastructure.

HIGHLIGHTS OF THE REPORT PROVIDED TO THE CLIENT

Findings and recommendation made by the Cutter consultants can be briefly summarized as follows:

• If the client were to release the code as is, it would probably unleash about 1,500 defects.

• About 500 of these 1,500 defects could be serious enough to stop the JavaJoe application from running or create erroneous outputs.

• Overall technical debt amounted to $400,000-$500,000. The makeup of the technical debt is given in Figure 1.

  

  Figure 1 -- Makeup of technical debt at client.

• Lack of coverage notwithstanding, all in all, the quality of the source code was not too bad. Using a scale of A to F, the consultants rated JavaJoe's code as "C."

• The consultants were concerned about the potential for errors in complex and duplicate code. They identified the specific Java classes that had an unacceptably high level of complexity ⁶ and recommended a strategy for reducing/eliminating code duplication over time.

• To prioritize the reduction/elimination of the technical debt, the consultants produced and examined numerous heat maps of the code. Figure 2 represents the map used to project error-proneness. ⁷ Based on this figure, the consultants recommended starting the payback effort with projects P1 and P5.

RETROSPECTIVE: ASSESSMENT OF THE ENGAGEMENT

The technical debt assessment at Intrigue successfully addressed three levels of concerns:

1. Strategically, it reassured the VC and CEO that the code base was worth continuing investment. In particular, the consultants were able to put things in perspective for the VC by indicating that they had seen worse in other technical debt assessments they conducted.

2. Tactically, it gave the new CTO the tools and visibility he needed to manage the code base on a day-by-day basis.

3. Operationally, it enabled the development team at Intrigue to focus on improving quality in a biggest-bang-for-the-buck manner.

Moreover, the engagement gave the CEO the metrics required for effectively governing the software process and realigning it with marketing and sales activities. As the CEO was not technical, the ability to get an aggregate dollar figure for "paying back" debt and plugging it into Intrigue's financial models was of great value to him and to the company.

In the course of conducting the technical debt assessment, the Cutter consultants identified various deficits in Intrigue's software process. In particular, requirements management and program management were flagged as lacking in rigor. The Cutter consultants suggested various methodical improvements that would make a difference in future releases.

When this case study was written, Intrigue was about ready to launch Release 1.0 of JavaJoe. The net present value of the expected revenue stream is two orders of magnitude the amount of money spent on acquiring, developing, and reducing the technical debt of JavaJoe.

ENDNOTES

¹ Intrigue represents a real company in which the technical debt assessment was successfully conducted in 2010. Intrigue is not the real name of the company. Likewise, JavaJoe is not the real name of the product.

² For more about technical debt, see Gat, Israel. "Intrinsic Governance: A Guide to Holistic Feedback." Cutter Consortium Agile Product & Project Management E-Mail Advisor, 6 May 2010; and Gat, Israel. "Technical Debt: A Unifying Metric for Governing Software Projects." Cutter Consortium Agile Product & Project Management E-Mail Advisor, 15 April 2010.

³ Highsmith, Jim. "Intrinsic Quality?" Cutter Consortium Agile Product & Project Management E-Mail Advisor, 3 July 2008.

⁴ Highsmith. See 3.

⁵ Scratching the left ear is a good metaphor for the difference between intrinsic and extrinsic quality. One can scratch the left ear using the left hand or the right hand. The functional effect is identical. The complexity of scratching with the right hand is immensely higher than with the left hand. Likewise, two samples of code can implement the same functionality while differing drastically with respect to complexity (and other attributes of intrinsic quality).

⁶ Cyclomatic complexity for various classes exceeded 20.

⁷ Note that this heat map uses size to indicate complexity and hue to indicate quality. In other words, the size of the P1 rectangle should not be confused with lines of code.

ABOUT THE AUTHOR