Read the Executive Summary

Vol. 3, No. 2 Printer Friendly PDF version

FORENSIC SYSTEMS ANALYSIS: A METHODOLOGY FOR ASSESSMENT AND AVOIDANCE OF IT DISASTERS AND DISPUTES

by Dr. Stephen Castell

Consider the following alarming statistics reported in a 2 August 2004 article in Computer Weekly:

More than 40% of executives ... say they have wasted money on an IT investment. Thirty-two percent claimed to have been sold the wrong product by a supplier.... The main reasons given by directors were that they did not have a clear understanding of what the business required from IT. They also experienced unexpected problems when integrating systems.... IT is the third most pressing issue after financial and legal issues for seeking external advice, but businesses are unlikely to turn to a supplier for help.... A third of respondents said friends, family and colleagues were the main sources of advice. [1]

It gets worse. Those who have been involved in litigious disputes over failed computer software projects and/or contracts readily agree that, whatever their size in terms of the financial amounts at stake, whatever the facts and circumstances of the contract between the parties, and whatever the conduct of the software development project, software construction and implementation cases present complex, interwoven technical and legal issues -- and therefore prove very costly and time consuming to unravel.

One recent example in which I served as a computer expert witness involved investigation into why a major IT outsourcing deal broke down. This breakdown gave rise to the largest software contract dispute yet seen in the English High Court. At the heart of the dispute was a failed software development and implementation project, which took place within the context of the major IT outsourcing agreement. The software development project was key to managing the outsourcing deal. It entailed the building and delivery of one new single system, priced in the tens of millions of pounds, intended to replace many legacy systems. The outsourcing arrangement was to stretch over 10 years. When it failed, it gave rise to a legal action for recovery of in excess of £200 million (more than US $349 million).

The irony is that the actual outsourcing part of the deal was largely going well. However, everything became bogged down in the software development, and, because the software implementation project was tied into the main contract, the whole contract blew up when the software development "went DISPRO," that is, became a dreaded "DISaster PROject." As a result of the failure of the software development project, the entire outsourcing deal was canned, and the IT staff who had been moved across to the service provider had to be transitioned back inhouse (to both parties' credit, this was achieved within two months).

The case made it to the English High Court in autumn 2001. It settled shortly after the trial started. As is usual, the terms of such a settlement are strictly confidential. However, I have culled several important lessons to be learned from my expert investigations on this as well as from many other major IT disputes and litigation over DISPROs during the past 15 years.

Such DISPROs occur with alarming -- even increasing -- regularity. They can happen in every type of IT project: from the simplest installation of a small and apparently straightforward business software package; through light customization of an enterprise-wide applications software suite; to implementation of a mixture of "bespoke" (meaning custom-designed) software and tried-and-tested (but perhaps heavily customized) packaged programs; right on up to a major, mission-critical, full-system lifecycle of software and systems development and/or integration: one that is totally bespoke and has gone through feasibility; business case analysis; design and development; testing; acceptance; and go-live conditions.

I suspect that it will not surprise many readers to hear that the blindingly obvious conclusion I have come to, after 15 years of analyzing this wide spectrum of IT systems implementation failures and disputes, is that almost all of these disputes can be characterized by an unwillingness or inability on the part of those involved in their negotiation, management, and execution to heed one or more of the signs summarized in the sidebar "The Eight Red Warning Flares of Impending IT Disaster." Blindingly obvious, maybe. However, it may be considered astonishing that even today, IT systems customers and suppliers alike often set sail and then remorselessly glide on toward the real and present menace of the "IT disaster rocks," ignoring the alarming red glow of these readily visible flares falling around them.

THE EIGHT RED WARNING FLARES OF IMPENDING IT DISASTER Red warning flares should go up if any of the following eight items are absent at the inception (i.e., contract and/or project initiation stage) of an IT systems development/implementation project. Each one of these eight items needs to be present, in writing, and ideally included as a formal document attached to the contract. Statement of requirements -- must be clear, comprehensive, detailed, unambiguous. Prioritization of most important requirements -- i.e., those that will have the most (serious) business consequences if not delivered. Data transfer/migration specification -- answers the what, who, and how. Is new software needed? Is there a definition for testing integrity, completeness, and accuracy of data? Capacity and performance requirements -- unambiguous and complete specification of user response times, throughput, and any other "nonfunctional" requirements under defined load conditions. System cycle -- which modules/programs will be run by whom, when, how often? What are the true critical operational needs of the business? User training -- people, needs, plans/timetable, materials, data. Acceptance criteria, plans, processes, testing, responsibilities -- must be lucid, understandable, definite, and objective. Project management -- roles, responsibilities, dependencies, and processes.

My experiences in this field have led me to develop a range of techniques for assessing and reading the "technical entrails" of failed, stalled, delayed, or generally troublesome software development projects, which these days are often a contractually uncertain mixture of "customized" software packages and "bespoke" construction.

This Executive Report presents this inquisitorial method, Forensic Systems Analysis (FSA),¹ which focuses on testing of the software in dispute. It has a proven capability in providing illuminating insights into the objective state of the software and contributing to saving time and costs in, and facilitating rapid settlement of, such technically complex disputes.

Furthermore, FSA techniques can be utilized not only in a dispute context but also to assess the fragile status and troublesome characteristics of specific problem software projects and contracts before they stall, fail, or sink into litigation -- that is, well before they head, beyond rescue, toward rapidly foundering on those IT disaster rocks. More generally, FSA can be used as a positive and rigorous litigation-sensitive software quality assurance and project management audit method for large software construction and implementation projects.

The report also presents a case study based on a fictional but nevertheless highly realistic IT dispute scenario: the case of Rich & Trusting PLC (R&T) vs. Gosh-Golly Systems Ltd. (G-GS) and the X-PACMAS implementation project.

The case study of R&T (Claimant) vs. G-GS (Defendant) is used to illustrate how Forensic Systems Analysis methods are applied to:

• Identify common technical issues arising in such IT disputes
  
  
• Arrive at a rigorous expert assessment of these issues
  
  
• Discover how the issues relate back to poor contract negotiation and project management
  
  
• Recognize early warning signs of IT project failure and how to avoid costly disputes
  
  
• Teach the steps that can be taken to avoid IT disaster projects and assist in ensuring successful IT investment
  
  

The sidebar "Expert Tips from the IT Litigation Trenches" offers some insights into achieving the best outcome if, despite all efforts, you do get embroiled in an IT legal action.

EXPERT TIPS FROM THE IT LITIGATION TRENCHES Tip #1: Don't let the dispute escalate into a legal action in the first place. If applicable, warn your insurer as early as possible of an impending dispute; the insurer may well be able to help you extricate yourself amicably and effectively from it. Consider use of an independent expert in negotiation between the parties and/or in, for example, a dispute board adjudication. If it looks as though the dispute is headed inevitably toward legal action, consider the full range of alternative dispute options before issuing proceedings, such as mediation, arbitration, and expert determination. Tip #2: Use an IT expert to best effect in assisting with litigation resolution and settlement. If, notwithstanding all your best efforts, you do get into litigation, here are a few practical things to bear in mind to help your IT expert achieve the best results for both the client and the court: Appoint an IT litigation expert as early as possible. Waiting until just a few weeks before trial to hire an expert will not give him or her adequate time to fully investigate your situation and is to be avoided. Allow your IT expert to discuss the case with counsel before pleadings are settled. Having the expert and the lawyer meet early on and try to achieve a common, harmonized appreciation of and insight into the knotty "technical-legal-factual-financial matrix," which is the hallmark of the typical complex IT systems dispute, is money well spent. There is then a good chance that the Particulars of Claim, or Defense and Counterclaim, can be settled and expressed comprehensively, clearly, and most effectively at the outset, on a sound technical basis. Avoid asking your IT expert to assist counsel with very late amendments to pleadings. I have lost count of the times I have been asked to assist counsel in dealing with the (very expensive) challenge of reworking an initial 100-page pleading extremely late in the day, when, say, I have just been frantically appointed and have barely had a chance to form some preliminary idea of what the technical issues are in a particular case. Advance planning is key! Photocopy the expert's initial reference set of all the pleadings, case disclosure documents, and so forth, onto a distinctive color paper (like yellow). It is difficult, especially when time is tight, to try to decipher all the varied comments scribbled on yet another piece of white paper. The simple change of color-coding documents (e.g., to show that the document was one originating within the actual IT project in dispute -- i.e., it is disclosed documentary evidence -- and not one produced recently, during and for the litigation, as a working/discussion paper by someone on the client, legal, or expert team -- i.e., it is by contrast confidential, privileged, nondisclosable) makes things much clearer. Make sure that the client has secured a complete, reinstallable system image of the software in dispute.

TECHNICAL ISSUES

Before we discuss the individual components of FSA, it is important to identify two key issues with which an IT expert witness is often faced in arriving at an "expert opinion" during an IT systems dispute: software quality and testing incident reports (TIRs).

Software "Quality"

The most common, and arguably most important, issue on which the computer expert is inevitably asked to give an opinion in software development or implementation cases is this: What was the quality of the delivered software, and was it fit for purpose? This raises the question of just what is meant by software quality. The ready answer from the experienced IT expert is that quality can only mean fitness for purpose in the sense of "Does the delivered software meet its stated requirements?"

Thus, "quality" of software is a concept that is essentially dependent on the specification of what the software is expected to do and how the software is expected to perform in its defined environments. In other words, the yardstick for measuring and judging whether software is of appropriate quality and fit for its intended purpose is the statement of requirements defining what is required or expected of it.

In addition, testing software against its statement of requirements is the only practical and universally accepted method of judging the quality of the software and whether or not the software is fit for its intended purpose. Consider, for example, the following actual cases:

• In a case concerning an in-store electronic funds transfer at point of sale (EFTPOS) system for a major national retailer, the crucial issue was whether or not the software supplier was likely to have fixed the many outstanding errors and have had the sys.tem ready to roll out in time for the pre-Christmas sales rush. What was the objective technical evidence of the software house's "bug find and fix" performance? Were the bugs escalating, or was the software converging onto a stable, performant system? Were, rather, the constant changes in customer specification -- as alleged by the supplier -- perhaps to blame for the delays and the inability of the software to pass a critical acceptance test?

• A case concerning a large university consortium similarly focused on the apparent inability of the software developer to present a main module of the software system in a state capable of passing formal repeat acceptance tests, as a number of faults were appearing at each attempt at "final" testing (even though three earlier main modules had been successfully developed and accepted). How serious were these faults, and were earlier faults that had been thought to have been fixed constantly reappearing? Was the customer justified in terminating the contract on the grounds of a "reasonable opinion" that the software supplier would not resolve all the alleged faults in a "timely and satisfactory manner"? Was the supplier's counterclaim for a large financial amount for "software extras" valid, and could that explain the inability of the software to converge onto an "acceptable" system?

• In another case, which concerned a real-time, computer-aided mobilizing system for a large ambulance brigade, the focus was on the response times of the software in a clearly life-or-death application. How well were the desired response, availability, reliability, and recovery targets for the software contractually defined, and what was the evidence of the system's actual performance under varying load conditions?

It may be seen from these examples that this critical focus on testing the software in dispute against its statement of requirements will need to have a different emphasis for different specific cases.

Testing Incident Reports

The IT expert witness is usually presented with large volumes of project documentation, an important element of which is the set of software testing records. Typically, these are in the form of TIRs, which can number in the several hundred to thousands or even tens of thousands for large-scale bespoke software development contracts.

To simplify, disputes often come down to something like this: The customer alleges that the TIRs represented errors in the software that were serious, incapable of being remedied, too numerous in number, or in some other way either were, or summed up to, a material breach of the contract by the software supplier, entitling the customer to reject the software and terminate the contract. The software vendor/developer, on the other hand, retorts that the TIRs did not constitute "showstopper" faults, that any and all (alleged) faults were readily technically rectifiable, and, besides, they principally arose from the many and continuous changes in specification made by the customer; therefore, the customer was not entitled to terminate and had itself repudiated the contract in so doing.

THE FORENSIC SYSTEMS ANALYSIS METHODOLOGY

FSA offers IT experts, whether they are hired by either party in the dispute (or, indeed, jointly), a number of components to assist in evaluating the case, including:

• EFLAT -- Expert's Fault Log Analysis Task (tied strongly to the important concept of "material defect")
  
  
• EAT -- "Extras" Analysis Task
  
  
• FORBAT -- FORensic Bug Analysis Task
  
  
• POLIT -- Performance, Operability, and Locking Investigation Task
  
  
• FUDDER -- FUndamental Database and DEsign Review
  
  
• PROMADET -- PROject Management And Delay Examination Task
  
  
• EVOCRAT -- EVOlution of Changes to Requirements Analysis Task
  
  

These methods, the most important of which are likely to be the first two -- EFLAT and EAT -- are presented in this section.

EFLAT

During software development or customization, defects are routinely encountered, and routinely fixed, and there is generally nothing alarming about their occurrence. For the purposes of rejection of software and termination of a software development contract, any alleged defect must therefore be assessed using a strict test as to whether it is truly a material defect, that is, as to whether "the contract cannot be considered to have been performed while this defect persists."

EFLAT, developed over the years through careful debate with many firms of instructing lawyers and learned counsel, relies on a sound and objective protocol for testing whether any given software fault, in terms of its relevance to a breach and termination of a contract, is truly a material defect.

This protocol essentially states that to be a material defect, an alleged software fault must be all of the following:

1. Of large, consequential (business) effect

2. Impossible, or would take (or have taken) a long time, to fix

3. Incapable of any practical workaround

The customer is quite properly entitled to define what is a "large," consequential business effect, and the supplier, equally, may put forward an appropriate sizing for a "long" time to fix -- each from the standpoint of his or her own business/technical knowledge and experience and in the context of the particular contract/project. Both views ought to be evidentially supportable. Both views, as well as if there is indeed a practical workaround, would be the subject of expert scrutiny and opinion.

EFLAT constitutes a careful rerunning of the appropriate acceptances tests, under expert observation, with each TIR (or fault log) raised during the test assessed according to this material-defect rule. The outcome, in the English legal system, is typically an exhaustive listing called a Scott Schedule (of Software Defects). In this Scott Schedule, each fault is particularized, stating why each was considered a breach of contract (by reference to specific, contractually defined requirements); what the consequential effect was estimated to be; what the technical time to fix was (or was estimated to be); and whether there was any practical workaround available. It allows space for the expert to give his or her independent view on all these individual elements, with, finally, an opinion as to whether the specific fault in total was (or was not) a material defect.

EFLAT is undertaken by the IT expert with a prototyping orientation, assessing first only a limited proportion of the TIRs so that experience may be gained as to how difficult the full task is likely to be, how long all the TIRs will take to assess, and whether there may be technical obstacles in, say, reproducing the exact conditions of the acceptance test corresponding to those in place when the alleged faults were originally found. Not the least of these obstacles can be the basic evidential uncertainty over whether the version of the applications software and/or the database configuration and/or the hardware and systems software environment available to the expert (perhaps months or even years later) precisely correspond to the exact image of the system involved in the litigation.

Obstacles apart, early prototyping of EFLAT enables estimates of how much time (and therefore cost) is likely to be needed to complete the Scott Schedule in its entirety, enabling clients and instructing lawyers to take a considered view as to the full extent of expert investigation to be commissioned.

EAT

In software development or implementation projects of any significant size, there are inevitably many contract variations caused by the unavoidable shift in the customer's or users' perceptions of what they require as they see the software actually being built, tested, and delivered. This specification drift or constant changes to requirements is a well-known phenomenon in almost all engineering construction disciplines. It presents a particular challenge to well-ordered project management in terms of ensuring that such variations are at all times properly documented and controlled and that both parties understand and agree on the impact on project scope, timetable, and costs that implementing all requested software changes could have.

Typically, for the software systems project that collapses and ends in dispute or litigation, the IT expert witness is asked to give an opinion on whether there were indeed changes from the originally contracted software and, if so, what the quality was of the additional software built and to what financial remuneration the supplier is entitled for providing such software "extras."

EAT comprises a methodical analysis of:

1. The contractual documentation (in particular, the statement of requirements, including any amendments or reissues thereof during the project)

2. The work records of the software engineers who did the construction of the "extras"

3. The elements of software design, source code, functionality, execution, and performance that it is alleged to have been produced as a result of all this extra work

4. The financial amount claimed, and whether or not it is consistent with items 1 though 3, and also, for example, passes a "sanity cross-check" as may be provided by assessing the "$ per delivered line of source code" software metric

This results in production of a further detailed list called a Scott Schedule (of Software Extras). Once again, a prototyping methodology is used to give the client and instructing lawyers an early indication as to the likely time and costs needed to reach a complete opinion on all items of software extras claimed.

FORBAT

Recognizing that during software development defects are routinely encountered, and routinely fixed, and there is generally nothing alarming about their occurrence, the overall numbers of such bugs (as logged by the TIRs) and the pattern of their buildup and resolution are nevertheless important indicators of the progress of software construction and testing.

Such indicators are unfortunately often misread by both the software customer and the software developer: the dramatic increase in TIRs and apparent never-ending increase in bugs during systems testing can be misinterpreted. The point is that systems testing (usually the software developer's responsibility) is meant to find bugs and fix them. If there is not a large buildup in recorded TIRs, then it is not being done properly. This phenomenon contrasts with acceptance testing (usually the customer's responsibility), where only a small number of nonserious bugs is a reasonable expectation, particularly as acceptance testing should be undertaken with the appropriate attitude of acceptance, not rejection, of the software proffered for testing.

FORBAT uses standard quantitative analysis techniques to give an objective graphical presentation of the true "bug find and fix" performance of the software house, readily understandable to nontechnical clients, lawyers, or judges. The insights that emerge from these presentations are vivid (and can come as a surprise to the parties themselves). I focus on two examples, based on a realistic software project scenario, in the case study later in this report.

POLIT

Even when the functionality provided by the software is acceptable, litigious allegations can be made over its runtime performance, perhaps expressed as "response times did not meet those stipulated in the contract." To form an expert view, a system sizing and performance model may be constructed.

POLIT uses such a model to establish, among other things, a resource usage profile for key transactions used in testing scripts in order to:

• Give a measure of the likely performance of a full-system workload
  
  
• Identify ways in which performance could be optimized or improved
  
  

Critical resources investigated are:

• Disk accesses
  
  
• CPU usage
  
  

Key variables examined include:

• Resources used by login processes
  
  
• Paging in of the application software the first time it is used
  
  
• Impact of different cache sizes
  
  
• Impact of different batch sizes
  
  
• Overhead of menu navigation when repeating the same process
  
  
• Effect of spreading the database
  
  
• Effect of shadowing (using two identical copies of the database)
  
  

POLIT can become deeply technical and abstruse, and it is a challenge to the IT expert to explain the methodology, the model, and the conclusions it delivers in a way that is understandable to lawyers and to the court. However, when that challenge has been met, the findings arising can also, once again, be highly illuminative and may even allow all parties involved in the dispute to carry out their own what-if exercises using the expert system sizing and performance model provided.

FUDDER

Often there are allegations of fundamental software design flaws. This task assesses a range of accepted software engineering design parameters, and their documentation, to give an opinion on the completeness, correctness, and robustness of the software, database, and communications architectures in terms of their suitability for building, testing, and implementing a system meeting all required contractual obligations.

PROMADET

Slippage in project schedules is extremely common in large software development projects. The expert is often asked to answer the question: Who was to blame for the delays? This task examines Gantt charts and the like, drawn and redrawn throughout the project, together with associated project management documentation (e.g., minutes of project board/committee meetings) and work records, as well as the "fossil record" of the evolution of the construction of the software source code itself.

It should be noted that it can be difficult to win a case only on an allegation of delay. The very meaning of "delay," and the evidence and reasons for its occurrence, is usually not easy to determine, or present clearly, for any software project that has gone on for several years. Sometimes the best that can be said is only that "everyone was equally responsible," which is not a particularly helpful opinion for an expert to give!

EVOCRAT

As I have said, constant contract variation caused by specification drift is a common experience in most sizeable software projects. The expert is asked to give his or her view as to what all such changes amount to in terms of the questions: What was the final statement of requirements, and what was the detailed contractual specification at the end of this process of change? EVOCRAT assesses all documents purporting to state new or changed requirements and consolidates them into a unified specification.

In my experience, the findings of such FSA investigations usually permit the IT expert to arrive at an objective and compelling independent expert opinion for the mediator, tribunal, or court on all the technical issues in dispute/at trial for an IT case of this nature. Let us now see how they may be applied in an actual case study and what insights are delivered.

CASE STUDY

For the purposes of keeping within the confines of a report of this length, only the findings of the FORBAT task are illustrated. IT disputes, when they get to litigation, are going to be determined by lawyers, in a court of law, and not by technical practitioners. The most important talent of an IT expert witness, therefore, is the ability to express his or her findings, insights, conclusions, and opinions in good, clear, compelling English -- and not buried unfathomably in obscure technical analyses, tables, graphs, formulas, and so on. Beware the IT expert who seems comfortable only when endlessly producing Excel spreadsheets! The savvy expert adjusts a well-known saying to bear in mind that "A picture obscures a thousand words," knowing that the court will fundamentally decide the case on (thousands of) words, not a few pictures.

Having said that, FORBAT graphs are so powerful in "speaking for themselves" that I have no hesitation in using FORBAT analyses as illustrative examples for this report (see Appendix C).

A Typical IT Case Scenario

In what follows, any resemblance to persons or companies, living or dead, is unintentional and coincidental.

A not uncommon scenario for a computer software implementation action in the English High Court is summarized in Appendix A. The key kickoff document in any such action is the Claimant's Particulars of Claim (PoC); for this case study, the PoC sets out the alleged contractual breaches as in" Appendix A under "5. The Pleadings."

The principal specific complaints concerning the software/system made by R&T, the Claimant in this case, are particularized and set out in a 300-item Scott Schedule of X-PACMAS System Defects to the PoC. This was drawn up after preliminary investigations by Harry Keene, R&T's appointed IT expert witness. Mr. Keene undertook an EFLAT analysis, involving the rerunning of a selection of acceptance tests and assessing several dozen TIRs raised during the customization project.

G-GS, the Defendant in this case, has appointed Dr. Albert Wise as its IT expert witness. In line with a recent court direction, the experts agree to meet "to identify and define the software and other material that the experts need to examine, what system testing should be done, and how the results should be presented to the court; to isolate, and attempt to narrow, the technical issues; and, where possible, to reach agreement and take technical matters out of issue."

As a result of this meeting, and subsequent discussions with and between respective instructing lawyers, the List of Issues shown in Appendix B is agreed upon among the parties and their professional advisors.

Expert Investigations

Experts frequently encounter nontrivial difficulties in carrying out FSA analyses on the road in forming their opinions.

For example, in our R&T vs. G-GS case scenario, let's look at the following question from the List of Issues:

3. Did the System as supplied by the Defendant comply with the requirements specified in the AEC Report? If it did not comply, are the respects in which it failed to do so set out in the Claimant's Scott Schedule of X-PACMAS System Defects?

The experts are aware that there was no actual requirements specification document as such in, or appended to, the contract; furthermore, when the details of the "functionality modifications required" described in the AEC Report are examined, a distinct lack of completeness, clarity, and precision with regard to specifying G-GS's business and system requirements is discovered. Thus, deciding on the extent to which the system may or may not "comply" necessarily rests on troublesome ambiguity and uncertainty and leaves too much scope for possible reliance only on a "bleedingly obvious requirement" -- always a dangerous approach for any IT expert in a litigation.

Consider question 15 from the List of Issues:

15. What were the losses, if any, suffered by the Claimant as a result of the Defendant's alleged breaches of the Contract and misrepresentations, and to what damages do they give rise?

The experts may find that the link that should be established in the Claimant's Scott Schedule of X-PACMAS System Defects, per item of alleged software defect or deficiency, is not, it seems, capable of being particularly well spelled out by the Claimant. This link, or "causal chain" as lawyers refer to it, has (and should in theory always have) the following key connected elements:

• Obligation -- what was the contractual obligation, and how is it evidenced?
  
  
• Breach -- was there a breach of that obligation, and how is it evidenced?
  
  
• Liability -- who was liable for that alleged breach, and how is that evidenced?
  
  
• Causation -- how did that breach cause what outcome or results, and how is that evidenced?
  
  
• Consequences -- why, when, and where were the alleged associated business consequences suffered?
  
  
• Quantum -- how much money in business losses, damages, etc., was the product of such business consequences?
  
  

In this case, putting aside any difficulties of which the above illustrations are examples, Dr. Wise, on instruction from the Defendant's lawyers, focuses on the following question from the List of Issues:

12. Was the Claimant entitled to reject the System? Did the Claimant give adequate notice and opportunity to repair?

Dr. Wise carries out some FORBAT analyses on the many hundreds of TIRs logged in the G-GS project control documentation (in this case scenario, Tracey Blemish, G-GS software implementation manager, for all her faults in the eyes of Len Loutish, the R&T IT systems manager, has at least been diligent enough in her role to establish and maintain reasonably complete project records).

Dr. Wise produces two standard FORBAT graphical representations revealing the "bug find and fix" performance of Ms. Blemish's G-GS software implementation team, as shown in Figures 1 and 2 in Appendix C. Based on these clear and objective findings, he delivers the opinions as shown in the extracts from his Expert's Report presented in Appendix C.

These FORBAT analyses, and the illuminating insights that emerge from the graphical findings that they deliver, could readily have been produced during the project itself by any competent IT professional trained in FSA techniques. In particular, they could have been used as an evidential basis for rationally persuading R&T that to "lose patience" and terminate the project, alleging that G-GS was incapable of delivering a contractually compliant system within a reasonable time frame, was probably not going to be objectively justifiable and could prove to be a foolhardy decision. This might have saved the project, resolved the dispute, and avoided litigation entirely.

AVOIDING IT DISASTERS

In this section, we take a look at how to avoid IT disasters and how trial issues relate to poor contract negotiation and management.

The Five Phases of Software Supply

Consideration of the technical issues arising during the trial of the R&T vs. G-GS software implementation dispute indicates that problematic features occurred in five major phases of the X-PACMAS implementation project. From my experience, I suggest these problems are rather generally endemic characteristics of software procurement, development, and supply processes, as follows:

1. Selling software:

• Pre-contractual (mis)representations are often made ad libitum by salespersons.

• Inadequate requests for proposals (RFPs) or invitations to tender (ITTs) frequently occur, even after extensive and costly preparation phases, involving big-name consultancies or other advisors.

• These advisors do incomplete evaluation/demonstration in software selection.

• The IT "innocence" of customers regularly results in reliance on vendor expertise.

2. Contracting for supply of software:

• Lawyers may not understand software specification, construction, and acceptance processes, nor utilize independent IT expertise in drafting contracts.

• Vital clauses or schedules are often completely missing from (or, at best, poorly defined in) the contract, for example:

  • Obligations (e.g., requirements definition)

  • Project processes/conduct/management provisions

  • Monitoring and control (e.g., meetings, reports, provisions for corrective action)

  • Testing and acceptance

  • Live/parallel running (e.g., "go live" conditions)

  • Uncorrected problem handling/dispute escalation

  • Clear contract termination criteria/"code"

  • Dispute resolution methodology (e.g., negotiation, dispute board adjudication, mediation), expert determination, arbitration, litigation

• The courts are increasingly casting doubt on the "reasonableness," or effectiveness, of limitation of liability clauses in software contracts.

• Tortious allegations are becoming more common/extreme, with increasing sums being claimed for negligent/fraudulent/pre-contract misrepresentations and consequential damages/losses.

• There has been an alarming growth in large financial quantum of claim elements for loss of profits due to "failure to meet a business target" allegedly caused by faulty system installations, with sums involved out of all proportion to the original contractual price for the system to be supplied.

• Tortious claims, if upheld by the courts, are not restricted by contractual limitation clauses (if they themselves survive unscathed).

• How precisely do you either prove or disprove use of "reasonable skill, care, and diligence" by a software or other IT professional?

3. Designing software:

• Consider sizing and performance models. Are these developed/used anymore?

• Beware of "incremental design." This is great for demonstrating that no one can subsequently tell what constitutes a contractual "extra."

• Consider class models, data flow diagrams, physical/logical data models, and design architectures. Could a formal FSA fundamental database and design review be carried out on technical documentation, and what would it reveal?

4. Building software:

• Software metrics that objectively help to prove a software developer's (in)competence cannot be derived if basic software construction (particularly systems testing) data is never collected.

• Software designers and builders generally have no experience with and/or do not understand the legal concepts of "material breach" or "material defect," as well as the importance of the consequential effect of a bug, as contrasted with the technical difficulty or time to fix a bug. The same goes for customers, but perhaps even more so.

5. Installing and implementing software:

• Is time itself ever really "of the essence"? Are implementation schedules and Gantt charts nothing more than the well-known myths of project management?

• Change control and the tracking, and documentation, of changes in requirements is an important activity that is "never done properly."

• Acceptance testing should perhaps more often be called acceptance contesting. There is frequently a lack of understanding of or agreement on the meaning of an "incident report" or "error log," or on what process is supposed to be a contractually sound one to eliminate "faults found during acceptance procedures" and what the standard or threshold should be to entitle the customer to reject rather than accept.

• Legally, you may need only one bug to show a material breach of contract; on the other hand, a thousand bugs do not necessarily constitute any automatic breach -- if, for example, they are all reasonably/readily fixable; on the third hand, lawyers are attracted to the "death by a thousand cuts" argument.

Why Should I Worry?

When litigation over a software supply contract occurs, the allegations and counterallegations and the claims and counterclaims arrive thick on the scene and become expressed in legal pleadings (i.e., in lawyer-moderated language) in a manner, style, and density that presents a formidable (and expensive) management challenge for the nonlawyer businessman and IT professional to address comprehensively and convincingly. This can suddenly become a very practical and costly worry when/if legal proceedings are launched and the unfamiliar "forensic methodology" of the civil justice system suddenly imposes itself on the litigants.

Some features common to many such cases are noted below:

• Both customer and supplier often think, wrongly, that only "light customization" of an "existing software package" is to be involved in the "implementation project."

• At the point where the project breaks down, and a contract termination/repudiation happens, there is a large proportion of "software work in progress" so that the ability of either side to present evidence of a (delivered) software product's "fitness for purpose" (or otherwise) becomes difficult; this work-in-progress characteristic is especially evident in "customized software package" contract litigation.

• Although arbitration may often be chosen as the mode of dispute resolution (in general reckoned to be simpler, cheaper, and quicker than court litigation), the individual party costs and timescales can, in either litigation or arbitration, still run to hundreds of thousands of dollars and take months or years to reach a closure, with perhaps a settlement reached just before the hearing or trial -- not an unusual scenario and one that still does not avoid large costs.

• There may be a hidden agenda, unconnected (time) pressures, or something else entirely that, in truth, causes these contracts to be terminated. Such considerations frequently have nothing to do with any merits of the arguments as to quality of the software; technical performance of the software engineers in building/modifying code and finding and fixing bugs; capability of the project management (on either side); setting/meeting of deadlines; and all the other things that are typically held up as the reasons for the failures, faults, claims, and counterclaims in the actual legal pleadings. The original contract, and the limitations on possible remedial actions available to the parties under that contract, are usually unable to address or accommodate such agenda, so that termination and litigation become the only route possible.

• Such hidden agenda ideas aside, the overriding reasons why the relationship between the parties break down may fundamentally be (1) a loss of confidence and (2) a lack of creative thinking or experience in breaking out of the escalating DISPRO dispute situation in which both parties find themselves. Again, typical software contracts seem to find it difficult to accommodate such (eminently foreseeable!) features of software development and implementation projects. This may perhaps have something to do with the fact that most software contract lawyers, whether "IT lawyers" or not, have little practical project management appreciation, training, or experience.

SOME CONCLUSIONS

Given that the key to arriving at a useful and helpful expert opinion in complex software contract cases is the testing of the software in dispute, in my experience the techniques and findings of the Forensic Systems Analysis methodology can provide extremely strong, objective support in pursuit of such litigation, mediation, or arbitration and may readily assist in resolving and/or settling such disputes.

Better yet, applying some of these rigorous FSA techniques during the conduct of a systems implementation project, before it starts to "go DISPRO" and head uncontrollably toward the IT disaster rocks, I believe can provide illuminating and dramatic insights into the ongoing status of the software and project and contribute mightily to IT disaster avoidance.

As a matter of good project and contract management, software developers/vendors and IT systems suppliers, their customers (whether in the public or private sector), and their legal advisors could therefore usefully adopt a new attitude, or methodology, designed to answer at any moment such (unpalatable) questions as:

• If this software design/construction/implementation job/project were terminated today (whether I wish for that to happen or not), and a court was asked to make a judgment as to the quality of the software development and project management procedures and deliverables that are in progress (in accordance with the contract with which all parties are supposed to be complying and whether the job/project/work is currently completed or not), what would that court's decision likely be?
  
  
• What technical or other evidence could and should readily be available to assist the court in making its determination, and how should easily understood analyses of and arguments based on that evidence be presented in a way that would inevitably lead to a compelling (because objectively justifiable) conclusion in favor of the decision I genuinely believe is the correct one the court ought to reach?
  
  

The techniques of Forensic Systems Analysis can constitute just such a methodology to assist in software quality assurance, dodge DISPROs, and avoid costly and time-consuming IT disputes.

NOTES

¹The name Forensic Systems Analysis was first coined by Larry Traynor, a senior CASTELL associate consultant working as part of the CASTELL Consulting expert team during 1991-1992 on the GEC Marconi Ltd. vs. London Fire and Civil Defence Authority case [1989-ORB-No. 1208]. It is a proprietary name to CASTELL Consulting. Forensic Systems Analysis should not be confused with "computer forensics," a more recent, and nonproprietary, expression that occurs in relation to "low-level" technical tools and techniques for obtaining or retrieving (possibly hidden) computer data evidence from, for example, computer disks in principally criminal cases.

REFERENCE

1. Huber, Nick. "Suppliers Sold Us Wrong Products, Claim a Third of Smaller Business." Computer Weekly, 2 August 2004.

ABOUT THE AUTHOR

Dr. Stephen Castell, CITP, is an internationally acknowledged independent computer expert, consultant, and project manager. He is a Medallist, BCS IT Consultant of the Year 2004. Dr. Castell has established a reputation for initiating, leading, or assisting in the building of multimillion-pound businesses in voice and data communications and broadcasting, information, and software services. As an IT expert witness in computer disputes, he has been involved in a wide range of computer litigation over many years, including the largest and longest computer actions to have come to trial in the English High Court. He is widely published, a correspondent of the Computer Law & Security Report, and a noted conference and seminar speaker.

APPENDIX A: A TYPICAL IT CASE SCENARIO

CLAIM No. HHJ-2006-42 (RvG)

IN THE HIGH COURT OF JUSTICE QUEEN'S BENCH DIVISION
TECHNOLOGY AND CONSTRUCTION COURT

BETWEEN

RICH & TRUSTING plc (R&T) Claimant

and

GOSH-GOLLY SYSTEMS LTD (G-GS) Defendant

1. Overview of and Background to the Contract

R&T is an established, reputable, medium-sized financial services company offering to, for example, high net-worth individuals and small private trust funds investment advice and recommendations, trade executions, and fund management services.

This project was to upgrade R&T's existing disparate computer systems (developed inhouse over a number of years, by several different people and teams, in various computer languages) by way of a more modern "unified and integrated" package implementation/installation, with a £2.5 million fixed-price standard package-purchase Contract for supply to R&T by G-GS of the latter's "Extra-Portfolio, Asset and Currency Management and Accounting System," or X-PACMAS.

The Contract called also for £500,000 worth of custom modifications, i.e., the Contract was for a Total Purchase Price of £3.0 million.

The modified system was to be installed and working by the Implementation Date, which was given in the Contract as 12 months after the Contract Date. However, one of the standard Contract terms is that "Time is not of the essence in this Contract. G-GS will nevertheless use its best endeavours to meet the Implementation Date, subject to any unforeseen circumstances or events."

The standard X-PACMAS package-purchase Contract also contains a limitation of liability clause, seeking to restrict G-GS's exposure in all circumstances to no more than the Total Purchase Price.

G-GS had (and still has) more than 20 other "happy customers" for the same original X-PACMAS standard "respected and established" software package, which is also described as "tried and tested" in the G-GS sales literature.

However, no other G-GS customer had previously had anything like the degree of customisation required by R&T (typically, previous customers had had only £50,000-£100,000 worth of custom modifications).

2. Requirements Specification and Package Selection

There was no Requirements Specification document as such in, or appended to, the Contract. A line in the schedule of costs in the Contract merely states:

"Custom Modifications......... £500,000."

A 10-page "Report on Functionality Modifications Required to meet R&T's Core Strategy and Future Business Expansion" was written pre-Contract by R&T's Project Manager, the external consultancy Arthur Enderon Consultants (the AEC Report).

In a pre-Contract letter to Ken Sharper, R&T's Finance Director, G-GS's Managing Director Bill Blander also stated "the R&T customisation will be capable of easily accommodating the needs outlined in the AEC Report. These needs will be further defined after project launch in User Workshops involving all R&T staff."

The X-PACMAS package was identified and chosen by R&T on the basis of a selection process and "feature scoring" methodology devised and managed by Arthur Enderon Consultants on its behalf. It is not clear to what extent R&T's end-user staff members were involved in this selection process, nor how thoroughly the package was demonstrated to or tried out by R&T, nor how far other existing G-GS customer reference sites were visited.

3. The Conduct of the Implementation Project

The initial X-PACMAS package installation was delayed by at least 6 months beyond the Implementation Date. The R&T customisation work needed was badly underestimated by the G-GS Software Implementation Manager Tracey Blemish.

The system when finally delivered -- Version 7 of the customised package -- was then first implemented on a "pilot" basis only. Problems with inter alia (i.e., among other things) data migration from a key legacy R&T system then bedevilled running/testing during the pilot phase. Each party accused the other of failing to take responsibility for clean (or cleaning the) legacy data. R&T complained of "many serious functional faults and deficiencies" and an "inability of G-GS to fix bugs and keep them fixed."

The working relationship between Len Loutish, the R&T IT Systems Manager, and Tracey Blemish quickly developed into a very unhelpful one, characterised by mutual mistrust, aggression, and anger. The Project Management Consultant assigned by Arthur Enderon Consultants to the project, 25-year-old Ian Vizzable, seems to have been conspicuous by his absence during these difficult times, and more generally it was, and is, not certain what was his precise contractual role and obligations in this software implementation project on behalf of R&T.

The Contract is completely silent on data migration, data quality, etc., and who should/should not be responsible for achieving it. The Contract is also silent as regards specifying "nonfunctional requirements," such as system uptime and service levels, disaster recovery standards, performance and throughput targets, and user response times.

After a month of (attempted) pilot running (during which a further release, Version 8, of the customised package was delivered), R&T "lost patience" and formally gave G-GS 30 days' notice, by way of a solicitor's letter, to "deliver a functionally complete, operationally reliable, and contractually compliant" system. When (R&T alleges) G-GS failed to do so, R&T rejected the software and terminated the Contract.

4. The Legal Action

Since termination of the Contract by R&T, several attempts have been made by both sides to reach a settlement of their dispute. A formal 2-day mediation was unsuccessful and may even have exacerbated matters. The mediator had no knowledge of IT or experience with IT disputes. He allowed some sessions of the mediation to become extended shouting matches between Len Loutish and Tracey Blemish, without being able to find a way of moving on productively. The outcome was the parties simply sinking further into "positional bargaining," with apparently no creative dispute resolution solutions on offer.

R&T is now suing G-GS for fundamental and/or repudiatory breach, alleging that the system is "worthless and useless; and contains many fundamental design and other flaws, deficiencies, and further unresolved problems." R&T is claiming more than £10 million in compensation for losses and damages, plus interest.

G-GS asserts that its last delivery to R&T, Version 8 of the customised package, is "materially functionally complete, performance, and stable," that R&T has never fully tested it with data of appropriate quality, and that, if it had, it would have discovered that any remaining (nonmaterial) failings in the system "could readily have been rectified within a reasonable time period if R&T had permitted us to do so and not wrongfully terminated the project." G-GS is counterclaiming for £1.5 million of unpaid invoices/licence fees, plus interest.

5. The Pleadings

Particulars of Claim (PoC)Defence and Counter Claim (D&CC)Reply and Defence to Counter Claim (R&DCC)Witness Statements (WSs)Forensic Accountants Reports (FARs)Expert Reports (ERs)

The breaches alleged by the Claimant in the PoC are essentially:

1. Breach of the express term of the Contract that the system would be fully delivered and operational by the Implementation Date.

2. Breach of the express terms of the Contract that the system provided would be functionally complete, be compliant with its contractual requirements, and perform/respond in accordance with the reasonable expectations of the Claimant's user staff.

3. Breach of the implied term that the system, having been sold as a "tried and tested" system, would be free of bugs when delivered and/or would generally be of merchantable quality and reasonably fit for purpose.

4. Breach of the Defendant's common law duty as information systems experts to use reasonable skill, care, and diligence in the performance of software and system design, construction, testing, delivery, and implementation, including but not limited to the tasks of capturing system requirements and user needs or expectations, data migration, and project management.

5. Negligent or Fraudulent Misrepresentation by the Defendant in promoting and selling its X-PACMAS software package as a "tried and tested" system, when its Source Code is likely to exhibit inter alia many areas of incomplete design and construction, fundamental design flaws, and inadequate testing; and when, further, the Defendant was well aware that its own HelpDesk Call-Log database of problems reported by many other customers was likely to demonstrate and reveal operational faults, over a period of many months, unreasonable both in nature and number in respect of a purportedly "tried and tested" system.

APPENDIX B: LIST OF ISSUES FOR A TYPICAL IT CASE SCENARIO

CLAIM No. HHJ-2006-42 (RvG)

IN THE HIGH COURT OF JUSTICE QUEEN'S BENCH DIVISION
TECHNOLOGY AND CONSTRUCTION COURT

BETWEEN

RICH & TRUSTING plc (R&T) Claimant

and

GOSH-GOLLY SYSTEMS LTD (G-GS) Defendant

LIST OF ISSUES

1. Does the Contract constitute the entire contract between the Parties relating to the System? If not, what else is contained in the entire contract? Has the Contract ever been validly amended? (legal issue -- other possible contractual documents include sales literature and PowerPoint presentations, pre-contract correspondence, e-mails)

2. What, if any, legal effect is to be attributed to:

   1. the operation of the System during the Pilot Phase (How representative was this of live running: quality of data, volume of data, background workload on the system, number of concurrent users?)

   2. other matters occurring before the date of the Contract (e.g., pre-contract demonstrations attend by AEC)

3. Did the System as supplied by the Defendant comply with the requirements specified in the AEC Report? If it did not comply, are the respects in which it failed to do so set out in the Claimant's Scott Schedule of X-PACMAS System Defects?

4. Was the hardware properly specified? If so, by whom? Did the Defendant or the Claimant's consultant AEC advise on the hardware configuration?

5. Were any performance requirements specified for the system in terms of response times, throughput, or batch processing times, and in each case with what background workload? If so, were they met? If not, was the performance acceptable by industry standards?

6. Did the Claimant comply with the testing procedure as defined in Section XX of the Contract? If it did not comply, in what respects did it fail to do so? Did any such failure by the Claimant disentitle the Claimant to reject the System? What part did the Claimant's consultants AEC play in this process?

7. Were the results of the testing conducted by the Claimant such as to justify the Claimant's claim that "the System could never be fixed nor work satisfactorily" (paragraph YY of the Claimant's Particulars of Claim)?

8. Did the Claimant properly report to the Defendant the results of its acceptance tests?

9. Did the Defendant and the Claimant, following receipt by the Defendant of Error Reports from the Claimant, deal with the correction of alleged defects identified in the Error Reports either adequately or at all? Were these Error Reports properly categorized as to their severity for the Claimant's business?

10. Did the Defendant's staff have the required and relevant technical and project management expertise, experience, and competence?

11. Were there one or more misrepresentations made by the Defendant prior to the date of the Contract, referred to in paragraph ZZ of the Claimant's Particulars of Claim?

12. Was the Claimant entitled to reject the System? Did the Claimant give adequate notice and opportunity to repair?

13. Was the Claimant entitled to treat the Defendant as being in fundamental or repudiatory breach of the Contract?

14. Was the System as supplied to the Claimant in serviceable condition, of satisfactory quality, and fit for purpose; and did the Defendant supply to the Claimant services with reasonable care and skill; or was there a failure of supply of such System and services in the terms described in paragraphs Z1-Z2 of the Claimant's Particulars of Claim?

15. What were the losses, if any, suffered by the Claimant as a result of the Defendant's alleged breaches of the Contract and misrepresentations, and to what damages do they give rise?

16. If the Claimant was not entitled to reject the System nor to treat the Defendant as in fundamental breach of the Contract; and if the System as supplied was in serviceable condition, of satisfactory quality, and fit for purpose (or may have been capable of being so supplied within a reasonable time), and the Defendant did supply to the Claimant services with reasonable skill and care, then what sums are due under the Contract from the Claimant to the Defendant?

APPENDIX C: EXTRACTS FROM THE EXPERT REPORT OF DR. ALBERT WISE ON BEHALF OF THE DEFENDANT G-GS

Figure 1 (work in progress) represents the cumulative number of TIRs outstanding at any point in time and is formed by subtracting the cumulative total number of resolved TIRs from the cumulative total number of reported TIRs on a day-by-day basis. A trendline has been added to the graph to indicate the overall tendency of the underlying data, but otherwise the graph is an objective portrayal of basic project statistics.

Figure 1 -- Work in progress.

From this graph it can be seen that the number of outstanding TIRs peaked at more than 200 before the release of Version 4, but that these outstanding TIRs had been addressed and resolved, with the exception of fewer than 20, by the time Version 4 was released. This is in keeping with what would be reasonably expected if G-GS were efficiently handling the "errors" reported, eliminating as many bugs as possible before releasing a new version of software.

The steep drop-off in the number of outstanding TIRs from day 300 to day 320 is an indication of a significant amount of work performed to address and resolve outstanding TIRs prior to release of a new version of the software. The inferences that can be drawn from this graph and its underlying data are that, for each major release of the software (i.e., Versions 4, 5, 7 and 8):

1. The total cumulative number of outstanding TIRs decreased steadily from more than 200, to less than 100, to less than 50 to around 45.

2. The total number of outstanding TIRs reduced sharply before each software release (notice the steep
   drops in the curve before each major release, indicating a "cleanup" process prior to each release).

3. The total number of outstanding TIRs for each major release decreased from less than 20, to less
   than 10, to less than 5, to zero.

In my opinion, these results are consistent with what is reasonably to be expected in a well-run software development project environment. It is usual in such an environment to see a buildup of reported "errors" or "bugs" during systems testing as a major release approaches. It is also usual to see a steep drop-off in the number of these bugs outstanding just before each major release.

In summary, Figure 1, in my view, illustrates that the process to "find and fix bugs" was one in which G-GS became increasingly proficient and indicates that the software that G-GS was delivering was itself becoming more and more stable.

Figure 2 (Fix time: scatter with logarithmic trendline) depicts all TIRs in scatter format.

Each individual TIR is plotted on this graph at the point in time that it was reported versus the elapsed time in days that it took to resolve it. A logarithmic trendline has been calculated based upon the distribution of the base data used, with the resulting equation indicated in the upper right-hand portion of the graph and the trendline itself superimposed on the data. This trendline indicates an overall tendency of the data points plotted.

Additionally, a "worst case trendline" has been drawn (the dotted straight line running from the upper left-hand side of the diagram to the lower right-hand side). This "worst case trendline" uses the outlying data points to provide an indication of where the trend is going from the "worst case" perspective.

Looking at the "worst case trendline," its steep slope may be noted.

In my opinion, it appears that it would intercept the X axis near project day 550, or 1½ months after the project was terminated. This extrapolated interception, constituting a modest further elapsed time in the context of a software project that had already been running for more than 18 months, corresponds to the resolution of all the longest outstanding TIRs within the data population.