Forensic Systems Analysis: A Methodology for Assessment and Avoidance of IT Disasters and Disputes (Executive Summary)
Experienced observers of, and practitioners in, procurement, contracting, execution, management, and delivery of major IT systems agree that the IT profession, taken as a whole, is not getting any better at ensuring the quality and certainty of these processes. Major IT project disasters continue to occur, particularly, it seems, in the public sector, and those projects (actually the majority) that do not end in total disaster often still fail to live up to IT corporate investment expectations, coming in late, overbudget, or failing to meet business requirements (and, depressingly, often all three).
This has led to a growth in IT disputes, project abandonment, contract termination, and, sometimes, actual litigation -- an increase that shows no sign of abating. From my 15 years of experience with major IT disputes and litigation, I have culled several important lessons to be learned, including a range of techniques for assessing and reading the "tech.nical entrails" of failed, stalled, delayed, or generally troublesome IT projects. This inquisitorial method, called Forensic Systems Analysis (FSA), focuses on testing of the software in dispute.
The accompanying Executive Report details the components of FSA. It also presents a case study based on a fictional but highly realistic IT dispute scenario: the case of Rich & Trusting PLC (R&T) vs. Gosh-Golly Systems Ltd. (G-GS) and the X-PACMAS implementation project. This case study illustrates how FSA methods are applied to identify the technical issues; arrive at an expert assessment; discover how issues relate to poor contract negotiation and project management; recognize early warning signs of IT project failure; and teach how to avoid IT disaster projects.
A key issue in software disputes frequently is: What was the quality of the delivered software, and was it fit for purpose? To the IT expert, quality can only mean fitness for purpose in the sense of "Does the delivered software meet its stated requirements?"
The IT expert witness is usually presented with large volumes of project documentation, including software testing records, typically in the form of testing incident reports (TIRs), which can number in the several hundreds to even thousands.
Disputes then often end up looking like this: The customer alleges that the TIRs represent errors in the software that are serious and a material breach of the contract, entitling the customer to reject the software and terminate the contract. The software vendor/developer retorts that the TIRs do not constitute "showstopper" faults, that (alleged) faults arose principally from the changes in specification made by the customer, and that the customer is not entitled to terminate.
THE FSA METHODOLOGY
The IT expert may address these issues using several FSA components:
- EFLAT, or the Expert's Fault Log Analysis Task, is tied intimately into the key concept of material defect. During software development or customization, defects are routinely encountered and routinely fixed, and there is generally nothing alarming about their occurrence. For the purposes of rejection of software and termination of a software development contract, any alleged software fault must therefore be assessed using a strict test as to whether or not it is truly a material defect -- that is, whether it is of large, consequential (business) effect; is impossible, or would take (or has taken) a long time, to fix; and is incapable of any practical workaround. EFLAT constitutes a careful rerunning of acceptances tests, under expert observation, with each TIR raised during testing assessed according to this material defect rule.
- EAT, or "Extras" Analysis Task, reflects that in software implementation projects of any significant size, there are inevitably many contract variations caused by the shift in users' perceptions of what they require.
- FORBAT, or the FORensic Bug Analysis Task, uses standard quantitative analysis techniques to give a clear and objective graphical presentation of the true "bug find and fix" performance of the software house.
- POLIT, or Performance, Operability, and Locking Investigation Task, is used for litigious allegations over runtime performance (e.g., "response times did not meet those stipulated in the contract"). To form an expert view, a system sizing and performance model may be constructed.
- FUDDER, or FUndamental Database and DEsign Review, is used for allegations of fundamental software design flaws. This task assesses a range of accepted software engineering design parameters to give an opinion on the completeness, correctness, and robustness of the systems design and architecture.
- PROMADET, or PROject Management And Delay Examination Task, deals with the "slippage" in project schedules that is extremely common in large software development projects. The expert is often asked: Who was to blame for the delays?
- EVOCRAT, or EVOlution of Changes to Requirements Analysis Task, deals with specification drift. The constant contract variations caused by this drift give rise to the questions: What was the final statement of requirements? What was the detailed contractual specification at the end of these changes?
The findings of such FSA investigations usually permit the IT expert to arrive at an objective and compelling independent expert opinion on all technical issues in dispute/at trial for an IT case of this nature. The report illustrates how these tasks may be applied in the R&T vs. G-GS case study and what insights are delivered. In particular, the FORBAT analyses could readily have been produced during the project itself and could have been used as objectively justifiable evidence for persuading R&T not to "lose patience" and terminate the contract. This might have saved the project, resolved the dispute, and avoided litigation entirely.
AVOIDING IT DISASTERS: WHY SHOULD I WORRY?
When litigation over a software supply contract occurs, the allegations and counterallegations arrive thick on the scene, expressed in legal pleadings in a manner, style, and density that present a formidable, expensive management challenge for the nonlawyer businessman and IT professional.
Given that the key to arriving at a useful and helpful expert opinion in complex software contract cases is testing of the software in dispute, the FSA methodology can provide extremely strong, objective support in assisting with resolving and/or settling such disputes. Better yet, applying some of these rigorous FSA techniques during the conduct of a systems implementation project, before it starts to head toward the IT disaster rocks, can deliver illuminating and dramatic insights into the ongoing status of the software and project and contribute mightily to IT disaster avoidance.