Between 70 and 79 percent of legacy modernization projects fail to meet their stated goals. That statistic appears in every article on the subject, and then every article proceeds to offer the same five generic explanations: poor stakeholder alignment, budget underestimation, scope creep, vendor selection errors, and treating technology as the solution to what is really a business problem. These explanations are not wrong. They are also not post-mortems. They are diagnoses made from the outside of programs that went wrong, without access to the internal evidence that would reveal the specific, causal, addressable failure at the center of each case.
A post-mortem is a structured analysis of what happened, what caused it, and what specifically was the root cause, the failure that, if corrected, would have changed the outcome. Post-mortems matter because they distinguish between symptoms and causes. Budget overrun is a symptom. The root cause of the budget overrun was that the scope estimate was built on an incomplete understanding of the system being replaced, because the structural analysis that would have revealed the true scope was never done. Fixing the budget process does not fix the root cause. Doing the structural analysis before making the scope commitment does.
Extract the Business Logic First
SMART TS XL produces the actual program count, dependency graph, and dead code percentage before any modernization budget is approved.
FIND OUT MORE…What a Real Modernization Post-Mortem Contains
The post-mortems that appear in project closure documents and consulting retrospectives typically contain the following: a timeline of key decisions, a list of what went over budget and by how much, a section on “lessons learned” that recommends better communication and stakeholder engagement, and a conclusion that the team did their best under difficult circumstances. These documents serve organizational purposes, closing the accounting, protecting reputations, but they rarely identify the specific, technical, addressable root cause that would have changed the outcome.
A post-mortem that is useful for the next program includes:
Stated goal and actual outcome. What the program was designed to achieve, and what it actually delivered, in specific, measurable terms. Not “improve agility” but “replace the claims processing system with a cloud-native platform that reduces claim cycle time from 8 days to 2 days.” The gap between stated goal and actual outcome is the starting point for root cause analysis.
Immediate cause vs root cause. The immediate cause is what the post-mortem typically surfaces: the cutover failed, the data migration produced incorrect results, the testing environment did not represent production. The root cause is why: the cutover failed because the dependency map was wrong and 14 systems that depended on the replaced application had not been updated; the data migration produced incorrect results because the schema had 40 undocumented variants that were not in the migration specification; the testing environment did not represent production because the production environment’s configuration had not been documented.
The evidence that was missing. What specific information, had it been available at the point of the key decision, would have changed that decision? This question identifies the failure of the intelligence process, the organizational and analytical process that was supposed to produce accurate information about the system being modernized.
What the structural analysis would have found. For every modernization failure with a code-level root cause, which is most of them, the post-mortem should document what a thorough pre-project structural analysis of the existing system would have revealed: the actual program count, the dependency graph, the data schema variants, the dead code percentage, the embedded business logic. This is the counterfactual that gives the post-mortem its practical value.
Root Cause 1: Undiscovered Complexity
The pattern: The modernization team estimates scope based on documented system components. The actual system contains more complexity than the documentation describes. The scope estimate is wrong at the point of commitment, when changing it is most expensive.
The TSB Bank case (2018). TSB Bank’s IT migration moved customer accounts from the Lloyds Banking Group platform to a new Proteo4UK system. 1.9 million customers were locked out of their accounts. The migration had taken three years of preparation. The immediate cause was the cutover event itself. The root cause, as documented in the Slaughter and May independent review, was that the migration scope did not accurately reflect the complexity of customer data, the number of account types, product variants, and edge cases in the customer population. The system that was migrated from had accumulated decades of product evolution; the system that was migrated to had not been validated against the full scope of what it was receiving.
The structural analysis failure: No known account inventory comprehensively mapped the full scope of account types and product variants before the migration design was finalized. The migration design was built on what the documentation said existed, not on what the system contained.
The code-level equivalent: A COBOL-backed core banking system where the documentation describes 12 account types and the actual FD entries in the source code define 47 account types, with REDEFINES variants for each, contains more complexity than any migration design built from the documentation. The structural analysis that reads the FD entries and copybooks finds 47. The migration design built from the documentation manages 12. The gap is discovered during cutover.
Root Cause 2: Business Logic Extraction Failure
The pattern: The replacement system is designed from a specification. The specification does not capture the full business logic of the existing system. The replacement system behaves differently in cases the specification did not describe, which are cases the existing system handled correctly.
The FBI Virtual Case File ($170 million abandoned, 2005). The FBI’s Virtual Case File project was intended to replace a paper-based case management system with a digital platform. After $170 million of development, the system was abandoned without deployment. The GAO post-mortem identified inadequate requirements as the primary cause. The deeper root cause: the requirements were derived from the specification of what the system should have been, not from a systematic analysis of what the existing process actually did. Cases that were handled by informal processes, accumulated workarounds, and institutional knowledge were not captured in the requirements. The replacement would have been unable to handle them.
The structural analysis failure: No systematic analysis of the existing process’s actual behavior, the equivalent of static analysis for a software system, was performed before the requirements were finalized. The requirements were a design document, not an extraction of existing behavior.
The code-level equivalent: A COBOL program with 847 conditional branches in its EVALUATE statement, representing 847 specific business rule conditions accumulated over thirty years of regulatory changes, cannot be replaced by a system specified from documentation that describes 20 conditions. The structural analysis that counts the EVALUATE WHEN clauses finds 847. The replacement that was specified from documentation implements 20. The remaining 827 are discovered when real production data encounters conditions the new system does not handle.
Root Cause 3: Dependency Discovery Mid-Execution
The pattern: The migration plan is built on the assumption that the system being replaced has a specific set of dependencies. During migration execution, additional dependencies are discovered that were not in the plan. The dependencies are discovered at the worst possible moment, after the timeline has been committed, resources have been allocated, and course-correction is most expensive.
The Queensland Health payroll system (2010). The replacement of Queensland Health’s payroll system produced a situation in which 35,000 hospital workers were incorrectly paid, some underpaid, some overpaid, for months after the go-live. The immediate cause was the complexity of healthcare worker pay rules: shift penalties, allowances, overtime calculations, and conditions-of-employment variations that the replacement system had not correctly implemented. The root cause was that the full scope of pay rule complexity, 24,000 distinct pay rules, as it emerged, was not known at the point of project scoping. The project was approved on an estimate built from partial knowledge of the pay rule inventory.
The structural analysis failure: The 24,000 pay rules that governed Queensland Health workers’ compensation were encoded in the existing payroll system’s processing logic. Extracting and validating the complete rule inventory before designing the replacement would have revealed the scope accurately. The project was scoped without this inventory.
The code-level equivalent: A COBOL payroll program whose complexity appears manageable from the documentation but whose PROCEDURE DIVISION contains 24,000 rule conditions distributed across hundreds of paragraphs and sections cannot be replaced by a system that was designed based on a review of the policy documentation. The structural analysis that maps the complete rule inventory, every IF condition, every EVALUATE branch, every COMPUTE formula, produces the specification. The specification produced from policy documents produces the 35,000 incorrect pay packets.
Root Cause 4: Data Migration Scope Error
The pattern: The data migration is designed based on the documented schema. The actual schema contains variants, extensions, and historical data structures not present in the documentation. The migration converts the documented schema correctly. Data in undocumented schema variants fails to migrate or migrates incorrectly.
The Veterans Affairs EHR modernization ($10 billion to $37 billion lifecycle cost). The VA’s Electronic Health Record modernization, intended to replace VistA with a commercial COTS system (Cerner), has expanded from a $10 billion to a $37 billion lifecycle cost estimate, with only a fraction of the 171 planned medical centers deployed after years of effort. The GAO’s assessment consistently cites data migration challenges as a central contributing factor: patient data in VistA accumulated over decades using locally evolved schemas, data entry conventions, and clinical terminology that differed across medical centers. The Cerner system’s data model did not accommodate all variants, producing data quality issues that affected clinical care.
The structural analysis failure: The full inventory of schema variants across 171 VistA implementations, each of which had evolved locally over decades, was not complete before migration design was finalized. Migration design that does not account for the full schema inventory cannot migrate the full data accurately.
The code-level equivalent: A VSAM file system where the record layout has evolved through 40 years of REDEFINES additions, with 47 documented layout variants plus an unknown number of undocumented ones visible only in the COBOL FD entries, cannot be fully migrated by a process designed from the 47 documented variants. The structural analysis of the FD entries and program logic finds all variants. The migration designed from documentation misses the ones the documentation missed.
Root Cause 5: Dead Code Counted in Scope
The pattern: The scope estimate counts all programs in the portfolio. A significant percentage of those programs have no inbound calls from any production execution path, they are dead code. The migration effort is committed against the full count, including dead code. Time and budget are spent migrating programs that will never run in the new environment.
This root cause is less dramatic than the others, it does not produce a cutover failure or incorrect paychecks, but it is pervasive and expensive in aggregate. Industry experience consistently shows that 15-25 percent of programs in large legacy portfolios are unreachable from any production execution path. A 4,000-program COBOL portfolio typically has 600-1,000 dead programs. Migrating those programs at any cost per program represents a significant, avoidable expense.
The structural analysis failure: Program inventory counts that come from PDS member lists or CMDB entries count everything that exists. Reachability analysis that traces every production execution path from every JCL job stream identifies only the programs that actually run. The difference is the dead code scope reduction that was available but not taken.
The Common Thread: Projects That Started Without Knowing What They Were Replacing
Reading across these five root cause categories, one pattern appears in every case: the program made its key commitments, scope, timeline, approach, budget, before completing a thorough structural analysis of the system being replaced.
TSB Bank committed to a migration design before the full customer account inventory was known. The FBI committed to requirements before extracting the complete rule set from the existing process. Queensland Health committed to a project scope before the complete pay rule inventory was known. The VA committed to a migration design before the full VistA schema variant inventory was known. Organizations committing to scope estimates regularly count all programs, including unreachable ones, before dead code analysis reduces the scope.
(cite index=”44-1″>Most post-mortems on failed modernization projects point to budget overruns, scope creep, or poor vendor selection. Those are symptoms. The disease runs deeper.</cite) The disease is specific: structural analysis of the system being replaced was either not done or not done completely before the commitments that determined the program’s shape were made. Every subsequent failure traced back to the gap between what was known at the point of commitment and what was true about the system being modernized.
The Pre-Mortem: How to Run a Post-Mortem Before Failure
A pre-mortem applies post-mortem discipline before the project begins. The team imagines that the project has failed and asks: what caused the failure? The exercise surfaces the most likely failure modes before they have occurred, when they are cheapest to address.
For a legacy modernization program, the pre-mortem questions map directly to the five root causes:
For Root Cause 1 (undiscovered complexity): Is the scope estimate derived from documentation or from structural analysis of the actual system? If documentation, what is the gap between the documented inventory and the structural inventory? Has a sample structural analysis been run to estimate the gap size?
For Root Cause 2 (business logic extraction failure): Has a systematic extraction of business logic from the existing system’s code been completed? How many business rule conditions does the existing system implement? Is there a validation plan that tests the replacement against the full rule inventory, not just the documented rules?
For Root Cause 3 (dependency discovery mid-execution): Has the complete dependency graph of the system being replaced been built from code analysis? Are there programs assumed to be standalone that have inbound dependencies not visible in the documentation? Has the migration sequence been validated against the dependency graph?
For Root Cause 4 (data migration scope error): Has the complete data schema inventory, including all REDEFINES variants, all COMP-3 fields, all historical data structures, been extracted from the actual code rather than from schema documentation? Has the migration specification been built from the structural schema inventory?
For Root Cause 5 (dead code counted in scope): Has dead code analysis been performed to identify programs with no inbound production execution path? Has the scope estimate been adjusted to exclude dead code from migration scope?
If any of these questions cannot be answered from evidence, not from documentation, from interviews, from estimates, but from structural analysis of the actual code, the pre-mortem has identified a vulnerability that may become the root cause of a post-mortem.
How SMART TS XL Eliminates the Pre-Commitment Intelligence Gap
Every root cause in this post-mortem collection traces back to incomplete knowledge of the system being replaced at the point of commitment. SMART TS XL produces the structural evidence that fills that knowledge gap.
The static code analysis capability produces the complete program inventory, including programs that documentation missed, with preliminary complexity scoring for every component. The actual program count, not the documented count, is the starting point for an accurate scope estimate. For each program, the business logic inventory, every conditional branch, every calculation formula, every rule condition, is the specification that the replacement must match.
The application dependency mapping builds the complete dependency graph: every inbound and outbound dependency for every program, including the dependencies that documentation does not describe. The dependency graph is what determines migration sequencing, and dependency discovery during migration execution is the third root cause of modernization failure. Discovering dependencies before commitment removes the mid-execution surprise.
The legacy modernization analysis identifies dead code, programs with no inbound references from any production execution path. Removing dead code from the scope estimate before commitments are made reduces the scope to what must actually be migrated, not what exists in the portfolio. The scope reduction is one of the most direct financial benefits of pre-project structural analysis.
The impact analysis capability answers the pre-mortem question for every proposed change: if this program is migrated, what is the full scope of what must be updated, tested, and validated? The impact scope, derived from the structural dependency graph, converts pre-mortem risk questions from qualitative concerns into enumerated, scoped answers.
The enterprise search capability makes the structural evidence queryable throughout the program: find every program that implements a specific business rule, every dataset that contains a specific schema variant, every program that depends on a specific shared component. This query capability is what enables the pre-mortem to get specific answers rather than general concerns, and what turns the post-mortem finding “we did not know the full schema” into a specific, avoidable failure with a specific, identifiable prevention.
Conclusion: The Post-Mortem You Cannot Afford to Write
The five root causes in this analysis, undiscovered complexity, business logic extraction failure, dependency discovery mid-execution, data migration scope error, and dead code counted in scope, are not random failures. They are predictable failures of the intelligence process: the process that should have produced accurate knowledge about the system being modernized before the program committed to its shape.
They are also preventable failures. Not by better project management, not by more experienced vendors, not by more rigorous stakeholder alignment, though all of these help. Preventable by doing the structural analysis of the existing system before the scope is committed, the approach is selected, and the budget is approved. The analysis that reveals the true program count, the actual dependency graph, the complete business logic inventory, and the real data schema variants is the analysis that converts a post-mortem finding from “we didn’t know” into “we found out in time.”
The post-mortem you write three years after a failed program is expensive and painful. The structural analysis you do before the program starts is neither.
