Search-Driven Root Cause Analysis

Search-Driven Root Cause Analysis for Production Incidents

The page comes in at 2:47 AM. The nightly balance job has abended with a data exception. The dashboard confirms what the page already said: one job, one step, one abend code, and nothing about why. The on-call engineer knows the fix will take twenty minutes once the cause is known. What follows is closer to three hours of finding the program, finding the field, finding the writers, and finding what changed yesterday, because the tools that detected the failure were never built to answer those questions.

Search-driven root cause analysis treats each of those questions as a query against an indexed model of the estate: the code, the data definitions, the dependency graph, and the change history. The symptom becomes the first search term, whether that is an abend code, a job name, a field, or an error message. Each query narrows the set of candidate causes and returns evidence that can be opened and checked, so the investigation moves by shrinking the search space instead of by guessing from memory.

Most root cause analysis guidance assumes a rich telemetry layer of logs, metrics, and traces, and it spends its effort correlating signals in time. That works when the system is instrumented to explain itself. Many production estates are not: mainframe batch, long-lived shared databases, and older services emit little beyond a failure code and a job name, and the real evidence sits in source and in the record of what changed. Incident research keeps finding that code and configuration changes sit behind a large share of production incidents, which makes the code and the change record the most valuable things to search.

Scope the Blast Radius Fast

SMART TS XL explains unfamiliar programs in plain language over verified context, with every finding linked to its source line.

FIND OUT MORE…

Why Incident Time Disappears Between Alert and Code

Incident management divides into detection, triage, diagnosis, and mitigation, and the first two have been heavily automated. Monitoring detects the failure within seconds and routing sends it to the right person. Diagnosis is where the clock runs, because it is still done by an engineer who must connect a symptom to a location in code and then to a cause. That connection is a sequence of lookups: which program, which statement, which data, which callers, which recent changes. Each lookup is quick when an index answers it and slow when a person assembles the answer from source libraries, wikis, and memory. The illustrative breakdown below shows how a three-hour investigation compresses when the lookups become queries.

Phase (illustrative)Manual investigationSearch-driven investigation
Detect and page2 min2 min
Locate the code55 min3 min
Understand the logic40 min8 min
Find what changed35 min6 min
Scope the blast radius25 min3 min
Fix and verify25 min16 min
Total182 min38 min

Notice what barely moves. The fix itself takes about as long either way, and verification still needs a human to confirm the outcome. Search does not shorten the repair. It removes the discovery cost that sits in front of the repair, which is where most of the clock was running.

What Search-Driven Root Cause Analysis Means

Search-driven root cause analysis is an investigation method in which every diagnostic step is expressed as a structured query against an indexed model of code, data definitions, dependencies, and change records. It is not a replacement for observability. It answers a different question. Telemetry tells you when and where a system degraded. Search tells you which code is responsible, what that code touches, and what changed near it. The two work best together, with telemetry supplying the symptom and timing and search supplying the location and the cause.

DimensionTelemetry-driven RCASearch-driven RCA
Starting pointAn alert or anomaly in a time seriesA symptom term: abend code, job, field, message
Primary evidenceLogs, metrics, traces, eventsSource, dependency graph, data definitions, change records
Core operationCorrelation across timeStructural query across code
Works best whenThe system is instrumentedThe estate is indexed
Weak whenLittle telemetry existsBehavior depends on runtime state the index cannot see
AnswersWhen and where it degradedWhere in the code, what it touches, what changed

The Seven-Rung Search Ladder

The method is a ladder because each rung depends on the one below it. A query at the top is only as good as the answers beneath it, and skipping a rung is the most common way an investigation drifts into guesswork. The first rungs convert a symptom into a location and a logic path. The middle rungs bring in dependencies and change history, which together produce the shortlist of likely causes. The upper rungs test the shortlist against hard evidence and then widen the view to everything else the cause may have touched. The final rung makes sure the next engineer starts further up the ladder than this one did. The table summarizes the climb, and the sections after it give each rung in turn.

RungNameQuestionQuery shapeEvidence returned
1LocateWhere is the failing code?Symptom term to artifactProgram, library, job, step
2UnderstandWhat does it do at the failure point?Artifact to logicStatement, control flow, fields involved
3ScopeWhat feeds and consumes it?Artifact to dependents and dataCallers, writers, readers, datasets
4ChangeWhat changed inside that scope?Scope to change recordsPromotions and edits in the change window
5ConfirmDoes the evidence support the hypothesis?Hypothesis to dump or logMatching bytes, values, or messages
6WidenWhat else is affected by the same cause?Cause to consumersSilent failures and downstream files
7RecordHow is the next one prevented?Cause to preventive controlLinked record, gate, saved query

Rung 1: Locate

The first query turns whatever the alert provides into a concrete artifact. An abend names a program and an offset, a failed job names a step, and an error message names a module or a table. Searching for that name returns the program, the library it lives in, and the jobs that run it. The target for this rung is minutes, not an hour, and the metric worth tracking is how long it takes to reach the first correct artifact.

Rung 2: Understand

With the artifact found, the next query asks what it does at the point of failure. A compile listing maps the failing offset to a statement, and the search layer supplies the surrounding control flow and the fields the statement touches. The aim is to read only the relevant path, not the whole program. A forty-year-old program should not need an expert to read four thousand lines before the engineer knows which five matter.

Rung 3: Scope

Scope is where search separates itself from reading. A where-used query on the fields in the failing statement returns every program that writes them, every program that reads them, and the datasets in between. The result is the candidate set for the cause, because a data exception in one program is almost always produced by a different program upstream. Without this rung, investigations stay inside the program that failed and miss the program that caused the failure.

Rung 4: Change

Change records mean little until they are scoped. A raw list of every promotion in the last week is noise. The same list filtered to the artifacts returned by the scope query is a shortlist, often of one or two entries. The window for the filter depends on the job’s cycle, and the next section explains why the most recent change is not always the right place to start.

Rung 5: Confirm

Search produces hypotheses, and hard evidence confirms them. The dump, the log record, or the failing input either contains the signature the hypothesis predicts or it does not. Skipping this rung is how a plausible but wrong cause gets fixed while the real one waits for the next run. Treat every search result as a lead until a byte, a value, or a message agrees with it.

Rung 6: Widen

A cause rarely produces one symptom. The same upstream change that crashed one consumer may leave another consumer printing blanks or writing wrong totals without any abend at all. The widening query follows the cause outward through its consumers and finds the failures that did not page anyone. This is the rung that prevents a fixed incident from reappearing a day later as a data quality complaint.

Rung 7: Record

The final rung writes the finding in a form the next investigation can use. A record that says only that a job failed because of bad data teaches nothing. A record that links the cause to the source artifacts, the change that introduced it, and the check that would have caught it turns a one-time repair into a reusable pattern. Repeat incidents then begin several rungs up the ladder.

A Worked Incident: S0C7 in the Nightly Balance Job

The log below follows the 2:47 AM page through the ladder. The figures are illustrative, but the sequence is how these investigations unfold. The failing statement is an addition that includes a packed decimal adjustment amount.

02:47  Page: NIGHTBAL STEP030 ABEND S0C7 in BALCALC
02:49  Q1 search BALCALC: library, job NIGHTBAL step STEP030 located
02:52  Compile listing: offset 0004A2 maps to line 1187 (COMPUTE using ADJ-AMOUNT)
02:55  Q2 where-used ADJ-AMOUNT: written by ADJLOAD, read by BALCALC and ADJRPT
03:01  Q3 changes in scope since last good run: ADJLOAD promoted 18:10 (CR-4471)
03:06  Q4 explain ADJLOAD change: new 'ZR' branch initializes the whole record to SPACES
03:09  Dump check: ADJ-AMOUNT holds X'4040404040' on the failing record, hypothesis confirmed
03:12  Q5 widen: ADJRPT also reads ADJ-AMOUNT, prints blanks, no abend; 2 downstream files affected
03:25  Fix applied: numeric fields initialized to zero in the 'ZR' branch; rerun from STEP020

The failing statement and the change that caused it sit in two different programs, which is exactly why the where-used query matters.

cobol

      * BALCALC, line 1187: fails when ADJ-AMOUNT is not valid packed data
       COMPUTE WS-NEW-BAL = IN-BALANCE + ADJ-AMOUNT.

      * ADJLOAD, promoted 18:10 (CR-4471): the new branch
       IF ADJ-REASON-CD = 'ZR'
           MOVE SPACES TO ADJ-RECORD
           MOVE WS-ACCT-NO TO ADJ-ACCT-NO
           WRITE ADJ-RECORD
       ELSE
           PERFORM BUILD-ADJUSTMENT
       END-IF.

In EBCDIC a space is hexadecimal 40, so a packed field filled by a group move of spaces holds bytes that are not valid packed decimal. The next arithmetic statement that touches it raises the data exception. The engineer who only had the abend code and the job name would have started at BALCALC and spent the first hour reading its logic. The engineer who searched from the field reached ADJLOAD in six minutes and the confirming dump in twenty-two.

Change Lifetime: Why the Last Deploy Is Not Always the Cause

Incident analysis often begins with a rule of thumb: the thing that changed right before the failure is the likely cause. In continuously running services that rule is a good first guess. In batch estates it breaks down, because a change does not take effect when it is promoted. It takes effect when the code that contains it next runs. A change to a month-end job can sit dormant for weeks, and a change to an annual regulatory job can wait nearly a year.

Incident-correlation research captures this with the idea of a change lifetime: each type of change has an expected delay before it can cause an incident and a point after which it is unlikely to be the cause. For search-driven analysis the practical translation is a change window defined by the failing job’s cycle, not by the clock.

Change typeWhen it can cause an incidentWindow to search
Code change in a daily jobAt the next scheduled runSince the last successful run
Code change in a month-end jobAt the next month-end run, up to about 31 days laterSince the last run of that cycle
Code change in a quarterly or annual jobAt the next cycle, up to a year laterSince the last run of that cycle
Copybook changeWhen each including program is recompiled and promoted, which can differ by programSince the copybook edit, per includer
Data or reference table changeAt the next readSince the last good run
Parameter or JCL changeAt the next job startSince the last good run

The window feeds directly into the change rung. The query becomes: show every change to the artifacts in scope between the last successful run of this job’s cycle and now. That one filter is what separates a useful shortlist from a week of unrelated promotions.

Query Patterns for Common Symptoms

Recurring symptoms deserve recurring queries. Teams that write their first two queries down for each common symptom stop reinventing the investigation every time the same code fires. The patterns below pair classic mainframe symptoms with modern ones, since many estates have both.

SymptomFirst querySecond queryTypical root cause class
ABEND S0C7 (data exception)Program and the fields in the failing statementWhere-used for each numeric field, focusing on writersInvalid packed data from upstream, uninitialized group
ABEND S0C4 (protection exception)Program, tables, and subscript fieldsCallers that pass the changed parametersSubscript out of range, linkage mismatch
ABEND S806 (module not found)Program name in the failing stepLibrary references in the job and recent promotionsMissed promotion, wrong library
SQLCODE -811Program and the SELECT INTO statementTable and key definitions, recent key changesDuplicate rows, lost uniqueness
SQLCODE -305Program and its host variablesColumn nullability changesNew nullable column, missing indicator variable
Wrong totals, no abendThe field feeding the totalWriters and recent logic changesSilent data or logic change
Java service errors after a deployEndpoint and exceptionCalling programs and consumed data contractsContract drift

Building the Search Layer Before the Incident

An index built during an incident is an index built too late. Search-driven analysis only works at 2:47 AM if the model already exists, is current, and has been used before. The checklist below describes the minimum state for the practice to pay off.

  • Every program, copybook, JCL member, and schema in production is indexed, including orphaned and unreferenced code
  • The dependency graph is rebuilt on every promotion, not on a weekly schedule
  • Where-used, caller, and downstream-consumer queries return in seconds
  • Change records can be filtered by the set of artifacts a search returns
  • The mapping from failing job to program to library is queryable
  • On-call engineers have read access and have used the tools outside an incident
  • Saved queries exist for the ten most frequent symptoms
  • Incident records link to source artifacts, not only to job names

Freshness deserves emphasis. A dependency graph that lags production by a week will confidently return the wrong callers, and an engineer who trusts it loses more time than one who had no index at all.

Measuring Whether It Is Working

A practice that is not measured drifts back to habit. Four measures capture whether the search layer is shortening diagnosis, and each maps to a rung of the ladder.

MetricDefinitionWhat it reveals
Time to first relevant artifactAlert to the first correct program or datasetEffectiveness of the locate rung
Time to scoped blast radiusAlert to a verified list of affected consumersEffectiveness of scope and widen
Repeat-incident rateShare of incidents matching a recorded earlier causeWhether the record rung is feeding learning
Linked-cause rateShare of root cause records tied to source artifactsQuality of evidence in the incident record

When Search Points the Wrong Way

Search narrows the space and does not guarantee the answer. Four situations produce confident but incomplete results, and each is worth naming in the runbook.

  1. Dynamic calls. A program that calls another by a name held in a variable hides that dependency from a purely static view. Results should be flagged where dynamic calls exist in scope.
  2. Generated or copied code. Code produced at build time, or copied between libraries, can diverge from what the index holds.
  3. Runtime configuration. Behavior driven by parameters, environment, or dependency injection may not be visible in source at all.
  4. Multiple contributing factors. Many incidents combine a latent defect with a triggering change. The goal is the point in the causal chain where a targeted change would have prevented the incident, not the first link found.

The safeguard is the confirm rung. Every hypothesis from search is tested against a dump, a log, or a failing input before anyone calls it a root cause.

Frequently Asked Questions

What is search-driven root cause analysis? It is an incident investigation method where each diagnostic step is a structured query against an indexed model of code, data definitions, dependencies, and change records. The symptom is the first query, and each answer narrows the candidate causes until evidence confirms one.

How is it different from observability-based root cause analysis? Observability correlates logs, metrics, and traces in time to show when and where a system degraded. Search-driven analysis queries source and structure to show which code is responsible and what it touches. They answer different questions and are strongest together.

Can it work for mainframe batch jobs that have no traces? Yes, and that is where it adds the most. Batch estates usually emit an abend code, a job name, and a dump. Those are enough to start the ladder, because the code, the data definitions, and the change records carry the evidence that telemetry lacks.

Does AI replace the search? No. AI is most useful on top of a verified result set: explaining an unfamiliar program, summarizing a diff, or drafting the incident record. Without the structured search underneath, an AI answer has nothing reliable to be grounded in.

How SMART TS XL, ASK AI, and the MCP Connector Support Search-Driven Root Cause Analysis

SMART TS XL supplies the indexed model that the ladder depends on, and its ASK AI and MCP Connector capabilities decide how fast an on-call engineer can climb it and from where.

Enterprise search across codebases covers the locate and understand rungs. An abend code, job name, field, or program name returns the artifact, its library, the jobs that run it, and the control flow around the failing point. Because the search understands COBOL, JCL, and the relationships between them, a query on a field returns readers, writers, and redefinitions, not every file that happens to contain the string. Application dependency mapping powers the scope and widen rungs by returning callers, called programs, shared copybooks, and dataset flows from a model rebuilt as the estate changes. Structural impact analysis extends the same graph to consumers beyond the failing program, which is how the silent failures that never paged anyone get found. Static code analysis supplies the structural facts used during the record rung, such as the pattern that would have caught the defect before promotion.

ASK AI for incident triage puts natural language over that verified context. Instead of an empty prompt box, the engineer works from a scoped result set of the actual programs, copybooks, tables, and jobs involved, and asks questions such as which programs apply a rule, how a job chain behaves, or what an unfamiliar program does in plain language. Findings stay linked to source lines, so the engineer can open the line behind any statement and confirm it, which keeps AI output inside the confirm discipline. Saved and shared prompts turn the symptom query patterns above into reusable runbook entries that any on-call engineer can run.

The SMART TS XL MCP Connector brings the same intelligence into the assistants an organization already approves, including ChatGPT, Claude, Copilot, Gemini, and Grok. An engineer paged from a phone can ask an assistant to explain the failing program and everything it touches, and the assistant calls tools for source retrieval, control flow, and cross-reference reports. The answer comes back grounded in SMART TS XL analysis, for example a program that reads two DB2 tables, is called by four batch jobs, and shares three copybooks with another cycle, with each step linked to its source line. Access is read-only, authenticated, and role-based, so the approved assistant works within the permissions the organization has granted.

Ladder rungWhat the engineer needsCapability that answers it
LocateProgram, library, and job from a symptom termEnterprise search
UnderstandControl flow and plain-language explanation of the failing pathEnterprise search and ASK AI
ScopeWriters, readers, callers, and dataset flowsApplication dependency mapping
ChangeThe set of artifacts to filter change records byDependency scope from the graph
ConfirmSource lines to compare against the dumpSource retrieval through ASK AI or the MCP Connector
WidenDownstream consumers of the same causeStructural impact analysis
RecordThe structural pattern that prevents recurrenceStatic code analysis

Two boundaries keep expectations accurate. SMART TS XL analyzes source and structure. It does not read live logs, dumps, or metrics, so the confirm rung uses those artifacts and the observability stack alongside it. And the change history comes from your version control or change management records, which SMART TS XL’s dependency scope helps you filter and does not replace. Used together, telemetry detects, search locates and scopes, evidence confirms, and the record carries the lesson forward.

Dashboards Tell You Where It Hurts. Search Tells You Why.

The 2:47 AM page will keep coming. Production estates fail for ordinary reasons: a branch added in a hurry, a field initialized the wrong way, a copybook edited on a Friday. What separates a three-hour night from a forty-minute one is not the repair, which takes about the same either way. It is how quickly the engineer can move from an abend code to the field, the writers, the change, and the blast radius.

Search-driven root cause analysis makes that movement a procedure instead of a feat of memory. It starts from the symptom, climbs seven rungs, respects the change lifetime of the job’s cycle, confirms every lead against hard evidence, and leaves a record that starts the next investigation higher up. Build the index before the incident, keep it current, and rehearse the queries. The cause was always in the code and the change record. Search is how you reach it before sunrise.