The page comes in at 2:47 AM. The nightly balance job has abended with a data exception. The dashboard confirms what the page already said: one job, one step, one abend code, and nothing about why. The on-call engineer knows the fix will take twenty minutes once the cause is known. What follows is closer to three hours of finding the program, finding the field, finding the writers, and finding what changed yesterday, because the tools that detected the failure were never built to answer those questions.
Search-driven root cause analysis treats each of those questions as a query against an indexed model of the estate: the code, the data definitions, the dependency graph, and the change history. The symptom becomes the first search term, whether that is an abend code, a job name, a field, or an error message. Each query narrows the set of candidate causes and returns evidence that can be opened and checked, so the investigation moves by shrinking the search space instead of by guessing from memory.
Most root cause analysis guidance assumes a rich telemetry layer of logs, metrics, and traces, and it spends its effort correlating signals in time. That works when the system is instrumented to explain itself. Many production estates are not: mainframe batch, long-lived shared databases, and older services emit little beyond a failure code and a job name, and the real evidence sits in source and in the record of what changed. Incident research keeps finding that code and configuration changes sit behind a large share of production incidents, which makes the code and the change record the most valuable things to search.
Scope the Blast Radius Fast
SMART TS XL explains unfamiliar programs in plain language over verified context, with every finding linked to its source line.
FIND OUT MORE…Why Incident Time Disappears Between Alert and Code
Incident management divides into detection, triage, diagnosis, and mitigation, and the first two have been heavily automated. Monitoring detects the failure within seconds and routing sends it to the right person. Diagnosis is where the clock runs, because it is still done by an engineer who must connect a symptom to a location in code and then to a cause. That connection is a sequence of lookups: which program, which statement, which data, which callers, which recent changes. Each lookup is quick when an index answers it and slow when a person assembles the answer from source libraries, wikis, and memory. The illustrative breakdown below shows how a three-hour investigation compresses when the lookups become queries.
| Phase (illustrative) | Manual investigation | Search-driven investigation |
|---|---|---|
| Detect and page | 2 min | 2 min |
| Locate the code | 55 min | 3 min |
| Understand the logic | 40 min | 8 min |
| Find what changed | 35 min | 6 min |
| Scope the blast radius | 25 min | 3 min |
| Fix and verify | 25 min | 16 min |
| Total | 182 min | 38 min |
Notice what barely moves. The fix itself takes about as long either way, and verification still needs a human to confirm the outcome. Search does not shorten the repair. It removes the discovery cost that sits in front of the repair, which is where most of the clock was running.
What Search-Driven Root Cause Analysis Means
Search-driven root cause analysis is an investigation method in which every diagnostic step is expressed as a structured query against an indexed model of code, data definitions, dependencies, and change records. It is not a replacement for observability. It answers a different question. Telemetry tells you when and where a system degraded. Search tells you which code is responsible, what that code touches, and what changed near it. The two work best together, with telemetry supplying the symptom and timing and search supplying the location and the cause.
| Dimension | Telemetry-driven RCA | Search-driven RCA |
|---|---|---|
| Starting point | An alert or anomaly in a time series | A symptom term: abend code, job, field, message |
| Primary evidence | Logs, metrics, traces, events | Source, dependency graph, data definitions, change records |
| Core operation | Correlation across time | Structural query across code |
| Works best when | The system is instrumented | The estate is indexed |
| Weak when | Little telemetry exists | Behavior depends on runtime state the index cannot see |
| Answers | When and where it degraded | Where in the code, what it touches, what changed |
The Seven-Rung Search Ladder
The method is a ladder because each rung depends on the one below it. A query at the top is only as good as the answers beneath it, and skipping a rung is the most common way an investigation drifts into guesswork. The first rungs convert a symptom into a location and a logic path. The middle rungs bring in dependencies and change history, which together produce the shortlist of likely causes. The upper rungs test the shortlist against hard evidence and then widen the view to everything else the cause may have touched. The final rung makes sure the next engineer starts further up the ladder than this one did. The table summarizes the climb, and the sections after it give each rung in turn.
| Rung | Name | Question | Query shape | Evidence returned |
|---|---|---|---|---|
| 1 | Locate | Where is the failing code? | Symptom term to artifact | Program, library, job, step |
| 2 | Understand | What does it do at the failure point? | Artifact to logic | Statement, control flow, fields involved |
| 3 | Scope | What feeds and consumes it? | Artifact to dependents and data | Callers, writers, readers, datasets |
| 4 | Change | What changed inside that scope? | Scope to change records | Promotions and edits in the change window |
| 5 | Confirm | Does the evidence support the hypothesis? | Hypothesis to dump or log | Matching bytes, values, or messages |
| 6 | Widen | What else is affected by the same cause? | Cause to consumers | Silent failures and downstream files |
| 7 | Record | How is the next one prevented? | Cause to preventive control | Linked record, gate, saved query |
Rung 1: Locate
The first query turns whatever the alert provides into a concrete artifact. An abend names a program and an offset, a failed job names a step, and an error message names a module or a table. Searching for that name returns the program, the library it lives in, and the jobs that run it. The target for this rung is minutes, not an hour, and the metric worth tracking is how long it takes to reach the first correct artifact.
Rung 2: Understand
With the artifact found, the next query asks what it does at the point of failure. A compile listing maps the failing offset to a statement, and the search layer supplies the surrounding control flow and the fields the statement touches. The aim is to read only the relevant path, not the whole program. A forty-year-old program should not need an expert to read four thousand lines before the engineer knows which five matter.
Rung 3: Scope
Scope is where search separates itself from reading. A where-used query on the fields in the failing statement returns every program that writes them, every program that reads them, and the datasets in between. The result is the candidate set for the cause, because a data exception in one program is almost always produced by a different program upstream. Without this rung, investigations stay inside the program that failed and miss the program that caused the failure.
Rung 4: Change
Change records mean little until they are scoped. A raw list of every promotion in the last week is noise. The same list filtered to the artifacts returned by the scope query is a shortlist, often of one or two entries. The window for the filter depends on the job’s cycle, and the next section explains why the most recent change is not always the right place to start.
Rung 5: Confirm
Search produces hypotheses, and hard evidence confirms them. The dump, the log record, or the failing input either contains the signature the hypothesis predicts or it does not. Skipping this rung is how a plausible but wrong cause gets fixed while the real one waits for the next run. Treat every search result as a lead until a byte, a value, or a message agrees with it.
Rung 6: Widen
A cause rarely produces one symptom. The same upstream change that crashed one consumer may leave another consumer printing blanks or writing wrong totals without any abend at all. The widening query follows the cause outward through its consumers and finds the failures that did not page anyone. This is the rung that prevents a fixed incident from reappearing a day later as a data quality complaint.
Rung 7: Record
The final rung writes the finding in a form the next investigation can use. A record that says only that a job failed because of bad data teaches nothing. A record that links the cause to the source artifacts, the change that introduced it, and the check that would have caught it turns a one-time repair into a reusable pattern. Repeat incidents then begin several rungs up the ladder.
A Worked Incident: S0C7 in the Nightly Balance Job
The log below follows the 2:47 AM page through the ladder. The figures are illustrative, but the sequence is how these investigations unfold. The failing statement is an addition that includes a packed decimal adjustment amount.
02:47 Page: NIGHTBAL STEP030 ABEND S0C7 in BALCALC
02:49 Q1 search BALCALC: library, job NIGHTBAL step STEP030 located
02:52 Compile listing: offset 0004A2 maps to line 1187 (COMPUTE using ADJ-AMOUNT)
02:55 Q2 where-used ADJ-AMOUNT: written by ADJLOAD, read by BALCALC and ADJRPT
03:01 Q3 changes in scope since last good run: ADJLOAD promoted 18:10 (CR-4471)
03:06 Q4 explain ADJLOAD change: new 'ZR' branch initializes the whole record to SPACES
03:09 Dump check: ADJ-AMOUNT holds X'4040404040' on the failing record, hypothesis confirmed
03:12 Q5 widen: ADJRPT also reads ADJ-AMOUNT, prints blanks, no abend; 2 downstream files affected
03:25 Fix applied: numeric fields initialized to zero in the 'ZR' branch; rerun from STEP020
The failing statement and the change that caused it sit in two different programs, which is exactly why the where-used query matters.
cobol
* BALCALC, line 1187: fails when ADJ-AMOUNT is not valid packed data
COMPUTE WS-NEW-BAL = IN-BALANCE + ADJ-AMOUNT.
* ADJLOAD, promoted 18:10 (CR-4471): the new branch
IF ADJ-REASON-CD = 'ZR'
MOVE SPACES TO ADJ-RECORD
MOVE WS-ACCT-NO TO ADJ-ACCT-NO
WRITE ADJ-RECORD
ELSE
PERFORM BUILD-ADJUSTMENT
END-IF.
In EBCDIC a space is hexadecimal 40, so a packed field filled by a group move of spaces holds bytes that are not valid packed decimal. The next arithmetic statement that touches it raises the data exception. The engineer who only had the abend code and the job name would have started at BALCALC and spent the first hour reading its logic. The engineer who searched from the field reached ADJLOAD in six minutes and the confirming dump in twenty-two.
Change Lifetime: Why the Last Deploy Is Not Always the Cause
Incident analysis often begins with a rule of thumb: the thing that changed right before the failure is the likely cause. In continuously running services that rule is a good first guess. In batch estates it breaks down, because a change does not take effect when it is promoted. It takes effect when the code that contains it next runs. A change to a month-end job can sit dormant for weeks, and a change to an annual regulatory job can wait nearly a year.
Incident-correlation research captures this with the idea of a change lifetime: each type of change has an expected delay before it can cause an incident and a point after which it is unlikely to be the cause. For search-driven analysis the practical translation is a change window defined by the failing job’s cycle, not by the clock.
| Change type | When it can cause an incident | Window to search |
|---|---|---|
| Code change in a daily job | At the next scheduled run | Since the last successful run |
| Code change in a month-end job | At the next month-end run, up to about 31 days later | Since the last run of that cycle |
| Code change in a quarterly or annual job | At the next cycle, up to a year later | Since the last run of that cycle |
| Copybook change | When each including program is recompiled and promoted, which can differ by program | Since the copybook edit, per includer |
| Data or reference table change | At the next read | Since the last good run |
| Parameter or JCL change | At the next job start | Since the last good run |
The window feeds directly into the change rung. The query becomes: show every change to the artifacts in scope between the last successful run of this job’s cycle and now. That one filter is what separates a useful shortlist from a week of unrelated promotions.
Query Patterns for Common Symptoms
Recurring symptoms deserve recurring queries. Teams that write their first two queries down for each common symptom stop reinventing the investigation every time the same code fires. The patterns below pair classic mainframe symptoms with modern ones, since many estates have both.
| Symptom | First query | Second query | Typical root cause class |
|---|---|---|---|
| ABEND S0C7 (data exception) | Program and the fields in the failing statement | Where-used for each numeric field, focusing on writers | Invalid packed data from upstream, uninitialized group |
| ABEND S0C4 (protection exception) | Program, tables, and subscript fields | Callers that pass the changed parameters | Subscript out of range, linkage mismatch |
| ABEND S806 (module not found) | Program name in the failing step | Library references in the job and recent promotions | Missed promotion, wrong library |
| SQLCODE -811 | Program and the SELECT INTO statement | Table and key definitions, recent key changes | Duplicate rows, lost uniqueness |
| SQLCODE -305 | Program and its host variables | Column nullability changes | New nullable column, missing indicator variable |
| Wrong totals, no abend | The field feeding the total | Writers and recent logic changes | Silent data or logic change |
| Java service errors after a deploy | Endpoint and exception | Calling programs and consumed data contracts | Contract drift |
Building the Search Layer Before the Incident
An index built during an incident is an index built too late. Search-driven analysis only works at 2:47 AM if the model already exists, is current, and has been used before. The checklist below describes the minimum state for the practice to pay off.
- Every program, copybook, JCL member, and schema in production is indexed, including orphaned and unreferenced code
- The dependency graph is rebuilt on every promotion, not on a weekly schedule
- Where-used, caller, and downstream-consumer queries return in seconds
- Change records can be filtered by the set of artifacts a search returns
- The mapping from failing job to program to library is queryable
- On-call engineers have read access and have used the tools outside an incident
- Saved queries exist for the ten most frequent symptoms
- Incident records link to source artifacts, not only to job names
Freshness deserves emphasis. A dependency graph that lags production by a week will confidently return the wrong callers, and an engineer who trusts it loses more time than one who had no index at all.
Measuring Whether It Is Working
A practice that is not measured drifts back to habit. Four measures capture whether the search layer is shortening diagnosis, and each maps to a rung of the ladder.
| Metric | Definition | What it reveals |
|---|---|---|
| Time to first relevant artifact | Alert to the first correct program or dataset | Effectiveness of the locate rung |
| Time to scoped blast radius | Alert to a verified list of affected consumers | Effectiveness of scope and widen |
| Repeat-incident rate | Share of incidents matching a recorded earlier cause | Whether the record rung is feeding learning |
| Linked-cause rate | Share of root cause records tied to source artifacts | Quality of evidence in the incident record |
When Search Points the Wrong Way
Search narrows the space and does not guarantee the answer. Four situations produce confident but incomplete results, and each is worth naming in the runbook.
- Dynamic calls. A program that calls another by a name held in a variable hides that dependency from a purely static view. Results should be flagged where dynamic calls exist in scope.
- Generated or copied code. Code produced at build time, or copied between libraries, can diverge from what the index holds.
- Runtime configuration. Behavior driven by parameters, environment, or dependency injection may not be visible in source at all.
- Multiple contributing factors. Many incidents combine a latent defect with a triggering change. The goal is the point in the causal chain where a targeted change would have prevented the incident, not the first link found.
The safeguard is the confirm rung. Every hypothesis from search is tested against a dump, a log, or a failing input before anyone calls it a root cause.
Frequently Asked Questions
What is search-driven root cause analysis? It is an incident investigation method where each diagnostic step is a structured query against an indexed model of code, data definitions, dependencies, and change records. The symptom is the first query, and each answer narrows the candidate causes until evidence confirms one.
How is it different from observability-based root cause analysis? Observability correlates logs, metrics, and traces in time to show when and where a system degraded. Search-driven analysis queries source and structure to show which code is responsible and what it touches. They answer different questions and are strongest together.
Can it work for mainframe batch jobs that have no traces? Yes, and that is where it adds the most. Batch estates usually emit an abend code, a job name, and a dump. Those are enough to start the ladder, because the code, the data definitions, and the change records carry the evidence that telemetry lacks.
Does AI replace the search? No. AI is most useful on top of a verified result set: explaining an unfamiliar program, summarizing a diff, or drafting the incident record. Without the structured search underneath, an AI answer has nothing reliable to be grounded in.
How SMART TS XL, ASK AI, and the MCP Connector Support Search-Driven Root Cause Analysis
SMART TS XL supplies the indexed model that the ladder depends on, and its ASK AI and MCP Connector capabilities decide how fast an on-call engineer can climb it and from where.
Enterprise search across codebases covers the locate and understand rungs. An abend code, job name, field, or program name returns the artifact, its library, the jobs that run it, and the control flow around the failing point. Because the search understands COBOL, JCL, and the relationships between them, a query on a field returns readers, writers, and redefinitions, not every file that happens to contain the string. Application dependency mapping powers the scope and widen rungs by returning callers, called programs, shared copybooks, and dataset flows from a model rebuilt as the estate changes. Structural impact analysis extends the same graph to consumers beyond the failing program, which is how the silent failures that never paged anyone get found. Static code analysis supplies the structural facts used during the record rung, such as the pattern that would have caught the defect before promotion.
ASK AI for incident triage puts natural language over that verified context. Instead of an empty prompt box, the engineer works from a scoped result set of the actual programs, copybooks, tables, and jobs involved, and asks questions such as which programs apply a rule, how a job chain behaves, or what an unfamiliar program does in plain language. Findings stay linked to source lines, so the engineer can open the line behind any statement and confirm it, which keeps AI output inside the confirm discipline. Saved and shared prompts turn the symptom query patterns above into reusable runbook entries that any on-call engineer can run.
The SMART TS XL MCP Connector brings the same intelligence into the assistants an organization already approves, including ChatGPT, Claude, Copilot, Gemini, and Grok. An engineer paged from a phone can ask an assistant to explain the failing program and everything it touches, and the assistant calls tools for source retrieval, control flow, and cross-reference reports. The answer comes back grounded in SMART TS XL analysis, for example a program that reads two DB2 tables, is called by four batch jobs, and shares three copybooks with another cycle, with each step linked to its source line. Access is read-only, authenticated, and role-based, so the approved assistant works within the permissions the organization has granted.
| Ladder rung | What the engineer needs | Capability that answers it |
|---|---|---|
| Locate | Program, library, and job from a symptom term | Enterprise search |
| Understand | Control flow and plain-language explanation of the failing path | Enterprise search and ASK AI |
| Scope | Writers, readers, callers, and dataset flows | Application dependency mapping |
| Change | The set of artifacts to filter change records by | Dependency scope from the graph |
| Confirm | Source lines to compare against the dump | Source retrieval through ASK AI or the MCP Connector |
| Widen | Downstream consumers of the same cause | Structural impact analysis |
| Record | The structural pattern that prevents recurrence | Static code analysis |
Two boundaries keep expectations accurate. SMART TS XL analyzes source and structure. It does not read live logs, dumps, or metrics, so the confirm rung uses those artifacts and the observability stack alongside it. And the change history comes from your version control or change management records, which SMART TS XL’s dependency scope helps you filter and does not replace. Used together, telemetry detects, search locates and scopes, evidence confirms, and the record carries the lesson forward.
Dashboards Tell You Where It Hurts. Search Tells You Why.
The 2:47 AM page will keep coming. Production estates fail for ordinary reasons: a branch added in a hurry, a field initialized the wrong way, a copybook edited on a Friday. What separates a three-hour night from a forty-minute one is not the repair, which takes about the same either way. It is how quickly the engineer can move from an abend code to the field, the writers, the change, and the blast radius.
Search-driven root cause analysis makes that movement a procedure instead of a feat of memory. It starts from the symptom, climbs seven rungs, respects the change lifetime of the job’s cycle, confirms every lead against hard evidence, and leaves a record that starts the next investigation higher up. Build the index before the incident, keep it current, and rehearse the queries. The cause was always in the code and the change record. Search is how you reach it before sunrise.


