AI Regression Detection From Crash Symbolication Data

Crash symbolication produces something rare in software engineering: a structured, version-tagged record of what a program was doing in the instant it failed. That structure is what makes it possible for AI models to watch crash data across releases and catch a regression before it spreads past the small slice of users who hit it first. Getting there means understanding a pipeline with real mechanics, not a black box: how crashes get captured, how they get grouped, how that grouping turns into a signal, and how a flagged regression gets traced back to a line of code.

Start with the raw material. A crash produces a dump file: a minidump (.dmp) on Windows, a core dump on Linux, or a.crash or.ips file on macOS and iOS. Each one captures the thread stacks, the CPU register values, the list of loaded modules, and a slice of memory at the moment things went wrong. None of that is readable on its own. Symbolication is the step that turns raw instruction pointer addresses into function names, file names, and line numbers, using platform-neutral symbol files (Chrome's Crashpad pipeline, for instance, runs on the Breakpad format). Read frame by frame, a symbolicated stack trace tells a story: the top frame is where the crash happened, and the frames below it trace the path the program took to get there. The point where application code hands off into a library or the OS is often where the actual bug lives.

None of that data is useful in isolation, either. Every crash report carries metadata: build version, OS name and version, device type, and a set of crash keys the app was tracking. That version-tagged envelope is the whole reason regression detection is possible later on. Without it, a crash is just a snapshot. With it, a crash is a data point that can be compared against every other crash from every other build.

The scale problem that makes manual triage unworkable

Automated crash reporting produces volume, and volume produces duplicates. A popular app on a bad build can flood an issue tracker with thousands of reports that all trace back to the same fault, and someone still has to sort through them. That workload isn't an occasional annoyance, it's structural: the more successful the crash reporting system is at catching failures, the worse the triage burden gets if nothing groups the reports first.

Making it worse, a single bug rarely produces one consistent stack trace. Small differences in inlining, thread timing, or memory layout can make the same underlying fault look like dozens of unrelated crashes, which breaks tools that group by exact stack location or literal call-stack match. The grouping problem isn't cosmetic. Counting how many distinct bugs exist in a given build depends directly on this.

Manual diagnosis runs into a harder wall than volume, though. A developer trying to reproduce a crash usually can't, because they don't have the user's device, data, or environment. Research and patented techniques address this gap directly, describing machine learning approaches built specifically to close it, because there's no other route to diagnosing a crash that only happens on a configuration the developer will never touch.

Then there's the failure mode that CI was never built to see. A pipeline can run green, no exit-code failure, no red build, while a regression quietly accumulates in the background and only becomes visible once it's reached a meaningful chunk of the user base. Standard CI operates on a stateless, binary pass-fail model. It checks whether the build compiled and the tests passed, not whether the last three builds have shown a slow climb in a particular crash cluster. That's a blind spot by design, not by neglect.

Crash clustering accuracy as the prerequisite for regression detection

A real regression shows up in symbolicated data in one of two ways. Either a new cluster of function-name and frame-sequence signatures appears that wasn't there in prior builds, or an existing cluster spikes in volume in a way that lines up with a specific build version. Both are detectable, in theory. But detecting either one depends entirely on having clean cluster boundaries to begin with.

Get the deduplication wrong and both directions of error become possible. A real new bug that gets scattered across several unrelated buckets never accumulates enough volume in any one bucket to trip a threshold, so the regression goes missed. Or ordinary background noise collapses into what looks like a single tight cluster, and the system raises an alarm for something that isn't actually new. Either failure defeats the purpose of monitoring.

Traditional machine learning models, Logistic Regression, Random Forest, SVM, Decision Tree, XGBoost, handle a lot of this reasonably well. Across studies, these baselines are around 0.85 to 0.86 accuracy with F1-scores in the 0.83 to 0.84 range, a stable, dependable floor. That's a stable, dependable floor. But these models struggle with anything that requires reading between the lines: long, free-text crash descriptions, implicit context that isn't captured in a handful of engineered features. They classify well on structure and miss meaning.

That gap is what pushed research toward similarity-learning approaches that bucket crashes based on stack trace similarity learned directly from the data rather than hand-built rules. This reflects a broader shift: crash bucketing is treated less like a classification problem, and more like a similarity-learning problem, which is closer to what the underlying task actually is.

Turning clustered crash data into a regression signal across releases

Once clustering is trustworthy, the detection task is to watch for a new symbolicated crash cluster that correlates with a specific build version, or watch for a spike past some threshold in a cluster that already existed. Either one, tied to a build change, counts as a regression signal.

U.S. Patent 11521086 describes a crash prediction system that categorizes every logged error, exception, or crash along several axes at once: error type, customer profile, navigation flow, device type, application version, hardware, OS, network. Of those, the application version axis is the one that turns a static classification system into a regression detector, because it's the dimension along which "before" and "after" actually mean something.

U.S. Patent 10956307 goes a layer further, describing a correlation between an operation failure spike in one development ring and a spike in a second ring. The patent defines an operation failure spike concretely: either a threshold number of failures within a threshold duration, or a threshold percentage of users hitting the same failure. That ring language maps directly onto how canary deployments actually work in production. A new build typically ships to a small slice of traffic first, while the large majority stays on the known-stable version. Watching crash clusters during that narrow canary window is exactly where early regression detection has to happen, because it's the only point in the release process where the blast radius is still small enough to matter.

LLM-based fault localization: from regression flag to specific code location

Flagging a regression is only half the job. Someone, or something, still has to find the line of code responsible, and stack traces are worse at this than most engineers assume. In one studied dataset, only 37% of the files actually responsible for a bug had their base filename show up anywhere in the stack trace. That's a real signal, worth having, but it means that in roughly six cases out of ten, the trace alone won't point at the guilty file. The root cause tends to sit upstream of the crash site, not inside it.

Older fault localization techniques were never cheap. Spectrum-based FL needs extensive test coverage data to work. Mutation-based FL needs to run the program under many artificially altered variants and compare outcomes. Both are expensive to run continuously in production. LLM-based fault localization sidesteps that cost structure entirely, because it works directly on the textual failure description, the stack trace and surrounding context, rather than requiring a live, instrumented execution environment.

Research on 492 real-world crash reports tested two ways of using an LLM for this job. The simpler one, Direct-LLM, uses the crash report's immediate context without iterative codebase exploration, and it lifted Top-1 localization accuracy from 10.6% on the original raw reports to 40.2%. The more involved one, Agentic-LLM, explores the surrounding codebase iteratively before answering, and reached 43.1%. Putting those two together, LLM-enhanced reports span a 40.2 to 43.1% accuracy range, a four-fold improvement over the unenhanced baseline.

The two approaches aren't interchangeable, though. Agentic-LLM produces stronger root-cause explanations and more actionable repair guidance, which matters most on crashes whose real cause sits well outside the visible frames. Suggested fixes generated from these enhanced reports scored around 56 to 57 on CodeBLEU, a reasonable mark for automatically generated repair suggestions. A user study with 16 participants found that enhanced reports made crashes noticeably easier to understand and resolve.

Direct-LLM is fast and cheap, but it can hallucinate a plausible-looking answer that's simply wrong. Direct-LLM is fast and cheap, but it can hallucinate a plausible-looking answer that's simply wrong. Agentic-LLM is more reliable on deep, non-obvious causes, but the iterative repository exploration costs meaningfully more compute. Neither one is a universal answer. The choice depends on how deep the bug tends to hide and how much compute budget the team is willing to spend finding out.

Data scarcity is a separate problem underlying all of this. Production environments hand over crash logs and stack traces, nothing more, and the historic volume of real crashes is rarely enough to fine-tune an LLM for localization on its own. One approach addresses this by generating synthetic crashes through code mutation, building a dataset of 64,369 crashes from 4.1 million mutations of the HANA codebase. The result: 66.9% accuracy at predicting root cause location, against baselines of 12.6% and 10.6%. Tested for generalization on other codebases, the same approach hit 63% accuracy on SQLite and 74% on DuckDB, which suggests the method transfers across projects rather than overfitting to HANA's particular quirks. That matters most where this pipeline is weakest: complex cases where the root cause and the technical crash site are two different places.

Where the pipeline breaks down in practice

Laid out straight, the pipeline runs in four steps. A crash gets collected with version-tagged metadata and symbolicated into function names, file names, and frame sequences. Machine learning or LLM-based deduplication groups the resulting reports into clean clusters. Those clusters get tracked across build versions, with statistical models watching for new clusters or threshold spikes tied to a specific build change. Once a cluster gets flagged, LLM-based fault localization takes over to narrow the regression down to a specific file or method.

Each step has a specific way it fails, and none of them are exotic.

Missing symbols break everything downstream. A frame that only shows a raw memory address, with no function name attached, is useless to every step that follows it. Getting the symbol path configuration right isn't a detail to handle later, it's a prerequisite the whole pipeline depends on.

Deduplication errors don't stay contained to Step 2. Wrong cluster boundaries produce false or missed regression signals in Step 3, automatically, because the detection step trusts whatever clusters it's handed. Cluster quality is load-bearing for everything built on top of it.

Embedding-based clustering brings its own mechanical problem: vectors with 3,072 dimensions, such as those returned by OpenAI's text-embedding-3-large model, create high-dimensional spaces that clustering algorithms weren't built for. Teams adopting embedding-based dedup need to plan for dimensionality reduction as a standard step, not an optimization to revisit later.

On the localization side, Direct-LLM's hallucination risk is real and needs to be treated as a known cost of using the faster method. Agentic-LLM cuts that risk down but at higher compute cost per crash. And the fine-tuning data shortage doesn't go away without deliberate synthetic augmentation, the mutation-based approach out of the SAP HANA research being one demonstrated way to manufacture enough training volume where real crash history falls short.

The 37% coverage gap underlies all of it. Stack traces alone don't name the buggy file in most cases, so any localization system that only reads the trace literally is capped well below the ceiling teams actually need.

The silent regression risk never fully disappears if monitoring stays binary. Moving from an exit-code CI model to continuous cluster monitoring across canary rings is what actually closes the gap, catching the slow-building failure before it reaches broad rollout instead of after.

There's an organizational piece too, easy to overlook next to the modeling work. The SAP HANA K-Detector deployment is instructive here: industrial teams care as much about pipeline continuity, auto-creating or auto-binding bug IDs so delivery schedules aren't held up, as they care about raw detection accuracy. A detection system that's accurate but slows down every release train is a hard sell, regardless of its numbers.

Applying this pipeline today, with the tooling that supports it

None of the AI layers matter if the symbol pipeline foundation is shaky. Before anything else, the symbol pipeline needs to be reliable: verified symbol paths, consistent metadata tagging across version, OS, and device type, and a crash ingestion system that posts dumps with full context attached. Skipping this step and jumping straight to clustering models is the most common way these projects stall.

Teams with modest crash volume don't need the most advanced tooling to start. Traditional ML baselines, around 0.85 to 0.86 accuracy, are often good enough for deduplication at early scale. Embedding-based approaches like dedupT and GPTrace should be evaluated once volume grows past what those baselines can comfortably handle, but they're not a day-one requirement.

On fault localization, Direct-LLM enhancement is the sensible low-cost starting point, moving Top-1 accuracy from 10.6% up into the 40.2 to 43.1% range for a fraction of the compute cost of the alternative. Agentic-LLM earns its extra cost specifically on complex regressions, the ones where the real cause sits somewhere the stack trace never shows. Knowing which of those two problems is on the table, before picking the tool, is most of the battle.