UC Riverside's SemLoc Beats Classic Bug Finders by 36 Points Using LLMs

Many fault-localization methods assume a failed test executes different code from a passing test. Semantic defects break that assumption: both runs may traverse identical lines while carrying different values. Researchers at UC Riverside and Cisco Research propose SemLoc, a framework that converts properties generated by a large language model into executable constraints, checks them during tests, and ranks the associated source lines.

The preprint reports 42.8% Top-1 and 68.0% Top-3 line-localization accuracy on a 250-program Python benchmark. The classic Ochiai coverage method reaches 6.4% Top-1. SemLoc also reduces the candidate set to about 7.6% of executable lines. Top-k accuracy, also written as Acc@k, measures how often the faulty line appears among the first k results.

Coverage misses value-level failures

Spectrum-based fault localization, or SBFL, records which statements each test executes and correlates that coverage with pass or fail outcomes. A line visited mostly by failing tests receives a high suspiciousness score. When every test follows the same path, the executed lines receive identical or near-identical scores. Numerical errors, boundary mistakes, missing normalization, and incorrect relational operators often produce this pattern.

SemLoc’s running example is a numerically unstable softmax implementation. Stable softmax subtracts the maximum input value before exponentiation, using an expression such as x - m. The faulty version computes x + m, which can produce plausible results for small inputs and NaNs for large ones. Every test still executes the same statements in the same order, leaving coverage unable to separate the faulty shift from the surrounding arithmetic.

A direct LLM prediction can nominate a line, but the prediction supplies no runtime evidence that can be compared across tests. Its location may also be ambiguous if the model refers to a variable with several definitions. SemLoc addresses both problems by grounding generated properties to specific program values and testing those properties during execution.