What the study found
The study found that four state-of-the-art reference-based code evaluation metrics can be strongly biased toward surface-level features instead of code functionality. It also found that these metrics perform significantly worse on LoCaL, a new benchmark designed to target cases where they are likely to do poorly.
Why the authors say this matters
The authors suggest that exposing code evaluation metrics to LoCaL-like data might help develop metrics that are robust to surface bias. They present this as relevant because reliable code evaluation metrics are described as crucial for progress in software engineering tasks.
What the researchers tested
The researchers critically evaluated four state-of-the-art reference-based code evaluation metrics, which score a candidate program by comparing it to a reference program. They also introduced LoCaL (Looks Can Lie), a benchmark with 3,117 code pairs at both the method and program levels, using differential fuzzing to assign functional similarity scores without predefined test cases.
What worked and what didn't
LoCaL includes code pairs that are surface-similar but functionally dissimilar, as well as pairs that are functionally similar but surface-dissimilar. The abstract says the functional similarity scores are made more reliable by executing an order of magnitude more tests than prior work. All four evaluated metrics showed significant performance degradation on LoCaL compared with baselines.
What to keep in mind
The abstract says prior datasets for these metrics rarely included the kinds of code pairs that LoCaL targets. It does not provide detailed limitations beyond this gap in existing evaluation data.
Key points
- Four reference-based code evaluation metrics were found to favor surface-level similarity over functionality.
- LoCaL is a benchmark of 3,117 code pairs at the method and program levels.
- LoCaL uses differential fuzzing to compute functional similarity scores without predefined test cases.
- All four evaluated metrics performed significantly worse on LoCaL than on baselines.
- The authors suggest LoCaL-like data may help develop metrics that are less affected by surface bias.
Disclosure
- Research title:
- LoCaL reveals surface bias in code evaluation metrics
- Authors:
- Simantika Dristi, Matthew B. Dwyer
- Institutions:
- University of Virginia, University of Virginia
- Publication date:
- 2026-06-30
- DOI:
- 10.1145/3797089
- OpenAlex record:
- View
Get the weekly research newsletter
Stay current with scholarly research without reading academic papers — one filtered digest, every Friday.