What the study found
The study found that Clotho, a task-specific pre-generation test adequacy measure, can estimate how difficult an input is for a large language model (LLM) by using hidden states, which are internal representations inside the model. It can also help rank unseen inputs by likely failure after a small reference set has been labeled.
Why the authors say this matters
The authors say this matters because testing LLMs on specific tasks is difficult and costly, especially when many prompts lack ground truth answers and output-based adequacy measures are only available after full inference. The study suggests Clotho may reduce LLM execution costs and complement post-generation uncertainty or confidence measures.
What the researchers tested
The researchers introduced Clotho and evaluated it across eight benchmark tasks and three open-weight LLMs. They used a Gaussian Mixture Model (GMM), a statistical model that groups data by patterns, to adaptively sample a reference set from a large pool of unlabeled inputs and then rank unseen inputs by likelihood of failure.
What worked and what didn't
Clotho predicted failures with a ROC-AUC of 0.716 after labeling reference sets that were, on average, 5.4% of inputs. The abstract also says it did this without generating outputs, and that when prioritizing test inputs for proprietary models it increased the average number of failing inputs from 18.7 to 42.5 out of 100 compared with random prioritization.
What to keep in mind
The summary does not describe detailed limitations beyond the scope of the evaluation across eight benchmark tasks and three open-weight LLMs. It also states that Clotho's adequacy scores learned from open-weight LLMs transfer effectively to proprietary models, but the abstract does not provide further detail on where this transfer may or may not hold.
Key points
- Clotho estimates LLM input difficulty before any output is generated.
- It uses hidden states and a Gaussian Mixture Model to choose informative inputs for labeling.
- In tests across eight benchmark tasks and three open-weight LLMs, it reached a ROC-AUC of 0.716.
- The labeled reference sets were, on average, only 5.4% of inputs.
- The abstract says Clotho's scores transfer effectively from open-weight LLMs to proprietary models.
Disclosure
- Research title:
- Clotho predicts LLM failures before generating outputs
- Authors:
- Juyeon Yoon, Somin Kim, Robert Feldt, Shin Yoo
- Institutions:
- Chalmers University of Technology, Korea Advanced Institute of Science and Technology, Korea Advanced Institute of Science and Technology, Korea Advanced Institute of Science and Technology
- Publication date:
- 2026-06-30
- DOI:
- 10.1145/3797114
- OpenAlex record:
- View
Get the weekly research newsletter
Stay current with scholarly research without reading academic papers — one filtered digest, every Friday.