What the study found
The study found that foundation model-generated workflows in domain-specific languages often contain defects, and that static analysis can identify some of them. The authors also present Timon, a static analyzer built for these workflows, and show that its feedback can be used to repair detected defects.
Why the authors say this matters
The authors conclude that detecting and repairing defects is a step toward more reliable and automated generation of executable workflows from natural language requirements. The findings indicate that improving workflow accuracy and reliability is an important need in this setting.
What the researchers tested
The researchers examined foundation model-generated workflows written in domain-specific languages and built an initial taxonomy of defect incidences with 20 types. They then developed Timon, a static analyzer for these workflows, and used its feedback to guide Pumbaa, a foundation model-based tool, in repairing defects.
What worked and what didn't
The study reports that 89.23% of the studied workflow instances contained at least one defect. It also reports that nine defect types could be effectively identified through static analysis, and that feedback from Timon could guide repairs of detected defect incidences.
What to keep in mind
The abstract does not describe the full details of the dataset, evaluation setup, or how well each repair performed. It also does not state whether all defect types can be detected or repaired, only that nine types were identified through static analysis.
Key points
- 89.23% of the studied FM-generated DSL workflows contained at least one defect.
- The authors built a taxonomy of 20 defect types in these workflows.
- Nine defect types could be identified through static analysis.
- Timon is presented as a static analyzer for FM-generated DSL workflows.
- Feedback from Timon was used to guide repairs in Pumbaa.
Disclosure
- Research title:
- Static analysis found many defects in FM-generated workflows
- Authors:
- Sogol Masoumzadeh, Keheliya Gallaba, Dayi Lin, Ahmed E. Hassan
- Institutions:
- Huawei Technologies (Canada), Huawei Technologies (Canada), McGill University, Queen's University
- Publication date:
- 2026-04-21
- DOI:
- 10.1145/3809500
- OpenAlex record:
- View
Get the weekly research newsletter
Stay current with scholarly research without reading academic papers — one filtered digest, every Friday.