What the study found
The study found that large language models (LLMs, AI systems trained on large text datasets) performed poorly when judged by whether their infrastructure-as-code (IaC, code used to provision cloud infrastructure) templates could actually be deployed. It also found that an iterative feedback framework and human guidance improved deployability.
Why the authors say this matters
The authors say deployability is the critical measure of whether IaC configuration files are useful, and that focusing only on syntax misses this point. The study suggests that continued research is needed because the generated templates showed low alignment with user intent and low security compliance.
What the researchers tested
The researchers built DPIaC-Eval, a deployability-focused IaC benchmark with 153 real-world scenarios across 58 services. They also proposed IaCGen, an LLM-based framework that uses iterative feedback through format verification, syntax checking, and live deployment stages, and they tested six state-of-the-art LLMs.
What worked and what didn't
On the first attempt, the six LLMs achieved only 20.8% to 30.2% deployment success. With IaCGen, 54.6% to 91.6% of generated IaC templates became deployable within the first 10 iterations, and human-in-the-loop feedback pushed performance to over 90% passItr@25 on all evaluated LLMs.
What to keep in mind
The abstract does not provide details about the exact systems used in deployment, the nature of the human feedback process, or how broadly the results generalize beyond the benchmark. It also reports poor user requirement coverage (25.2%) and security compliance (8.4%), but does not explain the causes of those outcomes in the available summary.
Key points
- Six LLMs had only 20.8% to 30.2% first-attempt deployment success.
- DPIaC-Eval includes 153 real-world scenarios across 58 unique services.
- IaCGen uses format verification, syntax checking, and live deployment feedback.
- IaCGen made 54.6% to 91.6% of templates deployable within 10 iterations.
- Human-in-the-loop feedback raised performance to over 90% passItr@25 on all evaluated LLMs.
- Reported user requirement coverage was 25.2% and security compliance was 8.4%.
Disclosure
- Research title:
- LLMs showed low first-attempt deployability for IaC generation
- Authors:
- Tianyi Zhang, Shidong Pan, Zejun Zhang, Zhenchang Xing, Xiaoyu Sun
- Institutions:
- Australian National University, Australian National University, Columbia University, Commonwealth Scientific and Industrial Research Organisation, Data61, Nanyang Technological University, New York University
- Publication date:
- 2026-06-30
- DOI:
- 10.1145/3797142
- OpenAlex record:
- View
Get the weekly research newsletter
Stay current with scholarly research without reading academic papers — one filtered digest, every Friday.