What the study found
The study reports that Liger+, a distributed large model inference system, can dynamically balance latency and throughput on multi-GPU systems. Its key idea is interleaved parallelism, which interleaves computation and communication across requests.
Why the authors say this matters
The authors say distributed large model inference faces a tradeoff between low latency and high throughput, especially when different parallelism strategies fix performance metrics in advance. The study suggests Liger+ can better match changing demand while maintaining performance across models and devices.
What the researchers tested
The researchers presented Liger+, which includes task-aware batch management and a distributed runtime module. The batch manager organizes batches based on discriminative and generative task features, and the runtime schedules computation and communication kernels across multiple requests, streams, and GPUs.
What worked and what didn't
The abstract says Liger+ achieved precise kernel execution order control by combining CPU-GPU synchronization and inter-stream synchronization. It also used fine-grained resource mapping and a contention factor strategy to anticipate resource contention, and it decomposed kernels into smaller units at runtime to increase overlap. In evaluations, it outperformed fixed parallelism strategies in most cases; for a 4-device discriminative task, it reduced P90 latency by 43.8% while keeping throughput the same as pipeline parallelism, and it improved throughput by 1.53× with improved P90 latency compared with tensor parallelism. For a 4-device generative task, it achieved an average 1.15× throughput improvement and a 26.2% reduction in P90 latency compared with tensor parallelism.
What to keep in mind
The abstract does not describe detailed limitations, and the reported results are limited to the models, tasks, and device settings tested. The summary also does not provide information about implementation cost or broader deployment constraints.
Key points
- Liger+ is presented as a distributed large model inference system for multi-GPU architecture.
- The system uses interleaved parallelism to combine computation and communication across requests.
- Task-aware batch management and a distributed runtime module are central parts of the design.
- In evaluation, Liger+ generally outperformed fixed parallelism strategies.
- On a 4-device discriminative task, it reduced P90 latency by 43.8% versus pipeline parallelism while keeping throughput the same.
- On a 4-device generative task, it improved throughput by 1.15× and reduced P90 latency by 26.2% versus tensor parallelism.
Disclosure
- Research title:
- Liger+ balances latency and throughput in distributed model inference
- Authors:
- Jinhui Wei, Shenggan Cheng, Wei Zhu, Jiazhi Jiang, Dan Huang, Zhiguang Chen, Jiangsu Du, Yutong Lu
- Institutions:
- China Mobile (China), National University of Singapore, Sun Yat-sen University, Sun Yat-sen University, Sun Yat-sen University, Sun Yat-sen University, Sun Yat-sen University, Sun Yat-sen University
- Publication date:
- 2026-02-23
- DOI:
- 10.1145/3797040
- OpenAlex record:
- View
Get the weekly research newsletter
Stay current with scholarly research without reading academic papers — one filtered digest, every Friday.