来源简介
arXiv:2605.04357v2 Announce Type: replace-cross Abstract: The landscape of large language models (LLMs) has become increasingly fragmented, with no single model dominating the field. Meanwhile, cloud providers offer a diverse array of mid-tier and older-generation GPUs. These heterogeneous resources provide better availability and deliver comparable performance-per-dollar to top-tier hardware. To efficiently harness these GPUs for concurrent multi-LLM serving, we introduce Coral: an adaptive, heterogeneity-aware serving system. The key idea behind Coral is the joint optimization of resource allocation and replica serving strategies across all models. To keep pace with shifting throughput demands and resource availability, Coral employs a lossless, two-stage decomposition. This approach preserves joint optimality while reducing online solve times from hours to mere tens of seconds. Evaluations across 6 models and 20 GPU configurations demonstrate that Coral reduces serving costs by up to 2.79$\times$ compared to the strongest baseline, while delivering up to 2.39$\times$ higher goodput under scarce resource constraints.
本条尚未生成中文 AI 摘要，以上展示来源提供的原始简介。
采集于 2026/10/20 21:06:31（北京时间）。 arXiv 日期为本次公告时间，论文初次投稿日期请查看原文版本记录。