LM Serving

LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization
Adaptive Quantization and Phase-Aware Partition for LLM serving atop heterogenous cluster.
Juntao Zhao, Borui Wan, Yuanghua Peng, Haibin Lin, Chuan Wu
PDF Cite Code Poster DOI Full paper