#llm-serving

A high-performance serving framework for LLMs and multimodal models, from a single GPU to large distributed clusters.

A high-throughput, memory-efficient engine for serving LLMs, with PagedAttention, continuous batching and prefix caching.