A high-throughput, memory-efficient engine for serving LLMs, with PagedAttention, continuous batching and prefix caching.
uv pip install vllmvLLM serves large language models fast and cheaply. It manages key-value memory with PagedAttention, and uses continuous batching and prefix caching to keep throughput high under real traffic.
uv pip install vllm
Start a server for the model you want, then point your app or agent at it.
SGLang project · Tool / CLI
A high-performance serving framework for LLMs and multimodal models, from a single GPU to large distributed clusters.
lyogavin · Tool / CLI
Runs 70B models on a single 4 GB GPU without quantization, by keeping only one layer on the GPU at a time.
Modular · Tool / CLI
Modular's open platform for AI development and deployment: the MAX framework, the Mojo language and an OpenAI-compatible server.