Modular's open platform for AI development and deployment: the MAX framework, the Mojo language and an OpenAI-compatible server.
One repo for the Modular Platform: the MAX framework for serving models, the Mojo language for writing fast kernels, and an OpenAI-compatible inference server.
Follow the getting-started guide linked from the README.
vLLM project · Tool / CLI
A high-throughput, memory-efficient engine for serving LLMs, with PagedAttention, continuous batching and prefix caching.
SGLang project · Tool / CLI
A high-performance serving framework for LLMs and multimodal models, from a single GPU to large distributed clusters.
lyogavin · Tool / CLI
Runs 70B models on a single 4 GB GPU without quantization, by keeping only one layer on the GPU at a time.