#inference

Runs 70B models on a single 4 GB GPU without quantization, by keeping only one layer on the GPU at a time.

antirez's focused engine for running DeepSeek V4 Flash and a few other large open models locally on Metal, CUDA and ROCm.

Modular's open platform for AI development and deployment: the MAX framework, the Mojo language and an OpenAI-compatible server.

A high-performance serving framework for LLMs and multimodal models, from a single GPU to large distributed clusters.

A high-throughput, memory-efficient engine for serving LLMs, with PagedAttention, continuous batching and prefix caching.