antirez's focused engine for running DeepSeek V4 Flash and a few other large open models locally on Metal, CUDA and ROCm.
DwarfStar is a deliberately narrow inference engine. Instead of supporting every model, it is built and tested around a few excellent ones, such as DeepSeek V4 Flash and GLM, on Mac, NVIDIA and AMD hardware.
Build it from the repository as the README describes, then download a supported model.
lyogavin · Tool / CLI
Runs 70B models on a single 4 GB GPU without quantization, by keeping only one layer on the GPU at a time.
vLLM project · Tool / CLI
A high-throughput, memory-efficient engine for serving LLMs, with PagedAttention, continuous batching and prefix caching.
Unsloth · Tool / CLI
Run, fine-tune and serve LLMs and diffusion models on your own hardware, with an OpenAI-compatible API for your agents.