Runs 70B models on a single 4 GB GPU without quantization, by keeping only one layer on the GPU at a time.
pip install airllmAirLLM cuts inference memory by loading one model layer onto the GPU at a time. The project says very large models, such as DeepSeek-V3, fit in about 12 GB this way.
pip install airllm
Load a supported model (Llama, Qwen, DeepSeek and more) and run it from Python.
antirez · Tool / CLI
antirez's focused engine for running DeepSeek V4 Flash and a few other large open models locally on Metal, CUDA and ROCm.
vLLM project · Tool / CLI
A high-throughput, memory-efficient engine for serving LLMs, with PagedAttention, continuous batching and prefix caching.
Unsloth · Tool / CLI
Run, fine-tune and serve LLMs and diffusion models on your own hardware, with an OpenAI-compatible API for your agents.