llama.cpp
LLM inference in C/C++
Open-source C/C++ software for running large language models locally, centered on efficient inference for the llama.cpp project.
Recent stories
Cohere added MLX support, Unsloth GGUFs, oMLX work, and updated docs for North Mini Code two days after launch, with llama.cpp still under review. The broader runtime coverage makes the 30B coding model easier to run on local Mac, quantized, and self-hosted stacks.
Google's new diffusion text model picked up same-day runtime support: vLLM added native diffusion-LM serving, Unsloth shipped GGUFs, and llama.cpp got local setup guidance. That shortens the path from release to local and hosted evaluation.
Google released Gemma 4 12B, an Apache 2.0 encoder-free multimodal model with native audio and vision for 16GB-class laptops. Day-zero support in llama.cpp, vLLM, Ollama, MLX, and SGLang should make local agents and on-device apps easier to deploy immediately.
llama.cpp now has an official website and a single-line installer that provides one `llama` entrypoint for running, serving, and agent integrations. The packaging change simplifies local setup while reusing GGUF models already on disk.
A new llama.cpp provider lets the AI SDK run directly inside a Node process without a separate server, while exposing reasoning, tool calling, image inputs, and prompt caching. The setup shortens local deployment paths for AI SDK apps that want llama.cpp bindings.