Qwen Inference Optimization Model serving llama.cpp Gemma Multimodal Open Models Hy3 Inkling GPU Infrastructure Developer tools Coding Agents Edge Compute Kimi

llama.cpp

LLM inference in C/C++

Visit site View stories

Open-source C/C++ software for local inference with large language models, including running GGUF models and serving them through local APIs.

Recent stories

11 linked stories

workflowPRIMARY2026-08-12

Qwen 122B runs on older laptop with llama.cpp in user test

A LocalLLaMA post ran a 122B Qwen model on an older laptop with llama.cpp. The run had very long load and generation times, while another report put Qwen 3.6 35B at 21 tok/s on a Radeon 7600 after ROCm tuning.

newsSECONDARY2026-08-08

Together AI ranks first or tied first on 3 of 4 Kimi K3 provider benchmarks

Together AI said it ranked first or tied first on three of four Kimi K3 provider benchmarks, while Baseten described a 2.8T-parameter Blackwell GB300 serving stack. Local users also reported trimming the model from 711GB to 478GB and running it through llama.cpp RPC across clusters.

releaseSECONDARY2026-08-07

Magnitude launches open-source offline coding agent for local models

Magnitude launched an open-source terminal coding agent that runs local models on-device without API keys. Its launch post says it profiles hardware and can use shell, file-editing, script, and skills tools.

workflowPRIMARY2026-07-16

LocalLLaMA users report near-6x Qwen 3.6 27B speedups with MTP

A LocalLLaMA benchmark on Qwen 3.6 27B and RTX 6000 PRO reports near-6x speedups from MTP, DFlash, and n-gram drafting. Related tests cover remote prefill, NUMA offload, GPU clock tuning, and llama.cpp Gemma 4 support.

newsPRIMARY2026-07-16

Inkling adds early llama.cpp serving via 1-bit GGUF

Inkling's 1-bit GGUF ran in llama.cpp at 30–40 TPS, and TokenSpeed added day-zero support with a flat KV cache pool. Arena posts put Inkling #10 among open models in frontend code and text, while docs drew scrutiny.

releaseSECONDARY2026-07-14

Tencent releases 1-bit and 4-bit GGUF weights for 295B Hy3 single-GPU runs

Tencent released 1-bit and 4-bit GGUF builds for its 295B Hy3 model with llama.cpp support and MTP. Posts cite 88–92GB local runs and SWE-Bench scores of 75.4% Verified and 53.9% Pro.

newsSECONDARY2026-06-11

North Mini Code adds MLX, Unsloth GGUFs, and oMLX support

Cohere added MLX support, Unsloth GGUFs, oMLX work, and updated docs for North Mini Code two days after launch, with llama.cpp still under review. The broader runtime coverage makes the 30B coding model easier to run on local Mac, quantized, and self-hosted stacks.

releaseSECONDARY2026-06-10

vLLM, Unsloth, and llama.cpp add DiffusionGemma support after launch

Google's new diffusion text model picked up same-day runtime support: vLLM added native diffusion-LM serving, Unsloth shipped GGUFs, and llama.cpp got local setup guidance. That shortens the path from release to local and hosted evaluation.

releaseSECONDARY2026-06-03

Gemma 4 12B ships encoder-free multimodal local model with 16GB target and 256K context

Google released Gemma 4 12B, an Apache 2.0 encoder-free multimodal model with native audio and vision for 16GB-class laptops. Day-zero support in llama.cpp, vLLM, Ollama, MLX, and SGLang should make local agents and on-device apps easier to deploy immediately.

releasePRIMARY2026-05-29

llama.cpp launches official site with one-line installer and unified `llama` CLI

llama.cpp now has an official website and a single-line installer that provides one `llama` entrypoint for running, serving, and agent integrations. The packaging change simplifies local setup while reusing GGUF models already on disk.

releasePRIMARY2026-05-16

llama.cpp provider adds in-process AI SDK support with tool calling

A new llama.cpp provider lets the AI SDK run directly inside a Node process without a separate server, while exposing reasoning, tool calling, image inputs, and prompt caching. The setup shortens local deployment paths for AI SDK apps that want llama.cpp bindings.