Magnitude claims 2.2× Ollama decode speed across 11 models

• Magnitude claims roughly 2.2× Ollama’s decode throughput across 11 models: tomgreenwald’s launch post reports 2.19× on M4 Pro and 2.17× on DGX Spark.
• Kernels are tuned on the user’s hardware, and models get automatically selected speculative-decoding drafters, as described in the launch thread.
• Concurrent agents share prefix caches and use quantized KV caches, which tomgreenwald confirmed in a reply.
• Free desktop apps cover macOS, Windows and Linux, according to the launch thread.

Both engines already have speculative decoding enabled in the benchmark chart. Magnitude’s README also documents an OpenAI-compatible API and a server service that keeps running after logout.

Magnitude’s developer reports averages across 11 popular local models, tested at 4-bit precision and 64k context. The comparisons use different Ollama backends on the two machines:

• Mac M4 Pro versus Ollama with MLX: 1.32× prefill throughput and 2.19× decode throughput.
• NVIDIA DGX Spark versus Ollama with llama.cpp: 1.34× prefill throughput and 2.17× decode throughput.

The headline 2.2× claim applies to token generation. Prompt processing gets the smaller gains listed above.

During model download, Magnitude spends up to a minute testing configurations on the actual chip. According to the README, the search varies:

• How many GPU threads cooperate on the work.
• How large a chunk each thread handles.
• How far ahead data is loaded.

Magnitude says it retains the fastest combination that preserves the model’s output. Per-machine tuning is the launch’s most interesting bet.

Each model also gets an automatically selected speculative-decoding drafter:

Magnitude describes three mechanisms for concurrent agent sessions:

• Quantized KV caches: a claimed reduction to about half the memory per agent.
• Elastic memory allocation: memory grows and shrinks with sessions, with allocations freed when agents stop.
• Shared prefix caches: sessions reuse cached computation for common prompt prefixes.

The desktop download connects to agents including Pi, OpenCode, Hermes and Codex. The code is published under Apache 2.0 in the project repository.

Beyond the desktop interface, the repository documents:

• OpenAI-compatible API access for clients outside the one-click integrations.
• A bundled magnitude CLI, with no separate CLI installation required.
• magnitude server setup on Linux and macOS, which installs a service that starts at boot, survives logout, updates itself and serves clients on other computers.

The README says prompts, files and models stay on the machine, with no internet required after a model has downloaded.

Apple Silicon support extends to the M1 family:

For an older-NVIDIA compatibility question, tomgreenwald said the Vulkan backend already offered a path and proposed adding a build for older CUDA versions:

An open compatibility request, filed October 15, targets:

• Tesla P100: sm_60.
• Tesla P40: sm_61.
• Tesla V100: sm_70.

Hardware support still leaves model-fit questions. After calling one setup a tight fit, tomgreenwald asked for its exact hardware and OS before investigating further.

Magnitude lists three additional optimizations as work in progress:

• Expert streaming from RAM or disk, to run models larger than GPU memory allows.
• Splitting a model across GPUs and CPU, to use multiple compute devices.
• Attention pruning, to skip low-impact computation and accelerate prefill.