Inference Optimization

Google's TPU 8tx pairs existing 8t silicon with AMD processors for confidential inference. The 8i Max combines two 8i chips for long-context workloads.

Nace AI released Drex, an open-weight multimodal decision-model family that outputs calibrated probabilities. Its training report describes verified decision training, while reports put the 27B Max response time at 183 ms.

Anthropic’s public beta lets clients continue cached conversations by sending a previous message ID. Anthropic reports 35% lower median time to first token on Claude Code requests with 500K–1M tokens, at standard token rates.

Unsloth says its UD3.1 GGUFs beat GSQ-RCO in a 35.1B-token rerun using longer contexts and revised grading. It says the competing team’s benchmark methods remain undisclosed.

Zyphra Hierarchical Attention adds learned tree routing to pretrained models for sparse retrieval. It retains the full KV cache; the large speedup figures are roofline projections, not measured serving gains.

Hao AI Lab released JetSpec++ code, models and an inference engine for pipelined speculative decoding. The lab reports over 4,000 tokens per second on Qwen3-8B with NVFP4 across two GB200 GPUs.

Unsloth released Dynamic v3.1 GGUFs for DeepSeek-V4.1-Flash-1017 and disputes GSQ-RCO's accuracy claims. Its comparisons examine reasoning length and truncation, with follow-up coding evaluations at 128K context.

Mistral released Large 4 weights with launch-day serving and LoRA training support. vLLM reports roughly twice baseline output throughput with its built-in drafter in specified tests.

VAGOsolutions released multilingual retrievers under Apache 2.0, ranging from 1.5M to 115M parameters. The team reports 8× encoding throughput versus Qwen3-Embedding-0.6B for the largest model.

Developers report higher Tensor Core throughput on older RTX GPUs, and an independent test found identical outputs. John Whitaker's local MoE tests showed under 10% faster prefill and essentially unchanged decode speed.

vLLM reports 196 tokens per second at P90 per user for DeepSeek V4.1 Flash on AgentX. Its post provides reproduction details and calculates $38.96 per GPU-hour in revenue at DeepSeek's off-peak prices.

Modular says Mojo and MAX are fully open for development, extension and commercial use, including compiler and runtime internals. Support for NVIDIA, AMD and Apple hardware is already public.

OpenAI made its Decisions API generally available for bounded text and image decisions without full responses. It adds native JSON inputs, prompt caching and up to 400 questions per request.

vLLM added NVFP4 weights, NVFP4 KV caching and DSpark support for MiniMax M3.1 Flash. It reports 76 concurrent sequences at 256K context on four B300 GPUs, versus 28 for M3.

Recursive released Flint, an Apache-2.0 coding model with 40B parameters and 2.5B active per token. The company reports over 125 tokens per second on an M4 Pro MacBook Pro at short contexts.

DeepSeek releases DeepInfer under Apache 2.0 for V4.1 Flash on Blackwell or Ascend hardware. The engine exposes techniques such as speculative decoding but omits some serving components.

FastVideo releases FastH3 Omni weights and training code, claiming 12× lower latency than base MiniMax-H3. The comparison covers 27 matched workloads on eight GB200 GPUs, and ComfyUI support is available.

FastVideo's FastH3-QAD-v2 and NVFP4 VAE decoders cut weights from 118 to 30 GiB. The team reports generating a five-second 480p clip in 52.7 seconds on one DGX Spark, down from 141 seconds.

Inception's Mercury 3 diffusion model launches with tool calling and structured outputs. Providers report generation above 1,000 tokens per second, while Parallel finds it Pareto-optimal in its web research tests.

vLLM reports 842 tokens per second at batch one on GB300 NVL72 with DSpark, plus up to 1.8× better P90 interactivity. Engineers dispute describing the implementation as a megakernel.

NVIDIA-submitted AA-AgentPerf results show Rubin delivering up to 5.7× GB300’s throughput per watt across four agentic coding workloads. The results detail per-user speed, power efficiency and serving stacks.

Magnitude’s developer reports roughly 2.2× Ollama’s decode speed across 11 models on M4 Pro and DGX Spark. The open inference engine tunes kernels locally and supports shared prefixes and quantized KV caches.

PhoneLLM Mini and Max are Nemotron fine-tunes claiming per-call time to first answer token below 100ms and 200ms, respectively. Those figures exclude network and tool latency; the production P95 target is below 600ms.

AA-AgentPerf finds MI355X delivers 15–40% more throughput per megawatt than B200 on DeepSeek V4 Pro at moderate user speeds. AMD’s relative results are weaker on Kimi K3 and GLM.

A Stanford and Together AI paper reports 62.5–72.9% less decoding compute with higher accuracy from Hybrid Reasoning Agents. HybridLink switches between latent reasoning and text-based tool calls.

SemiAnalysis reports TPUv7 FP8 delivers 1.67 times GB300 NVL72 FP4 performance per dollar on AgentX. Google’s four-chip vLLM submission leads across most tested operating points, with a low-interactivity exception.

OpenBMB released open MiniCPM-V 4.7 models in 35B-A3B and 1B sizes with mobile deployment support. It reports fivefold faster inference on a 16GB iPad Pro M4 using Lookahead Expert Scheduling.

OpenAI says GPT-6.1 Sol Ultrafast runs up to eight times faster than Standard. API pricing is $12/$60 per million input/output tokens, while Codex and ChatGPT Work access is limited to eligible plans.

Beam is a sparse mixture-of-experts model with 23B active parameters, a reported 1M-token effective context, and Apache 2.0 weights. Independent benchmarking is beginning.

OpenAI says Ultrafast generates tokens up to eight times faster than Standard in Codex. The tier is available through the API and selected subscriptions at a higher price; GPT-6.1 Sol support is planned.

Fireworks says Ember-1 used 71% fewer reasoning tokens and 39% fewer total tokens than K3 in a live coding-traffic test. The models achieved the same success rate.

Quail open-sourced an MIT-licensed engine that plans AI queries, batches inference, and reuses KV cache across filters and joins. Its authors report 1.84× faster execution than hand-tuned vLLM on 29 queries.

Parakeet Redux compresses NVIDIA's Parakeet from 1.2 GB to 178 MB with ternary weights. Its author reports 113× real-time CPU speed and stronger results on the 25-language FLEURS benchmark.

Developers report using Jev to score inbox content, classify invoices, and split documents into categories. DocJev’s author claims 6x lower latency than GPT-5.6 Luna at equivalent accuracy for document classification and splitting.

Kev is an Apache-2.0 family of 0.6B, 4B, and 8B decision models compatible with TypeSafe System One APIs. Its author reports that the 8B model reached 79.6% out-of-domain accuracy versus Jev’s 85.7%, while the 4B model runs on a 32 GB Mac.

Jev returns predefined choices, scores, and probabilities instead of free-form text for bounded software decisions. TypeSafe claims roughly 150 ms responses and lower inference costs for those tasks.

Cohere released an open-source serving system that fuses the LLM decode step into one GPU kernel launch. On North Mini Code with one H100, it reports up to 1.58x vLLM performance at the tested batch size.

Perplexity open-sourced Lily, a Rust and Metal engine for Qwen3.6-35B-A3B in Perplexity Computer's hybrid workflow. Perplexity reports 1.23× faster prefill and 1.35× faster decode on an M5 Max MacBook Pro.

A developer reports streaming 60% of Qwen3.8-Flash-Next experts from disk on demand, running the full model in 37 GB at 40 tokens per second. BF16 and GGUF weights are also available for local deployments.

A developer reports streaming 60% of Qwen3.8-Flash-Next's experts from disk, running the full Q4 model in 37GB of RAM at 40 tokens per second. A separate H100 deployment reports 160 tokens per second with EAGLE-3.

The Prefix Sliding paper introduces an inference method that preserves the task prefix and a recent-token window while discarding older reasoning tokens. Its authors report up to 3× faster inference without retraining and longer reinforcement-learning rollouts.

Qwen released Qwen3.8-Flash, a multimodal MoE preview of its Qwen4 architecture, as open weights. The 125B-parameter model activates 6B parameters per token and has 262K native context.

OpenAI says Jalapeño delivered 1.5–1.9× more work per watt and 1.7–3.6× lower end-to-end latency than NVIDIA systems in its tests. The company plans to deploy the inference chip in its compute infrastructure by year-end.

Groq 3 LPX adds dedicated token generation to NVIDIA Vera Rubin systems, with Groq and Nebius among planned deployers. Artificial Analysis measured about 3,400 output tokens per second on Gemma 4 31B.

Community tests report Qwen 3.8 27B handling coding, OCR, and long-context workloads locally. One vLLM setup reached 3,200 tokens per minute at 262K context on two RTX 3090s without NVLink.

A LocalLLaMA user reports reasoning loops caused most errors in a 2,483-task test of Qwen 3.8 27B. The report says 3–9% of inputs drove most failures because reasoning often did not terminate.

Sentence Transformers 6.0 adds MultiVectorEncoder for ColBERT-style training, inference, and interpretation, including visual-document retrieval. Index size can increase substantially: 4,874 passages expanded to 608,414 token vectors in one example.

A practitioner report found speculative-decoding acceptance fell from 0.71 to 0.18 beyond about 32K context. Separate DSpark and mlx-dspark tests reported speedups on RTX and Apple Silicon setups.

Meta released Muse Glimmer, a 30B Apache 2.0 dense model for local agent workflows. Reports cite 4-bit builds under 20GB, vision input, function calling, 131K context, and day-0 support in Hugging Face, vLLM, SGLang, Ollama, and MLX.

Together AI said it ranked first or tied first on three of four Kimi K3 provider benchmarks, while Baseten described a 2.8T-parameter Blackwell GB300 serving stack. Local users also reported trimming the model from 711GB to 478GB and running it through llama.cpp RPC across clusters.