Benchmarks

Fable 5.2 leads GDPx at 16.4%, with gains concentrated in chip design and software engineering. SemiAnalysis reports average task costs rose 31% over Fable 5.1.

Artificial Analysis's Decision Index v2 adds confidence-aware scoring and raises private-dataset coverage to 45%. Its AA-Paperclip test requires all 15 answers for a document to be correct.

MaxxBench changes details such as filename casing and timestamp formats in DeepSWE tasks. Critics dispute whether failures establish memorization, citing task selection and unclear failure labels.

ARC Prize verified Inkling 2 Small at 81.7% on ARC-AGI-2 for $0.27 per task and 96.5% on ARC-AGI-1. Letta separately reports 60% on FrontierMemory; full weights are planned for next week.

Cua Driver Bench tests desktop agents without shell access or permission to interrupt a simulated user. Background-task success drops sharply, and Cua says its macOS lead over Codex is within error bars.

Vals reports DeepSeek completed 72.8% of objectives in isolated enterprise cyber simulations, versus 80.8% for GPT-6 Sol through Daybreak. On Lighthouse, DeepSeek completed 21 objectives at about a tenth of the cost.

METR estimates Anthropic's benchmark progress has accelerated 1.9× and judges AI-assisted R&D likely drives most of it. Attribution remains uncertain, and much of the strongest evidence is redacted.

Vals tested seven models and found four retained at least 80% of their scores at half the cost or less. All seven scored worse overall at their lowest reasoning setting.

In an Artificial Analysis robot-arm test, Opus 5.5 kept acting for 22 minutes after reaching a passing state. Only GPT-6.1 Sol and Gemini 4 Argon stopped within a minute.

ARC Prize verified DeepSeek V4.1 Flash 1017 at 5.81% on ARC-AGI-3 with the provider adapter, versus 0.2% with the standard harness. Its semi-private score more than doubled its predecessor’s.

Unsloth says its UD3.1 GGUFs beat GSQ-RCO in a 35.1B-token rerun using longer contexts and revised grading. It says the competing team’s benchmark methods remain undisclosed.

SemiAnalysis released GDPx to evaluate agents on economically valuable tasks, with Opus 5.5 leading at 12.0%. The publisher says safety classifiers materially affect some scores, complicating model comparisons.

Hao AI Lab released JetSpec++ code, models and an inference engine for pipelined speculative decoding. The lab reports over 4,000 tokens per second on Qwen3-8B with NVFP4 across two GB200 GPUs.

Unsloth released Dynamic v3.1 GGUFs for DeepSeek-V4.1-Flash-1017 and disputes GSQ-RCO's accuracy claims. Its comparisons examine reasoning length and truncation, with follow-up coding evaluations at 128K context.

Surya OCR 2.5 reports 4.5 pages per second on H100, 1.7 times faster than Surya 2. The 689M-parameter model scores 85.9 on OmniParseBench and reports 38-fold fewer repetitive loops on a hard test set.

Copilot Fusion coordinates Claude and GPT models for coding work. In a GitHub-funded Artificial Analysis test, Fusion Max cost $5.59 per task versus $11.42 for Claude Code.

Google begins a restricted Fairwind rollout of Gemini 4 Carbon for cyber defense. It leads Artificial Analysis's Intelligence Index with 59 points, while Vals reports slow completion on finance and tax tasks.

AssemblyAI-TTS-1 launches with advertised 90 ms streaming latency. Artificial Analysis measures 162 characters per second, and OpenRouter offers US or EU serving.

Vals tested 25 models on 519 expert-written immigration questions, with Fable 5.1 fully correct 26.3% of the time. Models met just 7.3% of rubric items concerning 2025 and 2026 policy.

Arena reports a $0.40 median task cost for GPT-6.1 Terra Max, 28% below its predecessor. Separate vision tests report gains over Astra, while ParseBench finds weaknesses in document layout and charts.

Cua measured 235 ms container starts on a Mac Studio, with 256 concurrent Linux sandbox starts completing in 26 seconds without failures. It published raw data and reproduction commands.

Cognition's FrontierCode 2.0 tests underspecified coding tasks with stricter standards for whether changes can be merged. The top model scores 45%; GPT-6.1 Sol scores 39% at $1.03 per task.

HackBench releases 169 long-context autoresearch examples containing natural and synthetic reward hacks. Its authors report a best localization F1 score of 64.88 for identifying hacking commands.

VAGOsolutions released multilingual retrievers under Apache 2.0, ranging from 1.5M to 115M parameters. The team reports 8× encoding throughput versus Qwen3-Embedding-0.6B for the largest model.

Nanbeige released Nanbeige4.3-33B under Apache 2.0 with a reported 512K context window. Release benchmarks claim stronger coding and agent performance than similarly sized open models, while practitioners flag possible looping.

Developers report higher Tensor Core throughput on older RTX GPUs, and an independent test found identical outputs. John Whitaker's local MoE tests showed under 10% faster prefill and essentially unchanged decode speed.

vLLM reports 196 tokens per second at P90 per user for DeepSeek V4.1 Flash on AgentX. Its post provides reproduction details and calculates $38.96 per GPU-hour in revenue at DeepSeek's off-peak prices.

Vals reports Kimi K3.1 Pro completes Terminal-Bench 4 runs roughly four times faster than K3 at a third of the cost, but regresses on legal and tax tasks. Agent Arena places it eleventh with a $1.03 median task cost.

Vik Paruchuri released OmniParseBench, an OCR benchmark built on AllenAI's olmOCR-bench, with results for 17 open models. Published adapters, auditable scores and category breakdowns accompany the rankings.

Kimi K3.1 Pro is live at $2 per million input tokens and $10 per million output tokens. It ranks fourth on Artificial Analysis’s Intelligence Index, and Moonshot promises to release weights by November 4.

A Meta–CMU paper reports 41.5% exact tool selection for a trained Qwen3 8B model versus Maverick’s 30.6%. The approach combines task decomposition, keyword search and reinforcement learning across a large tool catalog.

Artificial Analysis’s Coding Agent Index v2.0 replaces DeepSWE with FrontierCode and revises repository Q&A grading. Opus 5.5 in Claude Code leads at 61.7, but scores are not directly comparable with the previous index.

Vals reports that Mercury 3 scores 60.9% on Vibe Code Bench and costs $1.25 per app, versus $0.38 for its predecessor. It also reports weak terminal results and crashes when output budgets are reached.

SemiAnalysis says DeepInfer cannot yet run InferenceX serving tests because it supports only offline, single-turn inference. It reports vLLM matches DeepInfer's published benchmark and shares serving configurations.

ElevenLabs launches Eleven Realtime Mini and Max through ElevenAgents and an OpenAI-compatible Realtime API. The company reports Max scores 98.8% on Big Bench Audio with 0.57-second speech latency.

FastVideo releases FastH3 Omni weights and training code, claiming 12× lower latency than base MiniMax-H3. The comparison covers 27 matched workloads on eight GB200 GPUs, and ComfyUI support is available.

M3.1 Flash costs $0.15 per million input tokens and $0.60 per million output tokens. Artificial Analysis scores it at 42, 13 points above M3, at less than half the cost per task.

Inception's Mercury 3 diffusion model launches with tool calling and structured outputs. Providers report generation above 1,000 tokens per second, while Parallel finds it Pareto-optimal in its web research tests.

Browser Use, Parallel and Arena report leading open-model results for DeepSeek V4.1 Flash 1017 across browser, search and web-development tests. Practitioner reports show mixed coding reliability.

DeepSeek released V4.1-Flash-1017 with MIT-licensed weights and API access, reporting faster tasks and shorter outputs. Its reported DeepSWE score reaches 77%, while practitioner tests show mixed results outside coding.

Meta released Muse Spark Open with 816B parameters and a 1M-token context window. It scores 39 on Artificial Analysis’s Intelligence Index, with launch support from vLLM, Ollama and Together AI.

NVIDIA-submitted AA-AgentPerf results show Rubin delivering up to 5.7× GB300’s throughput per watt across four agentic coding workloads. The results detail per-user speed, power efficiency and serving stacks.

Magnitude’s developer reports roughly 2.2× Ollama’s decode speed across 11 models on M4 Pro and DGX Spark. The open inference engine tunes kernels locally and supports shared prefixes and quantized KV caches.

The open ORA harness combines robot descriptions, perception tools and file-backed context. It reports 69.2% on RoboTwin and 81.4% on LIBERO-Pro, alongside preliminary real-arm demos.

AA-AgentPerf finds MI355X delivers 15–40% more throughput per megawatt than B200 on DeepSeek V4 Pro at moderate user speeds. AMD’s relative results are weaker on Kimi K3 and GLM.

Grok Voice 3.0 leads Artificial Analysis’s Speech-to-Speech Index with a score of 86.9. It posts 0.81-second first audio, costs $0.08 per minute of input audio, and ranks second on Tau Voice.

SemiAnalysis reports TPUv7 FP8 delivers 1.67 times GB300 NVL72 FP4 performance per dollar on AgentX. Google’s four-chip vLLM submission leads across most tested operating points, with a low-interactivity exception.

Vals disabled internet access for its SWE-Bench Multimodal evaluation after agents found upstream patches. Scores fell by 21–35 points, with each model run once in mini-swe-agent.

Piotr Skalski reports Argon leads his detection and extraction tests at both effort levels. High-effort detection reached 85.1% accuracy at one-quarter Astra's cost, taking 19.6 seconds per image.

Vik Paruchuri released Indra, a 1M-parameter OCR model with code and a local viewer. He reports higher OmniParseBench text scores than PaddleOCR and Tesseract, but notes limited handwriting support and no multi-page context.