DeepSeek V4 Flash benchmarks claim lower DeepSWE cost than GPT-5.6 Luna

• DeepSeek V4 Flash-0731 is winning on dollars, not single-shot quality: two Flash attempts solved more DeepSWE tasks than one GPT-5.6 Luna attempt at roughly one-third the cost, according to Together's comparison post.
• Luna still leads when attempts are matched: teortaxesTex's pass@k note says Flash pass@2 beat Luna pass@1, while Flash pass@4 narrowly trailed Luna pass@2.
• The wrapper moved the outcome: Composio found the same Flash model passed 17/30 tasks in Oh My Pi, 16/30 in Claude Code and Codex, and 14/30 in OpenCode, in its harness benchmark.
• The cheap-token window may be temporary: DeepSeek posted a notice saying API prices would rise significantly, shown in TestingCatalog's screenshot.

Together's full DeepSWE writeup has the awkward table: Luna wins pass@1, every equal attempt count, all five languages, and 7 of 8 domains, yet the DeepSeek-first cascade beats Luna alone on both cost and accuracy. The archived DeepSeek API docs say the deepseek-v4-flash slug now routes to DeepSeek-V4-Flash-0731, while Unsloth's local guide puts the model at 284B total parameters, 13B active, and a 1M context window. ARC Prize's task note adds the weird bit: on one public ARC-AGI-2 task, max reasoning used 76k tokens and got the answer right while high reasoning spent 98k tokens on the wrong hypothesis.

Together ran 900 DeepSWE rollouts across 113 long-horizon coding tasks, four trials per task, comparing DeepSeek-V4 Flash-0731 max against GPT-5.6 Luna max in its published analysis.

The headline numbers:

• Pass@1: Luna 67.2%, Flash 53.3%.
• Cost per rollout: Luna $0.61, Flash $0.10.
• Solves per $100: Luna 110, Flash 532.
• Median runtime: Luna 16 minutes, Flash 23 minutes.
• Median steps: Luna 92, Flash 148.
• DeepSeek-first cascade: 78.9% solved at $0.385 per task.

The cascade is the Christmas-come-early part for agent-loop nerds. Flash is weaker on one shot, but cheap enough to clear the easy half before Luna sees the rejects.

The pass-count debate started immediately because the first comparison was one Luna attempt versus two DeepSeek attempts.

Equal attempt counts keep Luna ahead in Together's table: 81.6% versus 70.1% at pass@2, and 90.3% versus 80.5% at pass@4. Cross-count comparisons flip the practical story because two Flash attempts cost about $0.20, while one Luna attempt costs about $0.61 in the same run.

A later SWE caveat framed the result as possibly coding-specific: Luna was described as being in the same league as V4 or lower on everything except Business and SWE.

ARC Prize posted a separate reasoning result, with DeepSeek V4 Flash at 61.4% on ARC-AGI-2 for $0.04 per task and 89.0% on ARC-AGI-1 for $0.02 per task.

teortaxesTex's ARC comparison compared that ARC-AGI-2 result with DeepSeek-V3.2 at 4.0% and three times the cost, calling it roughly nine months of progress. ARC said ARC-AGI-3 evaluations were still rolling out because they are more operationally intensive, in its follow-up.

The reasoning-effort detail is useful because more thinking was not monotonically better. On public task 28a6681f, ARC Prize's example says low reasoning spent 26k tokens and missed, high spent 98k tokens and missed, and max spent 76k tokens and solved it.

Composio ran the same Flash model through Claude Code, Codex, OpenCode, and Oh My Pi on 30 real multi-step tasks across Gmail, Sheets, Airtable, GitHub, Slack, Calendar, Notion, and PagerDuty.

The chart breaks cleanly:

• Success: Oh My Pi 17/30, Claude Code 16/30, Codex 16/30, OpenCode 14/30.
• Cost per successful task: OpenCode $0.073, Codex $0.081, Oh My Pi $0.103, Claude Code $0.195.
• Median time: Claude Code 122.7s, OpenCode 129.7s, Codex 245.0s, Oh My Pi 272.4s.

Harness choice alone changed success by three tasks, cost by almost 3x, and speed by 2.2x, Composio wrote in its benchmark summary. The task design mattered too: Composio's task note says every task passed only if all fixed checks passed.

Composio's replies added caveats that did not fit in the chart:

• Full prompts were not published, according to one Composio reply.
• Reruns preserved the ranking, according to another reply.
• Claude Code had been the most expensive harness across many of Composio's tests, one reply said.
• Hermes was missing because Composio had trouble running it, then said it would add Hermes in a later batch in one reply and another reply.
• OMP is a fork of Pi, according to Composio's clarification.

DeepSeek's API docs describe OpenAI-compatible and Anthropic-compatible endpoints, with https://api.deepseek.com and https://api.deepseek.com/anthropic as base URLs in the archived docs. The same docs say deepseek-v4-flash now calls the 0731 version without changing the invocation name.

Ollama made 0731 the default for deepseek-v4-flash on its cloud and claimed 120+ output tokens per second, zero data retention hosting in the US and Europe, and long uninterrupted coding-harness sessions in its rollout post. Wafer posted an OpenRouter provider table for DeepSeek-V4-Flash-0731-Fast, with Wafer at $0.28 input, $0.56 output, $0.07 cache read, and 225 tps in its provider screenshot.

Baseten became an official Hugging Face inference provider for Kimi K3, DeepSeek V4 Flash, and GLM-5.2, according to Baseten's Hugging Face post and the Hugging Face announcement. Baseten also said it was ARC Prize's inference partner for testing DeepSeek V4 Flash and Kimi K3 in its ARC post.

Claude Code users found a 1M-context gotcha: aibuilderclub_'s env-var post sets CLAUDE_CODE_AUTO_COMPACT_WINDOW=786432 so Claude Code can use more of DeepSeek's context before summarizing the session. A same-thread reply called it the officially recommended setting in the follow-up.

Together also shipped a docs-side explainer layer for developers asking about TTFT, context windows, sampling, fine-tuning, quantization, and deployment tradeoffs, in its Learn docs post.

Unsloth said DSpark lets DeepSeek-V4-Flash-0731 GGUFs generate about 1.4x to 2x faster with no accuracy change, and its guide lists the lossless Q8_K_XL file at 162GB and a 3-bit option at 103GB.

Atomic Chat's local guide gives a parallel quant ladder, from 162.1GB AD-BF16 down to 70.2GB AD-IQ1_M, and frames Flash as workstation-class because it activates 13B parameters per token. Ollama's model page lists the same 284B total, 13B active, 1M-context shape.

Local reports were split between speed claims and setup friction:

• Teknium reported around 40 tok/s on two Sparks connected with one cable and said it worked in Hermes Agent, in his local inference post.
• A reply from Teknium put one failing setup on the inference engine and named vLLM in the follow-up.
• lgrammel reported 35 to 40 generation tokens per second on a MacBook Pro M5 in his local post.
• One LocalLLaMA post hit 3 to 4 t/s on a Sapphire Rapids memory-bandwidth setup in the Reddit thread.
• Another LocalLLaMA post saw MTP at 30 to 40 t/s but DSpark at 1 to 2 t/s with its llama-server settings in the DSpark thread.

A third LocalLLaMA user said a 2-bit Flash quant wrote a custom Metal kernel for a Kimi quant in about 50 minutes, then reported 4 t/s decode and 20 prefill for Kimi K3 Q1_0 on a 512GB Mac Studio in the kernel thread.

Practitioners were not treating Flash like a pure benchmark artifact. They were leaving it in harnesses for hours.

The hands-on reports clustered around persistence and obedience:

• teortaxesTex said Flash-0731 was harness-sensitive, did not get confused, ran overnight from a /goal, and made "massive progress" by morning in his report.
• The same account later said a 7.5-hour OMP run on a tough problem "won" in a follow-up.
• Hangsiin said Prime Agent plus DeepSeek V4 Flash grasped initial context quickly and held up better over long contexts than OpenCode in his first-day report.
• yacineMTB said he reimplemented about 5,000 lines of Super Smash Bros. Melee C from his phone with Flash-0731 in his demo.
• The same thread says Sol reward-hacked his requirements while DeepSeek made mistakes but listened in yacineMTB's comparison.
• yacineMTB's tooling pulled Slippi replays, compared Dolphin against the reimplementation side by side, and generated divergence videos, according to his harness note and his tooling note.

There were counter-signals. teortaxesTex's WebGPU note described a case where Flash thought it found a Dawn bug and warned that it is easy to get "LLM psychosis'ed" around plausible model discoveries. one user report said Flash-0731 looked overfit to coding, fell down on a non-coding task, and got stuck in reasoning loops.

DeepSeek's cheapest-api story picked up a caveat on August 6: a platform notice said overall API pricing would increase significantly, with specific pricing still subject to official notice.

Wes Roth posted a before-and-after screenshot showing the notice had changed from peak-valley pricing language to a broader significant-increase warning in his screenshot. Rohan Paul noted current rates of $0.14 per 1M input tokens and $0.28 per 1M output tokens, then tied the warning to Ollama's fastest-growing-model claim in his pricing post.

thdxr argued the increase looked like traffic shaping rather than cost recovery, saying OpenCode had reproduced DeepSeek's current prices on rented GPUs in his price-increase read. A separate OpenCode traffic estimate put one record-breaking day at about $120,000 of DeepSeek Flash token value on OpenCode inference and about $20,000 on OpenRouter.

OpenRouter had already shown the Jevons pattern around Luna: its usage post said GPT-5.6 Luna token volume 10x'd after its OpenRouter price dropped by 10x.