Kimi K3 report details RL distillation and FlashKDA infrastructure

• Moonshot released Kimi K3 as an open-weight 2.8T-parameter MoE with 104B active parameters, native vision, a 1M-token context window, and companion infra, according to Kimi_Moonshot's release post.
• The post-training recipe is SFT to nine RL specialists to one distilled checkpoint: ZhihuFrontier's K3 pipeline breakdown maps the three domains by three reasoning-effort levels, while _lewtun's Journal Club summary identifies multi-teacher on-policy distillation as the central trick.
• The architecture is a scaled Kimi Linear descendant, with KDA, Gated MLA, NoPE, LatentMoE, and Attention Residuals doing the real work, per rasbt's architecture notes.
• The open stack is not cosmetic: FlashKDA claims 1.72x to 2.22x H20 prefill speedups in Kimi_Moonshot's FlashKDA post, and nrehiew_'s sandbox notes quote 51,219,741 sandboxes across 1,505,678 images during training and evaluation.
• The report leaves important blanks: stochasticchasm's missing-data note says training tokens and compute are not disclosed, and rohanpaul_ai's license summary says large MaaS providers need a separate Moonshot agreement after $20M in trailing group revenue.

The technical report is unusually useful because it puts architecture, RL, and serving infrastructure in one document. The official blog frames the headline as 2.5x intelligence per unit of compute over K2, while the vLLM deployment guide gets down to serve flags, KDA prefix caching, and DSpark speculative decoding. The AgentENV release post says its Firecracker-based sandboxing cuts agent environment costs by 88.6% to 96.8% under typical workloads.

Moonshot shipped more than weights. The release bundled the model, the report, the tech blog, and three pieces of infrastructure that explain why K3 can be trained and served at this size.

• Model: 2.8T total parameters, 104B active parameters, 1M-token context, native visual understanding, per Kimi_Moonshot's release post.
• Weights: the Hugging Face model card describes K3 as an open-weight, native multimodal agentic model.
• Report: the arXiv abstract says K3 still trails Claude Fable 5 and GPT-5.6 Sol overall, while leading the other open and proprietary models in Moonshot's suite.
• Infra: FlashKDA, MoonEP, and AgentENV were released alongside the model, according to Kimi_Moonshot's FlashKDA post, Kimi_Moonshot's MoonEP post, and Kimi_Moonshot's AgentENV post.

K3 is Christmas come early for architecture diagram people. The report's core pattern is three KDA layers plus one Gated MLA layer per block, with sparse expert capacity after attention.

• Sequence mixing: KDA handles efficient long-sequence recurrence, while periodic Gated MLA layers preserve global attention, according to the technical report.
• Position handling: K3 uses NoPE everywhere; rasbt called it the first frontier-level NoPE-everywhere architecture he knew of.
• Width: Stable LatentMoE routes each token to 16 of 896 experts, with 104B active parameters, per mervenoyann's spec list.
• Depth: Attention Residuals let later layers retrieve earlier block representations, and rasbt's architecture notes quote about 4% extra training cost and 2% extra inference cost.
• Stability: K3 uses lower-bounded KDA decay, SiTU-GLU, RMSNorm, and Quantile Balancing as scale-control mechanisms, as rohanpaul_ai's report summary lays out.

suchenzang put the design theme bluntly: sequence length, depth, and MoE width each needed its own dynamic-range bandages.

The post-training section is the most reusable part of the report. K3 does not train one generic RL policy and hope effort controls work later.

1. SFT: K3 starts from synthesized long-horizon agent trajectories, multi-stage verification, human annotation, a unified XTML format, and quantization-aware training with MXFP4 routed-expert weights and MXFP8 activations, according to ZhihuFrontier's K3 pipeline breakdown.
2. RL specialists: Moonshot trains policies across General, General Agents, and Coding Agents, each at low, high, and max reasoning effort, producing nine teacher policies, not to be confused with K3's 896 routed MoE experts.
3. Budgeted behavior: a cold-start model estimates an initial token budget, then domain-specific multipliers shape max, high, and low policies; stochasticchasm's effort note calls the post-SFT budget estimate especially clean.
4. MOPD: the final student generates its own rollout, then the matching teacher scores the sampled next token from the student's actual prefix, giving token-level supervision on the student's state distribution, per ZhihuFrontier's K3 pipeline breakdown.
5. No top-k win: nrehiew_'s pipeline note says Moonshot reported no advantage from top-k distillation.

Agent RL at K3 scale looks less like a benchmark loop and more like cluster orchestration for unreliable contractors. The report describes long-running tasks, resumable rollouts, external KV retention, and persistent sandbox state.

• Task synthesis: nrehiew_'s task-synthesis note says K3 builds a knowledge graph with web-search agents, samples keys, retrieves internet material, and synthesizes tasks from those seeds.
• Task mix: nrehiew_'s task dump lists general verifiable work, kernel optimization, personal assistant tasks, autoresearch, and webdev tasks including SVG and 3D.
• Partial rollouts: _lewtun's Journal Club summary says completed trajectories can trigger updates while unfinished ones are carried into later iterations and reprioritized.
• Sandbox scale: nrehiew_'s sandbox notes quote 51,219,741 sandboxes across 1,505,678 images during K3 training and evaluation.
• Isolation: the AgentENV blog says each task runs in a Firecracker microVM with on-demand image loading, incremental snapshots, copy-on-write forks, and elastic lifecycle management.

rauchg points to the security lesson buried in the report: container isolation was not enough when agents could trigger kernel panics and deadlocks.

K3 forced serving engines to handle two cache worlds at once: recurrent KDA state and MLA KV cache. That detail shows up all over the day-zero deployment work.

• FlashKDA: Kimi_Moonshot's FlashKDA post claims 1.72x to 2.22x prefill speedups over flash-linear-attention on H20, with a drop-in backend.
• MoonEP: the MoonEP GitHub README describes dynamic redundant experts, perfect token-load balance, zero-copy communication, and static shapes for expert parallelism.
• vLLM: vLLM_project's deployment thread says prefix caching had to be rebuilt around recurrent KDA state, then lists P/D disaggregation, KV offloading, tool calling, structured output, and DSpark speculative decoding.
• SGLang: lmsysorg's SGLang post reports 423 tok/s on GSM8K with fused KDA decode kernels, DP attention, DSpark, PD disaggregation, and KDA-aware prefix caching.
• Batch-one speed: vLLM_project's speed post reports 464 tok/s on a low-entropy reasoning workload with batch size 1 and 4x4 GB300.

nrehiew_ caught the important cache gotcha: K3 has KDA state and MLA KV cache, and recurrent-state shape makes block-level hashing too coarse.

The same K3 checkpoint behaved very differently depending on the agent harness around it. Composio ran 28 tasks through Kimi Code, Hermes, and Claude Code, then ThursdAI graphed the spread.

• Success band: thursdai_pod's harness result says success stayed between 71% and 79% across the three harnesses.
• Median tokens: thursdai_pod's token breakdown gives 61K for Kimi Code, 67K for Hermes, and 340K for Claude Code.
• Cost per task: thursdai_pod's cost breakdown gives $0.22 for Kimi Code, $0.28 for Hermes, and $2.00 for Claude Code, a 9.1x spread.
• Outcomes: thursdai_pod's outcome breakdown lists 22/28 for Kimi Code, 21/28 for Hermes, and 20/28 for Claude Code.

Cline went one step weirder: cline's self-improvement run says K3 modified the Cline harness for 17 hours, raising Terminal-Bench from 77.5% to 88.8% and cutting run cost from $79 to $49.8.

K3 is open-weight, not a permissive open-source release. the license writeup notes that Moonshot avoids calling the license open source in its own materials.

• MaaS trigger: rohanpaul_ai's license summary says model-as-a-service providers need a separate Moonshot agreement once licensee-plus-affiliate revenue exceeds $20M over any trailing 12 months.
• Product attribution: the same summary says commercial products over 100M monthly users or $20M monthly revenue must prominently display “Kimi K3.”
• No public pricing restriction: rohanpaul_ai's license summary says the public license does not specify royalties, revenue sharing, or mandatory commercial pricing terms.
• Missing training scale: stochasticchasm's missing-data note says the report does not disclose training tokens or compute for K3 or MoonViT 2, while cedric_chee's hardware-detail note says the report is light on hardware details compared with K2.5.
• Serving floor: ZhihuFrontier's hardware-cost breakdown says K3 weights occupy roughly 1.56TB and identifies 8x MI355X or 8x B300 as the lowest public load-class configurations, with production serving possibly moving toward 64-plus accelerators.
• Early API price parity: jaminball's pricing note says Baseten and Fireworks were both at $3 per million input tokens and $15 per million output tokens, matching Moonshot's direct pricing at the time of the post.