Together AI ranks first or tied first on 3 of 4 Kimi K3 provider benchmarks

• Together AI said Moonshot benchmarked Kimi K3 across major inference providers, with Together ranking first or tied first on 3 of 4 benchmarks in Together's provider benchmark post.
• Serving K3 is a full-stack problem: thursdai_pod's Baseten walkthrough note points to a 2.8T-parameter deployment on Blackwell GB300s, while Baseten's own writeup says the MXFP4 weights are over 1.4TB.
• The local crowd is already making the impossible merely painful: a LocalLLaMA pruning post reported a 711GB to 478GB GGUF cut, and a 16x GB10 cluster post reported 20+ t/s average on the full model.
• K3's distribution is spreading through hosted surfaces, including GitHub Copilot in GitHub Copilot's rollout post, LangSmith LLM Gateway in LangChain's LangSmith post, and Baseten on Hugging Face in Baseten's Hugging Face post.
• Capability claims are now task-specific: Together's Harvey LAB-AA clip said Kimi K3 scored nearly twice Claude Fable 5 on hard autonomous legal tasks.

Moonshot's Kimi K3 tech blog calls it a 2.8T-parameter, native multimodal model with Kimi Delta Attention, Attention Residuals, and a 1M-token context window. Baseten's day-zero API writeup says GB300 nodes come in fours, so its team ran tensor and expert parallelism across nodes on NVL72. The vLLM recipe asks for at least 8x GB300, CUDA 13, and an r580+ NVIDIA driver. GitHub's Copilot changelog says the K3 rollout was paused and resumed on Aug. 6 after a GitHub Actions incident.

Together said Moonshot benchmarked Kimi K3 endpoints across major inference providers and that Together ranked first or tied first on 3 of 4 benchmarks.

The provider story is shifting from “who hosts the weights” to “who preserves the evaluated model.” ComputeSDK said its independent provider benchmarks had been running weekly for three or four weeks and would move to daily test runs for providers in ComputeSDK's Benchmark Friday update.

The Artificial Analysis provider page separately lists 10 Kimi K3 API providers and reports a 281% output-speed spread between the fastest and slowest providers. Baseten's day-zero API writeup says Moonshot's Kimi Vendor Verifier was used to catch serving failures across tool calling, frontend parsing, and structured outputs.

The verification layer is the story under the benchmark chart. K3 is open weight, but the serving stack can still change the model users experience.

Baseten's K3 bringup reads like a checklist for 3T-class open-weight serving. Its day-zero API post says the team had early access to Moonshot's weights and worked with Inferact on vLLM and RadixArk on SGLang.

The concrete pieces:

• Hardware: NVIDIA GB300 NVL72 systems.
• Weight load: over 1.4TB of MXFP4 weights.
• Precision path: native MXFP4 weights with MXFP8 activations, used directly instead of porting to NVFP4.
• Engine bringup: prerelease vLLM and SGLang builds for Kimi K3.
• Architecture support: KDA, AttnRes, Stable LatentMoE, tokenization, tool calling, and structured outputs.
• Parallelism sweep: TP, EP, ADP toggles, batch sizing, speculative decoder draft lengths, linear-layer caching intervals, routing parameters, and engine settings.
• Cross-node detail: eight GB300 GPUs are required to fit the weights, but GB300 nodes come in fours, so Baseten ran TP and EP across nodes on NVL72.
• Tokenizer bottleneck: Baseten built a custom tokenizer it says is up to 18x faster than tiktoken on long inputs.

Baseten also says most quality issues it sees are frontend bugs, not quantization. That claim lines up with Moonshot's Kimi Vendor Verifier GitHub repo, which includes pre-flight checks for parameter constraints before benchmarks run.

The vLLM recipe is unusually explicit about the hardware floor.

The deployment details in the Kimi K3 vLLM recipe:

• Model shape: 2.8T MoE, 16 of 896 experts active per token, KDA, AttnRes, 1M context, native vision.
• Container: vllm/vllm-openai:kimi-k3.
• CUDA path: CUDA 13 build only, with r580+ NVIDIA driver required.
• NVIDIA hardware: at least 8x GB300, with multi-node setups for production traffic.
• ROCm path: vllm/vllm-openai_rocm:kimi-k3, with at least 8x MI355X or MI350X.
• Serve flags: --trust-remote-code, --load-format fastsafetensors, --enable-prefix-caching, --enable-auto-tool-choice, --tool-call-parser kimi_k3, and --reasoning-parser kimi_k3.

The vLLM launch post frames the hard part as making KDA, MXFP4 MoE, KV cache management, prefill/decode disaggregation, speculative decoding, and long-context recipes work together in one runnable engine.

The local experiments are less “consumer laptop K3” than “creative memory hierarchy abuse.” The [Kimi-K3-REAP GGUF link from the LocalLLaMA post]Kimi-K3-REAP GGUF points to a trimmed build that the post says cut an Unsloth IQ2-XXS artifact from 711GB to 478GB by removing multilingual weight.

The update in the pruning post said the 478GB reap576_iq2xxs variant solved three selected SWE-Lancer tasks that the standard 2-bit version failed in that user's environment. The same update said the run streamed experts from SSD through a patched llama.cpp path and averaged 2.5 hours per task.

Segmond described a separate llama.cpp RPC run across two clusters, with the main cluster still partially offloading because the system could not hold everything in memory in segmond's local run. The 16x GB10 cluster report surfaced by the GB10 cluster post claimed 20+ t/s average, 38 t/s peak, and 750 t/s prefill on the full model.

The weirdest local datapoint came from technaturalism's Metal kernel report: DeepSeek V4 Flash spent about 50 minutes writing a custom Metal kernel for an Unsloth Kimi K3 quant, producing roughly 4 t/s decode and 20 t/s prefill on a 512GB Mac Studio.

Moonshot's Kimi K3 repo names Kimi Delta Attention and Attention Residuals as the core architecture change, with Stable LatentMoE activating 16 of 896 experts.

nrehiew_'s KDA thread gives a compact mental model for why K3 does not need RoPE:

• RoPE uses a static rotation matrix.
• RoPE preserves norm through orthogonal rotations.
• RoPE composes rotations across positions.
• KDA can be read as read/write operations with varying strength.
• KDA can also be read as online regression against retrieval error with fast weights.
• KDA's transition matrix is data-dependent, with generalized Householder transforms and channelwise decay.
• KDA drops the RoPE orthogonality constraint because it accumulates one transform over query/key state instead of composing two independent absolute-position transforms.

In nrehiew_'s follow-up, the difference is framed as preservation versus update: softmax attention with RoPE preserves information for brute-force QK matching across context, while KDA removes and adds information through the update rule.

K3 is spreading through the boring places engineers actually call models from.

The shipped surfaces now include:

• GitHub Copilot: GitHub Copilot's rollout post says Kimi K3 is available in Copilot for VS Code, hosted by Fireworks AI.
• LangSmith: LangChain's LangSmith post says Kimi K3 can be called through LangSmith LLM Gateway using a LangSmith API key.
• Hugging Face: Baseten's Hugging Face post says Baseten became an official HF Inference Provider for Kimi K3, DeepSeek V4 Flash, and GLM-5.2.
• Databricks: the Databricks announcement says Kimi K3 is available through Foundation Model API with Unity AI Gateway governance and ZDR.
• Arc Prize: Baseten's Arc Prize post says Baseten is ARC Prize's inference partner for testing models including DeepSeek V4 Flash and Kimi K3.
• OpenRouter: OpenRouter's task leaderboard post says open models took two task leaderboards by real spend share, with Kimi K3 leading tool dispatch and DeepSeek V4 Pro leading shell execution.

The Hugging Face Baseten post says requests can route through Hugging Face billing or a custom provider key. That gives K3 a path into existing HF SDK workflows without each user wiring a separate Baseten account first.