Thinking Machines releases Inkling 2 Small weights under Apache 2.0

• Inkling 2 Small’s weights are downloadable under Apache 2.0, as Thinking Machines announced.
• Day-zero vLLM support includes vision, mixed-precision MoE layers and DSpark speculative decoding, according to vLLM’s launch announcement.
• Code Arena: WebDev scores rose from 1,408 for Inkling-Small to 1,647, +239 points, putting it first among US open models in Arena’s release snapshot.
• Agent Arena reports a $0.26 median task cost and second place among open models in its launch-day results.

The NVFP4 quantizer tries mapping each block’s maximum to both 4 and 6, keeping the lower-error result. The larger MXFP8 checkpoint gains just 1-2 evaluation points, according to Thinking Machines, while vLLM’s tool parser is unit-tested but still lacks an end-to-end tool-calling benchmark.

The November 12 weight release follows the model’s November 5 debut, dated in its model card. Existing Tinker access continues, according to cHHillee.

Inkling 2 Small has 594B total parameters and 17B active per token. Thinking Machines lists two hardware floors on Hugging Face:

• Native NVFP4 checkpoint: at least 500GB aggregate GPU memory, with examples of two B300s or four H200s.
• Alternative MXFP8 checkpoint: at least 800GB, with examples of four B300s or eight H200s.

Thinking Machines says pretraining, midtraining and post-training used NVFP4. The low precision is native to training.

The vLLM architecture comparison separates the retained backbone from the new components:

• Hybrid attention: 40 sliding-window layers and eight full-attention layers, preserving a 5:1 pattern. The local window doubles from 512 to 1,024 tokens.
• Relative-bias GQA: no RoPE, with 48 query heads and eight KV heads in every layer.
• Block attention residuals: learned queries mix the embedding and earlier six-layer block outputs before each sublayer.
• Gated attention: a sigmoid output gate and a per-token attention sink.
• Vision: a 23-layer ViT replaces Inkling’s four-layer hMLP encoder. Inputs are text and images; output is text. Audio input is absent.

Four width-4 short convolutions remain in each decoder layer. Inference therefore needs rolling three-token convolution state alongside the KV cache, as andrew_n_carr observed.

Every decoder layer routes each token to eight of 384 experts, with no shared experts. The released checkpoint splits expert precision by depth, as documented in the vLLM integration:

• Layers 0-39: NVFP4 experts.
• Layers 40-47: MXFP8 experts.
• Expert activations: quantized to match each layer’s weight format.

vLLM folds activation quantization into the kernels that produce those activations, eliminating separate quantization passes over the hidden states.

Initial support uses the vllm/vllm-openai:inkling-2-small image pending inclusion in a stable release. The published recipe verifies two GB200 configurations:

• One four-GPU tray, TP4: up to 262,144 tokens.
• Two trays, TP8: the full 1,048,576-token context.

Expert parallelism follows the tensor-parallel group: TP4 attention uses EP4 experts, and TP8 uses EP8.

vLLM publishes this TP8 launch command:

The bundled five-layer DSpark draft uses an RNN head and proposes seven tokens per step. vLLM reports an acceptance length around 3.5 at temperature 1.0 and up to 630 tokens/s for one user at TP4.

The day-zero integration came with serious kernel work. In its engineering write-up, vLLM reports:

• Five operations fused: all-reduce, short convolution, residual update, attention-residual mixing and RMSNorm share one NVLink-multicast kernel.
• Sequence-parallel prefill: an 8K-token prefill step takes 22% less time at TP4 and 26% less at TP8, freeing 1.3-1.9GiB per GPU for KV cache.
• Relative-attention decode: a Gluon kernel cuts the batch-one decode path at 100K context from 15.1ms to 4.8ms.

Its agentic throughput test used TP8 across two four-GPU nodes, prefix caching and probabilistic drafting. Each turn appended the previous answer plus roughly 2K tool-output tokens, then generated 1,024 tokens with reasoning enabled at temperature 1.0.

| Session context | Output tokens/s/GPU at 100 tokens/s/user | Concurrent sessions | Prefix-cache hit rate |
| --- | ---: | ---: | ---: |
| 32K | ~1,070 | ~84 | 91% |
| 64K | ~915 | ~72 | 95% |
| 128K | ~700 | ~55 | 98% |
| 256K | ~450 | ~37 | 99% |

The recipe reports 15-25% lower acceptance at temperature 1.0 with default greedy drafting. vLLM also checked speculative decoding against non-speculative runs on needle retrieval through 1M tokens, MMMU-Pro and HLE, reporting matching accuracy.

Inkling 2 Small ranked fourth among open models globally and 23rd overall in Code Arena: WebDev at the weight release. Arena’s 239-point improvement compares it with Inkling-Small, the previous variant.

Arena’s release snapshot covered 11.5K real-world agentic sessions and reported +2.9% net improvement, ranking Inkling 2 Small 22nd overall.

Two additional figures distinguish it within the open-model field:

• Steerability: +10.9%, the highest reported for an open model.
• Median task cost: $0.26 versus $1.22 for open-model leader Kimi K3.1 Pro, roughly 4.7 times cheaper.

Arena’s methodology estimates component effects through randomized assignments and session traces, combining user feedback with signals from tool behavior. Its net-improvement score measures the model’s contribution within that evaluation setup.

Access spans hosted APIs and coding tools:

• Together AI: multimodal inference with up to 1M context, according to Together’s announcement.
• Modal: an OpenAI-compatible Shared Endpoint preview ending November 27, plus dedicated deployment, according to Modal’s launch thread. Its pricing page lists $0.30 input, $0.03 cached input and $1 output per million tokens.
• Ollama Cloud: ollama run inkling-2-small:cloud, with zero data retention, according to Ollama.
• Vercel AI Gateway: the model slug is thinkingmachines/inkling-2-small, as published in Vercel’s announcement.
• Prime Inference: the launch pricing screenshot lists $0.15 input and $0.50 output per million tokens.
• OpenCode: temporary free access in selected regions, according to OpenCode’s announcement.

The vLLM recipe supports attention-projection and routed-expert adapters, but expert adapters require BF16 expert weights. For the released NVFP4/MXFP8 experts, its supported path is attention-only adapters.

SGLang and Miles also shipped day-zero support:

• Serving: both checkpoints, DSpark, image input, prefix caching and hierarchical caching across NVIDIA, AMD and Google hardware.
• Training: a BF16 Megatron trainer with SGLang rollouts serving the released quantized experts.
• Adapters: attention, routed experts and the output projection; image-prompt training uses a frozen vision tower.
• Validation: a rank-32 LoRA run on DAPO-Math-17k completed 93 GRPO steps on eight GB300 GPUs.

Thinking Machines expanded the model card with private and semi-private evaluations and official result verifications, according to ziqiao_ma.

Inkling 2 Small scores 29.8% on DeepFinance versus 40.0% for Claude Haiku 5.5, a 10.2-point gap in the table highlighted in juliusai’s release follow-up.

DeepFinance results in Thinking Machines’ expanded evaluation table