Inkling adds early llama.cpp serving via 1-bit GGUF

• Inkling shipped as a 975B total, 41B active multimodal MoE with full weights, Tinker fine-tuning, and partner support, according to Thinking Machines' launch thread and Soumith Chintala's post.
• The first llama.cpp path is already real: mervenoyann's demo showed Unsloth's 1-bit Inkling quant running at 30-40 TPS, while the follow-up command used llama serve -hf unsloth/inkling-GGUF:UD-IQ1_S for a 250 GB VRAM setup.
• Serving support landed unusually fast, with vLLM's day-zero post citing 380 tok/s/user with MTP on 4 GB200 GPUs and vLLM's TokenSpeed note pointing to a flat paged KV pool for heterogeneous attention state.
• Early public evals split by surface: Arena's frontend post put Inkling at #10 among open-weight frontend code models, Arena's text post put it #10 among open-source text models, and JJitsev's benchmark note flagged Terminal Bench 2.1 as a weak spot against GLM 5.2.
• The customization story is more interesting than the leaderboard story: Ben Burtenshaw's ECHO demo used Inkling, Tinker, and OpenEnv to train environment-response prediction in 15 steps for less than $1.

The official launch post frames Inkling as a broad base model rather than a peak benchmark model. The Unsloth run guide says dynamic 1-bit GGUF cuts BF16 weights from at least 1.9 TB to 270-285 GB, and the vLLM day-zero post explains why short convolutions and relative attention needed custom serving work. Artificial Analysis measured a 41 Intelligence Index score, 25K output tokens per task, and 256K Tinker API context, while the Hugging Face weights support 1M context.

Thinking Machines released Inkling with open weights, an Inkling Playground, and Tinker fine-tuning. The official spec is 975B total parameters, 41B active parameters, a 1M token context window, and native text, image, and audio input.

The launch stack has three pieces engineers will care about:

• Model: 975B total, 41B active, Mixture-of-Experts transformer, trained on 45T tokens of text, images, audio, and video in the official launch post.
• Weights: full weights on Hugging Face, with BF16 and NVFP4 links visible in the model card.
• Customization: Tinker fine-tuning and a Playground, with OpenAI-compatible and Anthropic-compatible endpoints mentioned by Soumith Chintala's API reply.

Chintala also called it the first public model out of the lab's model factory and said, "This is definitely day-1" in a follow-up. That reads like a release note disguised as humility.

mervenoyann's demo made the launch feel less abstract: Unsloth's 1-bit quant was generating in llama.cpp at 30-40 TPS, with the web UI showing a reasoning slider, HTML preview, MCP, and multimodal tabs.

The follow-up command was bare enough to copy:

That command came with a hardware joke: "if you have 250 GB VRAM" in mervenoyann's follow-up. The Unsloth run guide puts the 1-bit UD-IQ1_S artifact at 270 GB of disk and about 290 GB RAM plus VRAM, while BF16 needs at least 1.9 TB of disk.

Unsloth's published trade-off is stark:

• Dynamic 1-bit: 270-285 GB, about 74.2% top-1% accuracy retained.
• Dynamic 2-bit: 317 GB, about 81% top-1% accuracy retained.
• BF16: about 1.9 TB, treated as the 100% baseline.

Unsloth Studio added Inkling with interleaved tool calling, web search, code execution, numeric reasoning levels from 0.0 to 0.99, plus vision and audio support in Daniel Hanchen's feature list. The local path is finally concrete, even if "local" means a serious memory box.

The day-zero serving work was not just a provider badge list. Inkling's architecture forced runtimes to handle relative attention, short convolution state, shared expert sinks, MTP, and multimodal inputs.

The vLLM post says it supports thinkingmachines/Inkling-NVFP4 and BF16 with LoRA, TP/DP/EP/PP parallelism, prefix caching, disaggregated serving, and MTP. Its headline throughput number is up to 380 tok/s/user with MTP and 140 tok/s/user without MTP on 4 GB200 GPUs.

TokenSpeed's interesting bit is memory shape, not raw speed. vLLM's note on LightSeek's work describes a single flat paged pool with heterogeneous views that can hold full attention, sliding-window attention, and convolution states without padding or fragmentation.

The underlying TokenSpeed PR adds Inkling support across 125 files and includes Docker images plus a launch command for thinkingmachines/Inkling-NVFP4. That PR was still open when indexed, which matches the early-release smell around the stack.

Modal took a different serving angle: its Inkling post says it made Inkling available as a managed endpoint with token-based pricing, while Modal's launch post says a custom DFlash speculator improved throughput and interactivity by 67%.

Arena's first posts were strong for a US open model, especially on frontend and text leaderboards.

Arena reported:

• Frontend Code Arena: #10 among open-weight models, 1434 points, #37 overall.
• US open-weight field: best US open model in Frontend Code Arena, and the only US model in the top 15 open-model field.
• Category strengths: #4 in Brand & Marketing, #7 in Data & Analytics.

The Text Arena result was similar but less splashy:

• Text Arena: #10 among open-source models, 1447 points, #58 overall.
• US open-source field: #2, behind Google DeepMind's Gemma-4-31B.
• Highlighted categories: #10 hard prompts, #12 longer query, #14 coding, #15 instruction following.

Artificial Analysis put Inkling at 41 on its Intelligence Index, 3 points above Nemotron 3 Ultra, and called it the leading US open-weight release in its benchmark thread. Its full write-up also says Inkling averaged 25K output tokens per Intelligence Index task, versus 43K for GLM-5.2, 38K for Kimi K2.6, and 37K for DeepSeek V4 Pro.

The weak points showed up fast. JJitsev pointed to Terminal Bench 2.1 at 63.8% for Inkling versus 82.7% for GLM 5.2 in a benchmark reply, and Matt Busigin said Inkling was much weaker than GLM 5.2 in his use cases after a day of testing. Busigin still praised speed and tool invocation for HITL UX in a follow-up, then listed premature termination, prompt sensitivity, and degradation under noisy prompts in his issue list.

The cleanest workflow demo used Inkling less as a chat model and more as a post-training substrate.

Burtenshaw used Inkling, Tinker, and OpenEnv to train a terminal world model with ECHO. The setup predicts a terminal's responses to commands by applying supervision to environment-output tokens, without a verifier.

The linked [OpenEnv script]OpenEnv script builds masked token data for environment outputs, evaluates held-out environment-token cross-entropy, and runs a Tinker LoRA training loop. Burtenshaw said cross-entropy converged in 15 steps for the toy terminal world and cost less than $1 in the demo post.

Tinker also published early customer notes:

• Mantic said Inkling beat Kimi K2.6 on its forecasting evals with half the output tokens.
• Trajectory Labs praised concise reasoning, tool calling, and long-horizon agentic tasks.
• LightningRod AI called the model "thoughtful, original, and refreshingly unsycophantic."

Ben's separate launch thread called Inkling "built around post-training" and "perfect for adapting" in his release notes. That claim fits the product shape better than the leaderboard discourse.

One r/LocalLLaMA post complained that Inkling was "not fully open-weight" because access was region-restricted and a Hugging Face discussion about it had been closed in the Reddit thread. The post's language was overheated, but the access complaint was concrete.

Simon Willison's write-up raised a quieter docs issue: he called the model card much shorter than expected from US AI labs and said the linked training-data documentation had "almost nothing of interest" beyond broad references to public, third-party, and publicly accessible data.

Inkling-Small is also not fully released yet. Justin Lin asked whether the small model was open sourced, and cHHillee replied that the team still needed to finish final testing in the Inkling-Small reply.