PhoneLLM releases Mini and Max open-weight models with sub-200ms token claims

• PhoneLLM Alpha 2 ships two open-weight Nemotron fine-tunes: 30B Mini and 120B Max, according to the launch announcement.
• Claimed sub-100ms and sub-200ms answer-token latency applies per inference call. The production target is below 600ms at P95, as a follow-up clarifies.
• Max generates one sentence of reasoning before answering, adding roughly 100ms, according to the microthinking explanation.
• Mini has a pay-per-token endpoint on Cloudflare; Max runs on two B200s through Baseten or Modal, as the deployment reply explains.

The launch notes include quantized Max weights that fit on a DGX Spark. Mini’s model card disables reasoning in its bundled chat template.

Pipecat released Alpha 2 on October 15 as two text models for voice-agent pipelines. Both are full-weight supervised fine-tunes:

• Mini: Nemotron 3 Nano, with 30B total parameters and 3.5B active parameters. It operates without reasoning.
• Max: Nemotron 3.5 Super VL, with 120B total parameters and 12B active parameters. It uses a short reasoning step before replies or tool calls.

Max’s neat trick is a roughly 100ms reasoning budget, spent on a single sentence. Pipecat says this improves tool selection and instruction following.

The official serving recipes set temperature=0.0 for both models. Mini runs without a reasoning parser; Max enables thinking and uses --reasoning-parser nemotron_v3.

The headline numbers measure time to first answer token, or TTFAT. Pipecat separately reports time to first token of 17ms for Mini on one B200 and approximately 27ms for Max on two B200s; Max emits reasoning before its answer.

Production adds three costs beyond the quoted per-call figures, according to the launch author’s clarification:

• Tool use: two LLM round trips.
• Remote inference: network latency.
• Loaded servers: queueing and contention.

With those costs included, Pipecat’s production target is TTFAT below 600ms at P95.

Pipecat’s own benchmark reports substantial gains over the underlying Nemotron models:

• Mini versus Nemotron 3 Nano: 29.7% → 75.6%, +45.9 percentage points.
• Max versus Nemotron 3.5 Super VL: 65.1% → 83.0%, +17.9 percentage points.

The Max comparison changes the reasoning setting as well as the weights: base Nemotron runs with thinking off, while Max uses microthinking. The leaderboard also separates overall task performance from tool-call failures and major conversational errors.

Alpha 2 expands training in three directions:

1. Prompt flexibility: reduced sensitivity to wording and prompting styles compared with Alpha 1.
2. Task coverage: broader training for support, appointment reminders, scheduling, information lookup and voice front ends for software.
3. Pipecat Flows: synthetic sessions teach the models the state-machine harness’s prompting format and conversation structure.

Flows keeps prompts focused on individual subtasks. In customer testing, Pipecat says Max resolved more calls successfully than GPT-4.1 inside Flows, at around half the latency.

PhoneBench now tests correction of earlier conversational mistakes, instruction sequences and Flows scenarios. The training account describes fully synthetic datasets generated by open-weight teacher models; reinforcement learning remains experimental rather than part of the released models’ training recipe.

The deployment options span serverless token billing, dedicated inference and local hardware:

• Cloudflare Workers AI: Mini costs $0.06 per million input tokens and $0.24 per million output tokens. Its model documentation lists function calling, OpenAI-compatible chat completions and a 262,140-token context window.
• Dedicated inference: both variants have deployment links through Baseten and Modal. Max uses two B200s, according to the hardware reply.
• AWS: the release supports deployment through SageMaker AI inference clusters.
• Local Mini: NVFP4 weights fit on an RTX 5090. The release also provides FP8 and BF16 variants.

The official vLLM examples cap model length at 32,768 tokens, below Cloudflare’s listed context capacity.

Mini can run alongside local Nemotron ASR, retaining the last few minutes of context until a word or statement triggers action, according to the launch author’s local-use report.

Fast local responses expose turn-detection mistakes immediately: a false positive makes the agent interrupt the speaker almost at once.