Stanford and Together AI report hybrid agents cut decoding compute by 62.5–72.9%

• Decoding compute fell 62.5–72.9% while accuracy improved in Stanford and Together AI's HybridAgent research, according to rohanpaul_ai's summary.
• HybridLink connects latent reasoning with text-based tool interaction, as described in the announcement.
• Conversion requires post-training: omarsar0's breakdown describes supervised HybridSFT followed by reinforcement learning with HybridRL.

A continuous KV cache carries unwritten reasoning across tool calls, the neatest engineering detail here. The repository's inspector renders latent states as rough pseudo-text, including the wonderfully clumsy “Wait, I me the the speed of speed” in its example trace.

Researchers evaluated Qwen3-Base backbones at 4B, 8B and 14B across 11 benchmarks covering mathematics, science, medicine, search and agentic reasoning, according to the linked paper.

The evaluation reports:

• GAIA, 14B: text-only ARPO scored 43.7 versus HybridAgent's 52.4, +8.7 points, in rohanpaul_ai's comparison.
• Aggregate performance: a reported 10.4% average improvement over compared agents and frameworks.
• Token usage: 35.5–54.7% lower.
• Inference speed: 1.8–2.6× faster.
• Computational cost: reduced by a factor of 2.7–3.7 in the authors' accounting.

The paper frames that last figure as computational savings. Token reduction, inference speed and decoding compute are separate reported measurements.

The adapter adds approximately 10 million parameters to a 4B backbone, according to Section 3.1. It maps the last-layer hidden state back into the input-embedding space through two linear projections, an activation and a residual connection.

HybridLink and the agent's transitions between text and latent reasoning.

Generation switches modes through entry and exit anchors:

• Enter latent mode: the model emits a special entry token.
• Continue reasoning: HybridLink transforms each hidden state into the next step's input, bypassing normal token decoding and re-embedding.
• Return to text: the transformed state aligns most closely with the exit anchor's token embedding.

Each latent step still requires a transformer forward pass. Tool calls and final answers use text mode.

The agent's working memory is an accumulated, position-ordered KV cache spanning every transformer layer, according to the memory design. It retains entries from generated text, latent reasoning and tool observations.

Each tool turn resumes directly from that cache and appends new entries. Re-tokenizing the visible transcript alone cannot reconstruct the continuous latent portion of the context.

The post-training pipeline adapts an existing pretrained backbone in two stages:

1. HybridSFT: teach the entry/exit format, freeze the backbone while aligning HybridLink with target embeddings, then jointly train both components on the adapter's own predicted latent inputs.
2. HybridRL: optimize when to reason in latent space and when to act through text, using exact latent-context replay for policy updates over hybrid trajectories.

Training data assigns reasoning spans to latent mode using expert-model judgments and the target model's prediction entropy. Selected steps remain textual for tool interaction and inspection.

Inspection maps each latent step to its nearest token embedding by cosine similarity, as specified in the latent-mode interface. The resulting pseudo-text is a readout; generation continues from continuous representations.

Replacing latent spans with material from other problems or random tokens cut accuracy by approximately 45–51%, while tool-call execution success remained 99.7%, according to DAIR.AI's account of the experiments.

The MIT-licensed repository lists these artifacts:

• Models: togethercomputer/HybridAgent-8B-Agentic and togethercomputer/HybridAgent-14B-Agentic, both trained through HybridSFT and HybridRL.
• Data: two HybridSFT trajectory datasets, plus HybridRL mathematics and agentic-search datasets.
• Inference: JSONL batch processing with Python and search tools. Search uses a TAVILY_API_KEY.

Each model directory bundles the backbone with adapter.pt and adapter_config.json. The training launchers recommend 8 or 16 GPUs for HybridSFT and 32 GPUs across four nodes for HybridRL.