Together releases ThunderAgent for KV-cache scheduling in agent workflows

• ThunderAgent's core move is program-aware scheduling: each multi-turn agent workflow becomes a schedulable program with phase, KV footprint, and node placement tracked by the runtime, according to the program-abstraction post.
• The failure mode is KV cache thrashing during tool waits: the cache-thrashing example describes an agent losing its cache, recomputing its whole history, then evicting another agent's cache.
• Together reports more than 2× single-node throughput and roughly 10× lower P50 latency under high concurrency in the launch thread, with a batch-192 comparison showing 803 tok/s for ThunderAgent versus 390 tok/s for SGLang in the single-node benchmark post.
• Multi-node scaling is the bigger systems claim: ThunderAgent went from 671 to 2,248 steps/min from 16 to 64 GPUs, and its lead over SGLang Gateway widened from 1.79× to 2.39× in the cluster-scaling post.
• The adoption path is one extra field: the compatibility post says ThunderAgent adds program_id, supports OpenAI chat completions today, and has already been adopted by SkyRL and NVIDIA Dynamo.

The official blog has the cleanest SGLang comparison, but the paper adds two subsystems the launch thread barely dwells on: tool resource management and lifecycle-aware garbage collection. The API reference shows program_id can live either at the top level or inside OpenAI's extra_body, then gets stripped before the request is forwarded to the backend. The GitHub README also says the ICML Spotlight acceptance was top 2.2%, and the SkyRL recipe reports a 3.01× wall-clock speedup on a 40 H100 Qwen3-32B training run.

Together frames the bottleneck as a mismatch between agent workflows and request-level inference engines. Agent runs alternate between GPU-heavy reasoning and GPU-idle tool waits, while vLLM-style schedulers see only independent requests and evict KV cache with least-recently-used policies, according to the official blog.

The expensive path is simple:

1. Agent A pauses for a tool call.
2. Its KV cache is evicted to make room for Agent B.
3. Agent A resumes and recomputes its conversation history.
4. That recomputation evicts Agent C.
5. The cascade repeats at high concurrency.

Together says two common mitigations only delay the failure. The scaling-and-offloading post says static node pinning leaves some nodes starved and others idle, while CPU or disk offloading pushes the limit outward until the working set outgrows every tier.

ThunderAgent inserts a scheduling layer between agent clients and inference backends. The paper calls the unit an LLM Program, a persistent object that spans multiple model invocations and tool executions.

Each program tracks:

• workflow identifier
• execution phase, reasoning or acting
• scheduling status
• token counts and KV footprint
• node placement
• tool resources, including external assets such as disk memory and network ports

The useful engineering detail is the scheduler boundary: ThunderAgent does not ask vLLM or SGLang to infer workflow state from isolated calls. It gives the scheduler the missing program state directly.

ThunderAgent's scheduler uses program-level admission control under memory pressure. The official blog says it selectively pauses low-priority workflows so active programs keep higher KV hit rates.

The paper names two mechanisms:

• State-aware pausing: acting-state workflows waiting on tools are paused first, preserving memory for reasoning-state workflows.
• Dynamic migration: data-parallel GPU nodes share a global program-aware waiting queue, so resumed programs can move to a node with available capacity.

The blog adds that ThunderAgent treats HBM, CPU RAM, and disk-backed KV capacity as a unified pool when offloading is present. Offloading remains underneath the scheduler rather than replacing it.

Together evaluated ThunderAgent inside its synthetic data generation pipeline, where hundreds of coding agents interact with sandboxed environments over many turns, according to the official blog.

The headline single-node comparison:

• Setup: one 8×H100 node, HiCache KV offloading, batch size 192.
• SGLang baseline: 390 tok/s, 65s mean latency.
• ThunderAgent: 803 tok/s, 10.6s mean latency.
• Delta: about 2× throughput and roughly 6× lower mean latency in that table.

The cluster result moves from tokens to steps per minute:

• 16 GPUs: 671 steps/min.
• 64 GPUs: 2,248 steps/min.
• ThunderAgent versus SGLang Gateway: 1.79× at 2 nodes, widening to 2.39× at 8 nodes.

The paper reports broader evals across SWE-Agent, OpenHands, ToolOrchestra, and scientific discovery agents: 1.5 to 3.6× serving throughput gains, 1.8 to 3.9× RL rollout gains, and up to 4.2× disk memory savings.

The API surface is small because the program identity rides along with ordinary chat completions. The API reference defines POST /v1/chat/completions as an OpenAI-compatible proxy with one extra program_id field.

program_id can be supplied in either place:

or through the OpenAI Python client's extra body:

The API reference says ThunderAgent strips program_id before forwarding the request to the backend. The same docs expose /programs, /metrics, and /profiles endpoints for program state, backend KV usage, prefix cache hit rate, pause time, tool-call time, and per-program profiling.

The paper's quieter subsystem is tool resource management. It treats Docker sandboxes, network ports, remote APIs, and disk-backed environments as persistent resources attached to a program, rather than disposable side effects of a single request.

According to the paper, ThunderAgent overlaps I/O-heavy environment initialization with LLM reasoning and uses program termination signals for lifecycle-aware garbage collection. The GitHub README describes the same feature as tool-call lifecycle management with automatic resource reclaim for long-running rollouts.

That resource layer explains why the paper reports disk memory savings separately from throughput. The system is scheduling KV cache and cleaning up agent environments.

ThunderAgent is open source under MIT, implemented in Python, and supports vLLM and SGLang backends, according to the GitHub README. The same README lists 381 stars, 37 forks, and an April 30 news item saying the ICML 2026 Spotlight acceptance was top 2.2%.

The README also lists two integrations before the July 29 Together blog: NVIDIA Dynamo 2.0 integrated ThunderAgent's program abstraction as a first-class scheduling unit, and SkyRL integrated ThunderAgent for agentic RL training. The separate SkyRL ThunderAgent recipe reports Qwen3-32B training on R2EGym with 5 SLURM nodes and 40 H100 GPUs, with 8.84 hours versus 26.58 hours for 40 training steps, a 3.01× wall-clock speedup over its no-ThunderAgent baseline.