Sakana lists 11 ICML 2026 papers on LLM speed, memory, and agent evals

• Sakana's ICML 2026 thread lists 11 papers across sparse LLM systems, context handling, long-term memory, test-time scaling, multi-agent coordination, and agent evaluation.
• The TwELL paper post claims over 20% speedups for sparse billion-parameter LLM training and inference by pairing a sparse packing format with custom CUDA kernels.
• The SoftMatcha 2 post says its matcher searches 1.4T tokens in under 0.3 seconds and finds benchmark contamination that exact string search misses.
• The Sheaf-ADMM thread reports a 93% Multi-Agent Sudoku solve rate versus 11% for a parameter-matched message-passing baseline.
• The Fast-weight Product Key Memory post reports about 75% 5-needle NIAH accuracy at 128K context and up to -0.4 ΔNLL after reading 4M target-domain tokens.

Sakana's full lineup is unusually systems-heavy for a conference thread: sparse CUDA kernels, context re-positioning, corpus search, model merging, agent economics, and two memory papers. The sharpest engineering claims sit in TwELL's GPU sparse-kernel speedups, SoftMatcha 2's sub-second trillion-token search, Doc-to-LoRA's document-to-adapter memory, and FwPKM's sparse fast-weight writes at inference time.

Sakana said it will present 11 papers at ICML 2026 in Seoul, running July 6 to 11. The set clusters into five useful buckets:

• LLM efficiency: TwELL sparse packing and CUDA kernels for sparse Transformer training and inference. TwELL announcement
• Context handling: RePo, a learned context re-positioning module that assigns positions by content relevance rather than fixed index order. RePo announcement
• Agent evaluation and coordination: CoffeeBench for long-horizon economic agents, plus Sheaf-ADMM for differentiable local-to-global coordination. CoffeeBench announcement Sheaf-ADMM lineup post
• Optimization and test-time scaling: spherical black-box optimizer unification for model merging, plus UnMaskFork for masked diffusion language models. BBO lineup post UnMaskFork announcement
• Memory, data, and feedback: SoftMatcha 2 corpus search, Doc-to-LoRA document internalization, Fast-weight Product Key Memory, tandem speech front-end capacity, and Feedback-to-Rubrics. SoftMatcha 2 announcement Doc-to-LoRA announcement speech front-end announcement FwPKM announcement Feedback-to-Rubrics post

The through-line is practical machinery around models: cheaper execution, better long-context plumbing, sharper evals, and memory systems that avoid stuffing every fact into the prompt.

TwELL targets the part of Transformer LMs where sparsity can matter most: feedforward layers. Sakana says the work introduces a sparse packing format built for optimized tiled GPU workloads, then adds custom CUDA kernels for training and inference.

Reported claims:

• Billion-parameter-scale sparse models.
• Over 20% speedups in training and inference benchmarks.
• Higher savings in peak memory and energy than the speedup alone suggests.
• Collaboration with NVIDIA AI.

The key detail is the hardware fit. Sparsity only pays if the kernel path preserves accelerator efficiency, and Sakana's TwELL blog frames the format around tiled GPU execution rather than abstract parameter pruning.

RePo attacks a long-context nuisance: standard LMs treat the prompt as a rigid sequence where token index doubles as the only structural signal. Sakana says RePo lets a module learn positions based on content relevance, so facts buried in noisy inputs can be reorganized before the model spends capacity on them.

The paper's claim is architectural, not just retrieval flavored. The model still consumes context, but the positional assignment becomes content-aware instead of inherited from the prompt's physical order.

Sakana has two separate memory papers in the lineup.

Doc-to-LoRA:

• Input: a document.
• Output: a LoRA adapter that internalizes the document into the model's weights.
• Claimed effect: later questions can be answered without putting the source document back into the context window.
• Reported savings: lower latency and VRAM, with memory beyond the model's native context limit.

Fast-weight Product Key Memory:

• Mechanism: Test-Time Training plus Product Key Memory.
• Storage pattern: large sparse fast weights with sparse access and test-time writes.
• Long-context result: about 75% 5-needle Needle-in-a-Haystack accuracy at 128K context.
• Adaptation result: up to -0.4 ΔNLL after reading 4M tokens from a target domain.

The two papers sketch different answers to the same product problem. Doc-to-LoRA compiles a document into an adapter, while FwPKM writes into sparse memory during inference.

CoffeeBench evaluates an LLM agent as a coffee roaster over 90 simulated days in an economy with two farmers, two roasters, and two retailers. The benchmark measures net income rather than single-turn task success.

Sakana highlighted one concrete failure mode: idle drift in Claude Haiku 4.5, where the agent becomes inactive despite coherent reasoning traces. For agent builders, that is the spicy bit: the trace can look plausible while the business process quietly stops moving.

Sheaf-ADMM decomposes a task into overlapping local views, gives one agent each local piece, and coordinates the agents through ADMM with a network sheaf specifying which neighboring solution components must agree. Sakana also posted an official JAX/Flax implementation.

The negotiation loop has three steps:

1. Local guesses: each agent proposes a solution from its limited view.
2. Finding common ground: neighboring agents smooth conflicts on overlapping boundaries.
3. Remembering disagreements: unresolved conflicts become memory that pushes later compromise.

Reported results from Sakana's Sheaf-ADMM thread:

• Multi-Agent Sudoku: 93% solve rate versus 11% for a parameter-matched message-passing baseline.
• MNIST canvas-size domain shift: standard CNN at 11% accuracy, Sheaf-ADMM at 86%.
• Maze pathfinding: matched message-passing accuracy with a 5-dimensional communication channel, versus 42 for the baseline.

The framework is interesting because the coordination state is inspectable. Agents debate local constraints instead of hiding the whole process inside opaque message-passing vectors.

Sakana's black-box optimization paper claims that Evolution Strategies, Consensus-Based Optimization, Optimization via Integration, and related methods can be written as variants of one spherical update equation. The paper then uses that bridge to build hybrid optimizers, including AdaPol and SchedPol.

The practical target is model merging under constrained evaluation budgets. Evaluating large LMs at every step is expensive, and the thread says standard unimodal optimizers can overfit smaller evaluation sets; treating merging as multimodal let the hybrids find multiple distinct optima before selecting generalized merges.

SoftMatcha 2 is built for soft, word-order-aware search over trillion-token corpora. Sakana says it searches 1.4T tokens in under 0.3 seconds and catches benchmark contamination that exact search misses.

The useful distinction is soft matching at corpus scale. Exact string search catches verbatim leakage; SoftMatcha 2 is aimed at near matches, substitutions, insertions, deletions, and template-style contamination where the benchmark did not appear byte-for-byte.

UnMaskFork applies test-time scaling to Masked Diffusion Language Models. Instead of sampling many stochastic outputs, it uses Monte Carlo Tree Search to explore deterministic action branches while dynamically switching among multiple pre-trained MDLMs.

Sakana says the method outperforms Best-of-N and tree-search baselines on coding and math reasoning tasks. The core bet is that model diversity creates better search branches than random perturbation when generation happens through iterative unmasking.

Two smaller items fill out the lineup.

The tandem speech front-end paper removes layers from a KAME-style speech-to-speech front-end, fine-tunes each variant, and evaluates on Speech MT-Bench. The setup reflects a split architecture where a low-latency speech front-end handles interaction while a back-end LLM carries knowledge and reasoning.

Feedback-to-Rubrics tries to learn reusable natural-language rubrics from inline comments on drafts. The target is tacit expert criteria, such as house style or reviewer preferences, that are visible in comments but rarely written as a clean evaluation spec.