H-Net researchers report 3–5× training efficiency gains with learned tokenization

• Controlled H-Net tests show up to 3–5× training-efficiency gains, with the strongest results on code, according to the research announcement.
• A separate two-stage H-Net matches SmolLM3’s six-task average using roughly 10× less training data and compute, as the comparison charts report.
• The study covers pretraining only; its conclusion leaves post-training transfer for future work.

An invented word, borvexila, becomes a single chunk after repeated appearances in context. Removing BPE’s regex preprocessor compresses text more tightly while making training less compute-efficient.

Cartesia’s unusually careful BPE baseline gets a tokenizer trained on the same corpus. The matched one-stage H-Net keeps the Transformer backbone and training hyperparameters unchanged.

Its byte-processing outer networks match BPE’s embedding and output layers in parameters and FLOPs. An auxiliary loss holds compression at BPE’s average of 4.57 bytes per unit.

The fitted multipliers measure training FLOPs needed to reach the same loss:

• Held-out training distribution: 1.28×.
• Paloma: 1.83×.
• OLMES: 2.06×.
• In-domain code: 2.04× median.
• Out-of-domain code: up to 5.37×, on Prolog.

The 3–5× headline describes the stronger domain-specific results. Python’s smaller gain prompted a question about the headline comparison:

The larger two-stage H-Net reaches SmolLM3-3B’s final six-task average with roughly one-tenth the training data and compute in Cartesia’s larger-scale test. This run tests a broader architecture change than the tightly matched one-stage ladder.

H-Net researcher Albert Gu credited SmolLM3’s open recipe and checkpoints with making a comparison at the target size possible.

An H-Net processes inputs through five modules:

1. Encoder: reads raw inputs, such as bytes.
2. Chunker: learns to compress their representations into fewer units.
3. Main network: processes those coarse units.
4. Dechunker: restores the original resolution and adds a residual connection.
5. Decoder: finishes processing the combined representations.

The main network can itself be an H-Net, adding another level of learned chunking. Discrete boundary decisions make routing non-differentiable, which Gu likened to MoE.

Variable names in code can acquire context-dependent token boundaries:

The chunker also has unusual training dynamics: Gu said the token count initially collapses before the model starts segmenting again. He later acknowledged similar in-context behavior in BLT and, to a degree, the original H-Net.

Cartesia’s second study separates two scaling relationships:

• Compute-to-loss: training FLOPs needed to reach a given validation loss.
• Loss-to-loss: downstream loss at a given validation loss.

Three findings explain why the downstream advantage exceeds the validation-loss advantage:

• Pretokenization: the regex constrains which byte sequences BPE can merge. Removing it increased compression from 4.57 to 6.03 bytes per token, yet adding it back made compute-to-downstream evaluation almost 1.9× more efficient.
• Transfer: at equal validation loss, H-Net transferred to OLMES about as well as a pure byte Transformer and better than BPE.
• End-to-end learning: a BLT-like ablation kept H-Net’s hierarchy and compression rate but chose boundaries using a proxy model’s entropy. Its weaker loss-to-loss transfer supported Cartesia’s attribution of part of the gain to jointly learned chunking.

A prompt ending in a space can stop midway through a BPE token that normally includes leading whitespace. Correct byte-level probabilities require accounting for alternative tokenizations of the same text.

Cartesia applied a boundary correction to the BPE-without-pretokenization baseline because its uncorrected OLMES scores were badly distorted. Standard BPE benefited much less from the fix; the published standard BPE curves retain conventional evaluation.

• Larger runs: the headline comparison is roughly 3B-class; one response requested 8B or 30B experiments because scaling curves can bend.
• Existing models: Gu reported “somewhat negative” early attempts at adapting existing models and favored training such a different architecture from scratch.
• Post-training: the study’s conclusion leaves transfer beyond pretraining to future work.
• Release: Cartesia says it plans to release the improved H-Net version used in these experiments.

Cartesia places H-Net inside its long-term multimodal research agenda, with early results on DNA and images. Its stated audio goal is speech language models that learn directly from raw audio and reason natively over both text and sound, a bottleneck the company says first motivated this research.