Marin starts training open 535B-A23B model on 18.75T tokens

• Marin has begun training 535B-A23B, a mixture-of-experts model with roughly 535 billion total parameters and 23 billion active parameters, according to percyliang's run announcement.
• The public plan puts 80% of the schedule in pretraining and 20% in midtraining, on 18.75 trillion tokens across 11 GB200 NVL72 systems over roughly three months, per percyliang's plan.
• A four-rung scaling ladder preceded the hero run, spanning 1.6B-A61M on 48B tokens through 27.7B-A1.2B on 926B tokens, according to percyliang's announcement.
• Marin will compare intermediate loss against forecasts from the ladder, as percyliang's forecast update explains, while percyliang's tracking links point to the data mixture, live Weights & Biases run, and code.

The run configuration includes a contingency for hardware or utilization delays: early in the run, Marin can shorten the token horizon and retune the linear decay schedule. Its expert-parallelism design note packs expert IDs into the same activation collective rather than issuing a separate metadata collective.

The detailed configuration places EP64, its 64-way expert-parallel axis, inside each 72-GPU rack. The 11 racks are data-parallel replicas connected over the data-center network, with a global batch of 11,264 sequences at 4,096 tokens each.

The token accounting already has a useful wrinkle. The announcement calls the voyage 18.75T tokens; the job specification sets total_tokens to 18.0T, but also describes a 15T pretraining phase plus a 3.75T cooldown and simulated epoching against an 18.75T reference. The published configuration does not reconcile those figures.

Each of the model's 48 transformer blocks uses an MoE MLP. The configuration specifies 384 experts, top-8 routing, 6,144 hidden dimensions, 48 attention heads, two 3,072-wide shared SwiGLU experts, and 22.76B active parameters out of 535.3B total.

The run's routing stack has several material implementation choices:

• LatentMoE compresses token representations from 6,144 to 3,072 dimensions before dispatch, halving expert all-to-all transport bytes.
• A histogram-based quantile router applies a next-step, zero-mean bias to balance expert selection without changing router gradients.
• Capacity factor 1.15 and three static expert waves bound compute and transport buffers; sender and receiver token-drop fractions are logged.
• Three quarters of layers are local sliding-window attention, while every fourth layer and the final layer are global, full-causal layers.

Marin says the ladder costs about 1% of hero-run compute and serves as a reference for loss, gradient norms, token dropping, and evaluations across the roughly 100-day schedule. The run issue says a prior ladder exposed gradient-norm growth that led the team to add logit z-loss, after ablations found some high-batch runs could otherwise blow up mid-run.

The live Weights & Biases tracking page is part of the public artifact set.

The model starts at 4K sequence length, despite Marin's previous run beginning at 8K and later extending to 65K. According to the long-context plan, 4K doubles the number of sequences per batch for better expert balancing while the pooled-wave expert-parallel implementation remains experimental.

Prior tests saw token dropping rise from about 7% at 4K to about 40% at 65K. Marin reports roughly 3% dropping at 4K with pooled-wave EP, but expects 65K dropping could still be excessive.

A one-to-two-day cooldown 10 to 20 days into training is meant to test the extension path. The current schedule sketches 4K to 8K at 50% of the run, 8K to 65K around 95%, then a targeted 262K phase near the end; alternatives include a dropless ragged all-to-all implementation, higher capacity factor, or sequence-level balancing.