Open Athena begins training 535B-parameter Marin model

• Marin is training as a 535B-total, 23B-active MoE over 18T tokens, a configuration that percyliang's progress update placed 13% through the run.
• The release covers more than a final checkpoint: TheTuringPost's overview says code pipelines, training logs, and checkpoints are public, while Open Athena's official announcement calls it an open, live run.
• Marin's headline count resolves into 384 routed experts, top-8 routing, two shared experts, and a 4,096-token starting context in the GitHub job specification, a plan described as fully open by TheTuringPost.
• Funding for the CoreWeave run comes from the Jen-Hsun and Lori Huang Foundation, as percyliang's post says; percyliang's follow-up separately credits Google TPU Research Cloud support for Marin's earlier models.

The public tracker assigns about 1% of total compute to a scaling ladder, partly to catch failures before they reach the hero run. It also records pinned-host optimizer state, an hourly temporary checkpoint after preemptions, and a public W&B dashboard for the run.

The A23B label refers to per-token active capacity. The job specification gives the more exact totals as 535.3B parameters and 22.76B active parameters, with about 2.70e24 training FLOPs.

Its 48-layer, d6144 model has 384 routed experts and selects eight per token. Two dense SwiGLU shared experts run alongside the routed path; the configuration uses 48 attention heads, a 128,256-token vocabulary, and a 4,096-token maximum sequence length at the start.

Open Athena's announcement says it will make the training pipeline, logs, data recipes, weights, and infrastructure available. The project has published the configuration and scaling ladder while training is underway, rather than limiting disclosure to a post-training model card.

The foundation's compute gift pays for CoreWeave capacity on the hero run, according to percyliang's announcement. The project also credits TPU Research Cloud for earlier models and names the Siegel Family Foundation, Fulcrum Science, and Schmidt Sciences among its supporters in percyliang's funding follow-up.

The CoreWeave-backed run's tracker specifies 11 GB200 NVL72 racks and a 64-way expert-parallel group inside each rack.

• Each EP64 group spans 16 nodes and four GPUs per node.
• The 11 racks are data-parallel replicas connected across the data-center network.
• The plan sets a batch of 11,264 sequences, 4,096 tokens per sequence, and 390,139 training steps.
• The expert collective uses fixed, three-wave buffers, with expert IDs carried inside the activation collective rather than sent through a separate metadata collective.

The accompanying expert-parallel engineering note describes a latent-space transport path of width 3,072, half the model's 6,144 hidden width.

Marin begins at 4K context to put twice as many sequences in each batch for expert balancing, the run plan says. The same plan flags token dropping as the main long-context constraint: prior tests rose from roughly 7% dropping at 4K to roughly 40% at 65K, while the pooled-wave implementation measured about 3% at 4K.

The team plans a one- to two-day cooldown 10 to 20 days into training for full-scale RL experiments. Its contingent extension schedule is 4K to 8K around the 50% mark, 8K to 65K around 95%, and a targeted 262K phase near the end of the run.