MIT researchers use Astra to raise Qwen MATH-500 accuracy from 7.9% to 48.1%

• Qwen3.5-0.8B’s MATH-500 accuracy rose from 7.9% to 48.1%, +40.2 percentage points, with sampling held fixed, according to the benchmark report.
• The edit suppressed repetitive reasoning loops; a different model with ordinary math errors showed no clear improvement, as WesRoth’s account explains.
• The broader work extends LLM optimization from prompts and code into neural-network weights, as described in an author’s announcement. The experiments demonstrate model repair, not recursive self-improvement.

The Qwen experiment’s feedback budget allowed 50 batches of up to 512 practice problems, rather than just 50 questions. Elsewhere in the same study, Astra constructed a 1.48-million-parameter Go network in 336 lines of Python without playing a trial game.

MIT CSAIL researchers Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab published Sparks of Weight-Space Programming on November 2. Their Qwen experiment gave GPT-6 Astra a Python session containing the model’s weights, plus practice problems from the GSM8K and MATH training sets; test questions stayed hidden.

Astra’s repair followed three steps:

1. Diagnose repeated reasoning that prevented Qwen from finishing answers.
2. Identify a repetition-associated internal direction written by layers 15 and 19.
3. Subtract that signal’s contribution from the weights, changing roughly 131,000 parameters, or 0.016% of the model.

The share of answers exhausting the 32,768-token limit fell from 93% to 33.1%. The same edit also improved instruction following on IFEval by roughly 3×, without optimizing for that task.

Astra’s notebook investigation and targeted weight edit

The diagnosis-to-patch workflow is the gem here.

Astra was explicitly steered toward repetition and neurons that might suppress it, a detail giffmana highlighted after reading the prompt.

A shorter prompt removed the repetition hint. The published comparison, discussed in a close reading of the experiment, reported:

• 25 evaluation batches: both the original and shorter prompts reached 14.1% on MATH-500.
• 50 evaluation batches: the original prompt reached 48.1%; the table contains no matching result for the shorter prompt.

The shorter-prompt result leaves the headline gain’s dependence on that guidance unresolved.

The experiment also used temperature 0.6. Qwen’s model card recommends temperature 1.0 for thinking-mode text tasks, although sampling was unchanged between the experiment’s baseline and edited model.

Astra produced no clear math improvement in the researchers’ Gemma 3 270M control, which rarely looped.

For three MinAtar games, Astra started with zero-valued weights and received the rules, source code, and network layout. It wrote the weights directly as text, without tools, gradients, training data, or trial games, according to the experiment write-up.

Scores across 1,000 evaluation episodes were:

• Breakout: PPO at 100 million steps, 15.8; PPO at 500 million steps, 25.3; Astra, 517.4.
• Seaquest: PPO at 100 million steps, 5.6; PPO at 500 million steps, 88.9; Astra, 53.1.
• Asterix: PPO at 100 million steps, 25.8; PPO at 500 million steps, 55.2; Astra, 53.2.

The comparisons drew criticism from yacineMTB, who argued that the PPO baselines were insufficiently tuned.

The case studies largely apply known interpretability techniques, according to a detailed critique, which questioned how well those methods would scale to messier behavior in larger models.

Astra edited a separate model. Claims that it had self-improved were explicitly rejected by willdepue.

Jürgen Schmidhuber, author of the 1991 fast-weight work, called the result “old hat”. His earlier systems trained a controller through backpropagation to change another network’s fast weights.

MIT CSAIL coauthor Omar Khattab framed the new work around general-purpose LLMs reasoning directly about weights from natural-language goals. He also pointed out that the introduction already acknowledged networks trained to write other networks’ weights.

Khattab put the entire project’s cost at roughly $1,000 in a follow-up, then clarified in another reply that the estimate included credits.