multiPL-E regex bug corrupts MBPP benchmark language variants

• A substring replacement corrupted MultiPL-E's MBPP language variants: the intended python substitution instead replaced py wherever it occurred, as code_star's bug report explains.
• The audit reached beyond malformed prompts into test cases, formatting, and a handful of upstream MBPP defects, code_star's audit report says.
• Aggregate results kept the same directional signal across languages, but individual languages produced large winners and losers, according to code_star's follow-up.
• The repair comparison separates original data, a regex-only repair, and a full fix across 13 models and 141 benchmark shards in code_star's comparison charts.

MultiPL-E's documentation describes a suite of compilers that turns Python benchmark problems into parallel tasks in 18 languages, while the repository identifies HumanEval and MBPP as its source benchmarks. The investigation used Codex through MCP alongside nac, whose official write-up describes a thread-and-episode harness for long-running agent work.

The defect sat in MultiPL-E's MBPP subset. In code_star's thread, the stated intent was to replace the word python with a target-language name, but the actual operation replaced the shorter string py.

That turns names into artifacts such as rsthon, swthon, and gothon.

A code_star clip of the corrupted substitutions records those malformed strings appearing in the benchmark material.

The repair run compared 13 models over 141 shards, 72 HumanEval and 69 MBPP, for 1,833 completed model-shard tasks at 20 samples per task. Its reported metric is raw empirical pass@1.

The three data conditions are explicit in the charts:

• Original, the benchmark before repair.
• Regex repair, the narrow substitution fix.
• Full fix, which also incorporates the deeper defects found by the audit.

The broader comparison did not reverse the aggregate language-level direction, code_star wrote in a results follow-up, but it exposed substantial language-specific movement. A Ruby view in the attached charts makes the gap between the original and repaired curves especially visible.

The author used Codex through an MCP server to create 47 concurrent nac sessions, then said the entire MBPP and HumanEval-variant audit finished within two hours. That pass found issues in prompts, test cases, formatting, and upstream MBPP examples.

The reported scope matters because MultiPL-E is unit-test-driven: translated prompt text and the tests used to judge generated code belong to the same evaluation artifact. The disclosed full fix therefore extends beyond correcting the original string replacement.

code_star said in a release note that fixed version or versions of the eval will be open-sourced with a fuller blog post. The same post put the hosted nac investigation at a couple of hundred dollars and described a process that went from curiosity to result in hours rather than days.

The public artifact was still framed as forthcoming, not as an already published update to the canonical benchmark.