Meta publishes Muse Spark 1.2 multimodal evaluation results

• Tool access moved Muse Spark 1.2’s equal-weight 10-benchmark multimodal average from 59.8 to 72.0, a 12.2-point increase, as AI at Meta's tool-use post describes the model inspecting visual inputs before incorporating findings into its reasoning.
• Visual coding is judged on rendered behavior rather than source code alone, according to AI at Meta's visual-coding post, and rohanpaul_ai's Design Arena post places Muse Spark 1.2 first on Video-to-Website at 1279 Elo.
• Meta’s robotics setup separates a high-level planner from a Vision-Language-Action policy, with AI at Meta's bimanual robotics post describing the planner as the agent that decomposes a desk-tidying task.
• Audio-visual input is paired with agentic tools for video-heavy workflows, as AI at Meta's multimodal thread describes alongside a robot-navigation demonstration.
• Meta is previewing WildArtifactBench, an internal agent evaluation that compares generated deliverables pairwise, in the research post that Alexandr Wang linked.

The evaluation methodology caps image dimensions at 2,000 pixels, re-encodes them at Pillow quality 90, and blocks internet access in the coding and GUI containers. The research post also includes a single-page flipbook with page-turn animation and a cursor-tracked magnifying lens that Meta says the model generated.

Meta’s methodology defines its multimodal-intelligence score as an equal average of BabyVision, PerceptionBench, ZeroBench, WorldVQA, SimpleVQA, ERQA, OmniSpatial, CharXiv Reasoning, ChartMuseum, and ChartQAPro. The tool-enabled configuration uses headless coding execution and GUI interaction inside containers, rather than internet search.

The aggregate moves from 59.8 without tools to 72.0 with them. Muse Spark 1.1 moved from 60.2 to 69.1 in the same comparison, so 1.2’s direct-response score sits 0.4 points below 1.1 while its tool-enabled score is 2.9 points higher.

Individual public results are mixed in the Meta post:

• ZeroBench: 46.0% on Muse Spark 1.1 to 54.0% on 1.2.
• SimpleVQA: 73.3% to 75.0%.
• CharXiv Reasoning: 88.4% to 87.6%.

The evaluation uses GPT-OSS-120B as a semantic-equivalence judge on several benchmarks, while ERQA, OmniSpatial, and ChartQAPro use programmatic grading. Meta says third-party model runs are best-effort comparisons and may not reflect environments tuned for each proprietary model.

Meta frames visual coding as image or video translation into web pages and games whose output must render and behave correctly. Its loop has the model re-examine generated artifacts and refine them.

The August 18 Design Arena snapshot reports three separate leaderboards:

• Video-to-Website: Muse Spark 1.2 at 1279 Elo, ahead of Kimi K3 at 1243 and Muse Spark 1.1 at 1236.
• Image-to-Website: Muse Spark 1.2 at 1252, behind Kimi K3 at 1266 and ahead of GPT-5.6 Sol at 1251.
• Image-to-Frontend: Muse Spark 1.2 at 1272, behind Kimi K3 at 1296 and Claude Fable 5 at 1273.

Meta’s methodology notes that these are live public-leaderboard values, fixed to its August 18 read, with each model’s Design Arena default reasoning setting rather than the maximum reasoning effort used elsewhere in its report.

The robotics system uses a specialized Muse Spark variant as an orchestrator: it takes an instruction, decodes tool calls, observes results, and repeats the loop until completion. The high-level model handles scene understanding, similar-object disambiguation, and task decomposition; a lower-level Muse Spark Vision-Language-Action policy executes each subtask.

In the cosmetics-organizer example, the lower-level controller is an experimental smaller Muse-family VLA model, according to the research post. AI at Meta's robot-navigation thread separately shows the same observe-and-call-tools loop navigating an unstructured environment to find a rubber duck.

Meta pairs deep video understanding and dense captioning with web development, real-time search, and spatial-grounding tools. The system is intended to turn audio-video input into a multi-step workflow rather than a one-shot description.

Inside Meta, the research post says Muse Spark produces detailed captions as training data for Muse Image and Muse Video, and converts raw text, images, and video into downstream signals and insights. The post says Muse Spark 1.2 is available through Meta Model API and Muse Code.

WildArtifactBench is Meta’s internal evaluation for agent tasks with arbitrary deliverables and varying degrees of verifiability. It produces artifacts first, samples pairs against baseline-agent artifacts, asks a judge for a binary preference, then fits Elo ratings from those comparisons.

The methodology separates agentic auto-judge and human-judge results, with about 2,000 pairwise comparisons for each. Its common execution envelope is unusually explicit:

• Every trajectory has a 200-turn cap.
• Models use the same Linux sandbox with a desktop GUI, bash, multimedia viewing, and computer-use tools.
• Cost is cold-start input plus output at base on-demand rates, with cache, batch, contributor, regional, and long-context discounts excluded.

Meta has released 10 tasks from the benchmark preview. Its examples include analyzing a cardiac-ultrasound sequence and creating a DQN driving trainer.